Sunday, December 15, 2019

Trying to Fix Windstad Manor with Deletes

So I have a problem with Winstad Manor where I have two sets of walls active on the main hall.

I have loaded the save and gone into the hall, used the console to select the wall that I think should not be there.  Then used the "Disable" console command.  Nothing much happened.  I then exited the hall, went outside, then re-entered the hall. 

Problem.  It would appear that the object I targeted was infact the door, not the wall.  The door is now gone.

Ok,  tried again on the wall.  Select, disable, exit cell, re-enter... still there.  It would seem that there are different properties on some of the major geometry.

Need to try markfordelete.  Wish me luck.

So, after some trial and error:

Right hand lower floor wall
(030126a0) is the wall with doorways (keep it)
(03012053) is the wall without doorways ( markfordelete)

Left hand lower floor wall
(030126c1) is the wall without doorways (markfordelete)
(030126b2) is the wall with the doorway (keep it)

Rear lower floor wall
(030126a2) wall without doorways (markfordelete)
(03012054) wall with doorway (keep it)

Right upper floor wall
(03012679) wall with doorways (keep it)
(03012051) wall without doorways (markfordelete)

Left Upper floor wall

I did not need to fix this as I had the greenhouse wing which does not use the upper floor wall.  So I can live with having two walls if they are there(I suspect they are.  Same with multiple foundations)

(030126c0) wall without doorways (markfordelete) if you need it

Rear Upper Wall
(030126a1) No doorway (markfordelete)
(03012052) Doorway.  (Keep it)


The easy way to do this is to simply open the doors and select the wall that is in the door way. (obvious in hindsight..)

So that is all the unwanted walls removed from Windstad Manor for me.  Save a good save file and move on to hacking at Lakeview.

Additionally, some hacking that I have done recently has caused a coffin to appear in the cellar at Windstad Manor.  It think it was when I was fiddling around resetting the house build quests.  I think one of the stages represents you being a vampire. 

Anyway,  select coffin in the console, markfordelete.  Reload the cell and its now gone.  Finally I can invite the dudes from the Dawnguard over for Nord Mead again without them getting all suspicious...

More Lakeview debugging

So I have gotten the Creation Kit installed via steam.

It installed in the default location for the steam library which is not where I wanted it installed. I have a second drive that is where I am spreading out all the modding stuff and is out of the UAC write protected system.

However, Steam would not move the files for Creation Kit to the new location as there was already a steam library there.  (It contains skyrim).  So to get it all to happen, I had to rename the existing skyrim folder, (skyrimX) then use the move steam library tool inside steam to move Creation Kit... then merge the files it placed into my skyrim files.... then change the name of the "renamed" folder back to "Skyrim".  This fooled steam into moving the Creation Kit path correctly.

I could then install Creation Kit into Mod Organiser as an executable.  This allows me to run Creation Kit against my full mod stack, with the benefits of the MO virtualised file system.  Very nice.

Then manually unpack the source files that are distributed with Creation Kit into the Data\Scripts\Source folder.

There is still some issues with Creation Kit as it throws a bunch of errors on load.

C:\_Skyrim\Code\TESV\TES SHared\misc\BGSLocalizedString.cpp
C:\_Skyrim\Code\TESV\TES SHared\misc\BGSLocalizedString.cpp
C:\_skyrim\code\tesv\bssystem\BSTSmartkPointer.h

As these are throwing errors based on c++ source file locations its very deep in the guts of CK.  The solution I found is documented in the following nexus post. It involves adding the DLC .esm files to the load paths in the SkyrimEditor.ini. 

https://www.nexusmods.com/skyrim/mods/21690/

Once I had CK loading without errors... but still with a bunch of warnings... (Who leaves this many warning in production code? They don't look hard to resolve. Is this a measure of the resource constraints inside Bethesda? Probably. There are only so many hours in the day and hands at the wheel and it gives the modders something to do...)

Ok,  Crack open CK and start to fossic around.  Read through the minimum of the tutorials to get an idea of how to navigate around.  Load up some of the world space cells and get a look at how some of the bits are actually implemented.

https://www.skyrimsearch.com/categories/cells/10/

This database is useful for all sorts of random references inside the codebase.  It's actually something I was considering creating before I found it.  Nice.

Anyway,  home in on the cell with Lakeview Manor and Winstad Manor.  It was pretty much as I expected, all the geometry is stacked in position and ready to be turned on or off via scripts.  Now to figure out the scripts.

I started picking my way into the carpenter bench setting to get an idea of what it does.  Ok.  But when I try to open the source file for the script from within Creation Kit, it is greyed out.  Path error for some reason?  If I double click on a script reference, it just opens the properties dialog.  Useful but no way to get to the script...

https://forums.nexusmods.com/index.php?/topic/6845297-script-edit-source-greyed-out-story-time/

I found some notes on installing DLC scripts for FO4 edition of CK.  However it does not seem to get a win for skyrim version of CK. So ignore that.

The solution I found was in the preferences dialog in CK.  Go to the scripts tab and in the "On double click" dropdown select "Edit Script".  Apply and try.

Now CK will correctly open the script when I click on a script on an object in the scene.  Now I can right click on a script and get to both the properties and to the script itself.  Nice.

The inconsistencies in the coding of the CK tooling is interesting. Having written a lot of quick and dirty tools over the years I can see the same problems.  Lack of consistency, no followup once it works  "good enough".  However, this is a code base they have been using as a production tool for a long time.  Would it kill them to put a few hours into polishing it?  Anyway... resouces again... See reddit for other rants on CK tool quality if you are bored. 

However, like other problems with skyrim, its the sort of thing you could solve by releasing feral power of the modding community on it as an open source project... probably some IP reason not to... anyway. There appers to be some open source attempts at replacing Creation Kit but its hard to gauge how complete they may be.  https://github.com/Open-CK




Debugging Lakeview Manor in Skyrim DLC

Ok, enough with getting lost with the tools.  My task is to actually debug some stuff. My primary objective is debugging some of the mess that I have encountered with the Hearthfire DLC.

Here is my tail of woe.

I purchased and built out my Heljarchen Hall house.  No problem.  Learned how to do it, had some fun, moved stuff around then wished I had built different wing options.

Then I picked up Winstad Manor and started working on that.  Critically, I was still building and adding furniture options, ran out of resources and went adventuring to get some more.

Then I picked up Lakeview Manor and started working on that.  Then the bugs started.  I could not get the door between the starter house and the main hall to appear at the Lakeview site.  None of the construction for the main hall foundations and wall would appear.  So I tried a bunch of stuff... then went back to Winstad for supplies and noticed that shit had gone wrong there.

Where Previously I had already completed the rooms in the main hall, it seemed to have reverted the 3D assets and I now had both walls with doors into the wings and walls without doors.  Almost like the walls I was trying to build at Lakeview had turned up at Winstad.... sound of penny dropping!

It seems to me that within the script to build the different buildings, there is a script that is assuming which location is being build, or a variable that is setting the location, is not being correctly updated when I fast travel between the two building sites.  And so, stuff that is being build is turning up on the wrong site and trashing my existing house rather than being added to the correct build site.

No problem thinks I... Just try to roll it back... however, it between figuring it all out, I had done a bunch of other stuff,  played about with my mod list, made a mess of my save files and generally lost track of where I was up to.  So trying to dig into the scripts of the BYOHhouse scripts seemed like a simpler solution... ROFL.  Or at least more interesting to the programmer in me... famous last words.

So, I have started poking around, learning where stuff is, how it works etc.  Hindsight suggests that what I am trying to do is fairly ambitious and potentially really really ignorant as it requires doing a lot of stuff backward... but such is the situation.

So here is what I have learned so far.

The assets for the house building are from the Hearthfires DLC.  The source for this DLC is included with the Creation Kit.  So download and install the Creation Kit from Steam... figure out where the source files are, unpack, navigate... start playing a guessing game based on a very random naming convention and very little in the way of comments.

Dig into TES5Edit and poke around in the Hearthfires esp.  See a whole bunch of the named objects but still get no clear idea of how all the bits interact.

Look around the internet and find that the main solution to this kind of tangle is to roll back to an earlier save and get on with your life.  Fair enough, as trying any other solution is probably futile for the average gamer.

So, in my playing, I have formed a rough idea of where all the bits lay. As with any game, we have three bits to potentially mess around with:

1) The static game files which, together with the engine will at runtime, create the dynamic game state.

2)  A running instance of the game with all the assets and such loaded.

3) A save file, which represents some subset of the dynamic state, from which a game instance can re-create enough of the dynamic state to keep the player amused.

While its interesting to get to understand how the bugs happened and the scripts involved, the only way to "fix" the issues without rolling back to a save where the issue does not exist, is to mess with dynamic state or the save file.

I spent some time playing around in the dynamic game tying to patch the bugs.  I tried resetting the quest for the lakeview manor build but could not get it to "reset" the building site in any way.  It may be possible but I figure that there is already substantial "stuff" at the building site that cannot be deleted by any of the build scripts.  So resetting the quests seemed to be a useless idea.  I tried using console commands to delete some of the assets, but could not figure out how to target them, or if it would be possible to reset the build quest in such a way as to "re-start" the build process.

The save file starts looking like a potentially better angle of attack.  Surely someone must have figured out how to unpack the save?

So there is documentation on the save file structure,  however, there is no way I want to try to to a low tech hack on a binary file of about 20MB size.

https://en.uesp.net/wiki/Tes5Mod:Save_File_Format
http://www.uesp.net/wiki/Tes5Mod:File_Format_Conventions

So, look for tools or roll my own.  Also a huge job.  There are a number of save file editors:

Random bits on Github
https://github.com/navinparray/tesv-skyrim-save-reader
https://github.com/OpenNingia/tesvsavedump

There are also short notes and clues around the web about how to use TES5Edit to manipulate save games. 
https://www.reddit.com/r/skyrimmods/comments/2w31vh/tesv_ess_editor_is_great_and_fixed_a_broken_save/

Tools on Nexus

Savegame Script Scalpel (Also called Papyrus Data Transfer or PDT)
Save Game Script Cleaner (Also called SaveTool) (Bit minimal but may be useful.  Does not work with Crash Fixes mod)
FallrimTools - Script cleaner and more  (Also called ReSaver)

FallrimTools seems to be the most up to date and is still supported. (Fallrim Docs

Youtube Resources

PDT resources on youtube (Savegame Script Scalpel)
https://www.nexusmods.com/skyrim/videos/428/?  (No audio so hard to guess whats going on)


FallrimTools/ReSaver resources on youtube
Modding Short Tutorials #3 Using Resaver fixes a glitched quest  (Virtually no useful information)
UPDATE VIDEO BLOG 5 - FALLRIM TOOLS  (how to install. No useage information)
Skyrim Special Edition Script Cleaner Tutorial ( Fallrim/ReSaver Basic usage tutorial but dated...)
Fallrim Check  (Current 2019 with some interesting hints...)
Skyrim How To Clean Your Saves Tutorial (Fairly current, more basic usage)

Save Game Script Cleaner resources on youtube
Skyrim Modding Tutorial - Save Cleaning  (Basic Usage)
Skyrim | Save Game Script Cleaner (SGSC) Tutorial  (Usage information once he gets through some blather)
Skyrim Mod of the Day - Episode 260: Save Game Script Cleaner ( Fairly comprehensive usage)
Skyrim Save Game Script Cleaner Review!  (2014  Basic Usage )


A possible solution is to use Save Game Script Cleaner to reset the whole of hearthfire.esm.  This will trash all my houses and all their content, adoptions, etc etc.  Pretty much a sledgehammer to kill a mosquito.





Debugging Tools for Skyrim

This is a set of notes on the methods and mechanics for debugging skyrim mods that I have collected.  These notes are about working on the PC.  If you are on another platform or console, your mileage may vary... greatly.

Who is this for?  What is it for?

Myself mostly, but at a more general level, these notes are written for programmers.  The stuff that I'm attempting to debug is in the guts of other peoples mods, core scripts or if I get really bored, SKSE.  Debugging the skyrim engine is beyond what is humanly possible without access to the source code and a solid education in game engine design, debugging C++ and some serious hours.  Trying to do it via disassembly or in a real time debugger is a monstrous task. Patching it would also be the stuff of nightmares.

This is a list of helpful notes so I can remember where I am up to and what I have learned.  It's not intended to be comprehensive or a tutorial on anything in particular.  Its just useful information, collected in one place that I can look over when I need to get some perspective.
My problem is that I find bits of information scattered around the web and there are times when its hard to bring it back to mind or find the answer to a question as I can't remember how I phrased it the last time I stumbled accross some vital detail.  Hence,  writing it down as I go.  That's all this is.

Tools 

Console Commands

The console commands inside skyrim are some of the first debugging, bug avoidance tools you will need to be familiar with.  If you are in a state where the game is still responsive, but you are jammed up or experiencing some unexpected behaviour, then you may be able to "hack" you way around the issue with some creative console work. 

- Stuck in walls or between static elements

Use the console command tcl to toggle clipping off,  close the console and move to a safe area (slightly above the ground/floor) then use the console and the command tcl to turn clipping back on.  Close the console.  You will then gently float back to the ground.  

- Bugged Quests

There are all manner of quests that can get into an unplayable state.  This is due to two factors.  The first is that the original quest authors did not check for all possible cases in their quest scripts (and handle them).  The second is that with modding, the number of edge cases can change. As such, there are lots of ways that quests can get jammed.  
I have tried resetting quest variables and skipping ahead in some quests to try to get back on track.  However, this is a lot of hit and miss and there are some quests that I just cannot get to work. Mostly because I can't figure out what has gone wrong from the limited information that I can find.  

- Unfindable stuff

There are times in the game when its just hard to locate something and it starts getting un-fun to keep looking.  Lack of time and patience means that being able to use a console command to add a particular item is less immersion breaking than giving up and leaving the task unfinished.  Because you know that in a game as big as skyrim + mods, its possible you will never find your way back to this situation if you walk away.  So, find the items FormID via the help command or on the web  

-  player.additem *FormID* *quantityToAdd*

Mod Organiser 2

Apart from being a generally awesome tool for managing the chaos that is modding.. MO has a number of handy features that provide information and methods for debugging.  

 - Tools to manage the mods

Simply keeping on top of which mods are installed, what version they are, if updates are available, where to get the updates, where the notes for the mod are... etc.  This provides a lot of very valuable information about what is going into your skyrim game and allows you to fiddle with it very easily. 

- Profiles

The ability to mix and match mods on demand is nice, but the ability to run multiple profiles with different mixes turns this up to 11.  As a debugging tool, the ability to do comparative analysis is very very valuable. 

TES5Edit

Understanding what mods are doing to the base game and each other is really handy.  This is a kind of static analysis for the properties and variables that the mods mess around with.  It does not seem to provide any ability to see into the papyrus code that the mods run, but for trying to see which mods are stepping on each others toes, it does give you a view all the way down to the metal of the script engine.

- Cleaning dirty mods.

I recommend the auto clean function, tedious as it is.  (Would a batch method be too much to ask for?)   Keeping track of which mods need cleaning, which ones should never be cleaned and what other chaos you can self-inflict is all part of the TES5Edit experience.
This could be made simpler.

TES5Edit - Fixing Errors and merging mods

How to Make a Patch with TES5edit

zEdit

This is a kind of fork of xEdit.  https://z-edit.github.io/#/docs?t=Overview
 Looks like it has a similar feature set but with a different skin.  Still filling out the existing feature set.  Mostly targeted at working with mods and merging them.  Nothing obvious about hacking save files.  

Creation Kit

Still Exploring its debugging capacity. Certainly good for getting to grips with what the mods are doing and trying to understand all the guts of skyrim and how the mods function.  Those with any source code anyway.  Otherwise can just see some of the assets.

Creation Kit Source Files

Good for figuring out what the hellstorm is going on in the DLC's and general experience about how mods are built, styles of coding in the scripts and ways to actually do anything in Skyrim.

LOOT (Or BOSS)

These tools provide a way to manage load order dependencies.  LOOT also contains notes on various mods that need cleaning and some hints on resolving errors with some mods.
Resolving the dependency graph is a very useful process that you will need to perform each time you fiddle with your list of mods.

Papyrus Log / Skyrim Logging

The log from the scripting engine is ... a dumping ground for messages from scripts.  Its marginally useful,  contains a mess of false positives, warnings and spurious errors.  Once you mod list gets a bit long or you are trying to debug anything... it will start to fill up with a spew of useless information.  There is no way to address most of the issues that will appear here... so I am still unconvinced that its worth much.  However, you may see more value...

First you will need to turn your log on:

- Find your skyrim.ini file.
If you are using Mod Organiser then its under the Tools button > INI Editor
Otherwise, its in your skyrim folder.

- Turn on logging

Change the papyrus section to read:

[Papyrus]
fPostLoadUpdateTimeMS=500.0
bEnableLogging=1
bEnableTrace=1

bLoadDebugInformation=1


- Finding the log files
Once you have logging enabled, you will find the log files in your profile something like

C:\users\\My Games\Skyrim\Logs

Other people have reported that the logs will appear in the skyrim\logs folder.  However this should not be possible if skyrim is installed in a vanilla location as it should be write protected... so you may need to check for the logs in a couple of places.

Save File Cleaners

There are a couple of save file cleaners on nexus.  I have cracked open a file or two but not done anything substantial with them beyond reading my save files.  As the interaction between the save file and the game is a bit of a black box.  Currently trying to manually edit the save is only trial and error.

Save game script cleaner https://www.nexusmods.com/skyrim/mods/52363
Savegame script scalpel - Disassembler - Diagnostic Tool - Papyrus Data Transfer
https://www.nexusmods.com/skyrim/mods/53045

Tutorial - How to edit script variables inside a savegame

WRYEBASH

Another interesting and complex tool.  Still Exploring.  Good for packing mods and patches.  Similar to TES5Edit in that respect.

BSA Unpacker

Being able to unpack files from mod BSA packages.  Useful... but not essential yet.