The cannibal conundrum

 There is a point of view expressed by various folk in the covid debate that goes something like.. 

"We need to have a conversation about how many deaths we're willing to accept".  I may be paraphrasing but the current exemplar is Tony Abbot.  

So what's the problem?  

The problem is that this point of view is based on a fundamental assumption that works for narcissistic personality but which does not hold when extended to a society. Let me explain. 

The narcissistic point of view divides the world into a group around the individual narcissist (me/us/we) and other.  The other may be a homogeneous other or many small and large groups of others.  However they are all outside the group that the narcissist recognizes as extensions of themselves. 

So, in essence, us and them.  The division between the two groups is that the "us" is connected to the narcissist, has utility/value or relationships that the narcissist identifies with. The concept of identity is very important as it allows the narcissist to recognise and relate to people they have never met, but still can allocate into the "us" group.  Similarly, the narcissist is able to very quickly allocate identity and group membership to people in the "other" group.  Those to whom the narcissist has no relationship and thus, no value.  This is essentially an impairment of the ability to perceive or comprehend networks of value above a certain number of links.  Try playing "six degrees of separation" with this kind of personality and you will quickly see how they struggle to articulate how there is any commonality between their "us" group and their "them" group.  

This crippled ability to understand or value the connections in the network allows the narcisist to rationalise all sorts of policy and behavior that would otherwise be unacceptable. 

Again, what's the problem?  The problem is that this social structure is centered around the narcisist.  It does not generalise.  It's the "king" at the center of the world. The king decides who is in and who is out, and when the king is replaced, all the rules change because a different king decides a different set of "in" and "out".  

Watching our former prime minister articulate his mental illness on the international stage was pretty sad.  He wants to have a conversation about how many deaths a country is willing to accept.  The implicit assumption made by people trying to have this conversation is that the potential death(s) will not include his. 

This is where the narcisist point of view always falls down.  When the rules are applied to everyone fairly the logic fails.  If Tony had said, lets all draw straws for who will die and who will live,  and the drawing was actually fair. (random with replacement) and he took his chance the same as everyone else it would have been more convincing.  However he is not capable of that.  By some chain of rationalization he will always be above the rules.  His skill as a politician was not that he was crazy, but that he was able to conceal it as much as he did. 

His logic is perfectly sound, as long as you accept the assumption that there is an "us" and "them" divide and "us" can live perfectly well (or even better) without "them". 

There are so many fallacies that disprove this assumption but you cannot fix mental illness with "talk".  There are never enough words to fix whatever went wrong in his childhood.  

So what's the converse point of view and will the world end? 

The converse point of view (one of them atleast) is that the majority of people are connected via a network of trade, personal relationships, family relationships, cultural obligations, location sharing, legal entities, social group memberships etc.  The point is that this network is what we call "society" and you can't have the benefits of many parts without all the supporting messy networks.

This is not to say that every part of the mess is positive. It's just to say that there is interconnected-ness that does not go away no matter how much anyone wishes.  There is good and bad.  And there is interplay between the good and bad.  There are many emergent phenomena that could not exist without the tension between positive and negative.  

Anyway... its a very big picture and its scary.  There are lots of folk who for whatever reason are struggling to deal with the picture right in front of them, so I don't begrudge them not wanting to deal with one that encapsulates the whole of our collective time and space.  Simplification can help get you through some difficult times.  

And why is cannibalism a conundrum?  Because once you pop, you can't stop.  There's no natural limit once a group applies a strategy of cannibalism.  It's always an option to pick off the weakest member when rationalisation demands.  But there's always a weakest member until there is only 1.  

Once you start to rationalise that the group can do without a weakest/oldest/youngest/stupid/poor/blue eyed/crazy/ugly/sick etc... and the people making that rationalisation get a win from it... then its just a matter of time before they go back to that well and try again.  

Because at the end of the day, if it was about food or survival, there are natural limits.  But when cannibalism is for political or economic outcomes, its simply becomes an attractive option for a weak politician when other options are hard.  Because it gets multiple wins.  The first is that it reduces potential opponents.  The second is because it induces fear into other potential opponents and encourages self-censorship.  Lastly there are the resource benefits (these are used as the rationalization for the behavior, but the intangibles ability to control death are often more "politically" valuable to the politician personally)  Imagine the step for a poitician who has previously had to battle their opponents using all the usual indirect, underhanded arguments, bullying and competition to suddenly get the idea that they can directly influence the death of some of their unwanted "enemies within".  

Keep in mind that this is not a unique moment in history.  The cannabalism strategy has been confronted by every society and organisation ever.  The current COVID pandemic is just the current reason.  Watching large organisations go through various staff reduction programs is just the same rationalisation.  "We can survive... but only if "we" change the definition of "we".  

This kind of survival shedding of people is not anything unrealistic.  The main issue that I object to is the deception both of the person proposing the strategy and those who agree.  The deception is that the "we" who remain are the same "we".  They still identify with the same identity as they did prior to the split.  It would be more honest to modify the identity to reflect the trauma.  This acknowledges the event and gives something to both the survivors and those who got pushed off the liferaft.  By failing to acknowledge this in the organisation or group identity, they are not dealing with the fact that the event even happened and everyone who went through it will carry some of the scars.  The only people who do not carry a scar are those who were mentally ill in the first place.  Those who never conceived of the fact that it might be them who get selected to be sacrificed for the good of the group.  

Trying to Fix Windstad Manor with Deletes

So I have a problem with Winstad Manor where I have two sets of walls active on the main hall.

I have loaded the save and gone into the hall, used the console to select the wall that I think should not be there.  Then used the "Disable" console command.  Nothing much happened.  I then exited the hall, went outside, then re-entered the hall. 

Problem.  It would appear that the object I targeted was infact the door, not the wall.  The door is now gone.

Ok,  tried again on the wall.  Select, disable, exit cell, re-enter... still there.  It would seem that there are different properties on some of the major geometry.

Need to try markfordelete.  Wish me luck.

So, after some trial and error:

Right hand lower floor wall
(030126a0) is the wall with doorways (keep it)
(03012053) is the wall without doorways ( markfordelete)

Left hand lower floor wall
(030126c1) is the wall without doorways (markfordelete)
(030126b2) is the wall with the doorway (keep it)

Rear lower floor wall
(030126a2) wall without doorways (markfordelete)
(03012054) wall with doorway (keep it)

Right upper floor wall
(03012679) wall with doorways (keep it)
(03012051) wall without doorways (markfordelete)

Left Upper floor wall

I did not need to fix this as I had the greenhouse wing which does not use the upper floor wall.  So I can live with having two walls if they are there(I suspect they are.  Same with multiple foundations)

(030126c0) wall without doorways (markfordelete) if you need it

Rear Upper Wall
(030126a1) No doorway (markfordelete)
(03012052) Doorway.  (Keep it)


The easy way to do this is to simply open the doors and select the wall that is in the door way. (obvious in hindsight..)

So that is all the unwanted walls removed from Windstad Manor for me.  Save a good save file and move on to hacking at Lakeview.

Additionally, some hacking that I have done recently has caused a coffin to appear in the cellar at Windstad Manor.  It think it was when I was fiddling around resetting the house build quests.  I think one of the stages represents you being a vampire. 

Anyway,  select coffin in the console, markfordelete.  Reload the cell and its now gone.  Finally I can invite the dudes from the Dawnguard over for Nord Mead again without them getting all suspicious...

More Lakeview debugging

So I have gotten the Creation Kit installed via steam.

It installed in the default location for the steam library which is not where I wanted it installed. I have a second drive that is where I am spreading out all the modding stuff and is out of the UAC write protected system.

However, Steam would not move the files for Creation Kit to the new location as there was already a steam library there.  (It contains skyrim).  So to get it all to happen, I had to rename the existing skyrim folder, (skyrimX) then use the move steam library tool inside steam to move Creation Kit... then merge the files it placed into my skyrim files.... then change the name of the "renamed" folder back to "Skyrim".  This fooled steam into moving the Creation Kit path correctly.

I could then install Creation Kit into Mod Organiser as an executable.  This allows me to run Creation Kit against my full mod stack, with the benefits of the MO virtualised file system.  Very nice.

Then manually unpack the source files that are distributed with Creation Kit into the Data\Scripts\Source folder.

There is still some issues with Creation Kit as it throws a bunch of errors on load.

C:\_Skyrim\Code\TESV\TES SHared\misc\BGSLocalizedString.cpp
C:\_Skyrim\Code\TESV\TES SHared\misc\BGSLocalizedString.cpp
C:\_skyrim\code\tesv\bssystem\BSTSmartkPointer.h

As these are throwing errors based on c++ source file locations its very deep in the guts of CK.  The solution I found is documented in the following nexus post. It involves adding the DLC .esm files to the load paths in the SkyrimEditor.ini. 

https://www.nexusmods.com/skyrim/mods/21690/

Once I had CK loading without errors... but still with a bunch of warnings... (Who leaves this many warning in production code? They don't look hard to resolve. Is this a measure of the resource constraints inside Bethesda? Probably. There are only so many hours in the day and hands at the wheel and it gives the modders something to do...)

Ok,  Crack open CK and start to fossic around.  Read through the minimum of the tutorials to get an idea of how to navigate around.  Load up some of the world space cells and get a look at how some of the bits are actually implemented.

https://www.skyrimsearch.com/categories/cells/10/

This database is useful for all sorts of random references inside the codebase.  It's actually something I was considering creating before I found it.  Nice.

Anyway,  home in on the cell with Lakeview Manor and Winstad Manor.  It was pretty much as I expected, all the geometry is stacked in position and ready to be turned on or off via scripts.  Now to figure out the scripts.

I started picking my way into the carpenter bench setting to get an idea of what it does.  Ok.  But when I try to open the source file for the script from within Creation Kit, it is greyed out.  Path error for some reason?  If I double click on a script reference, it just opens the properties dialog.  Useful but no way to get to the script...

https://forums.nexusmods.com/index.php?/topic/6845297-script-edit-source-greyed-out-story-time/

I found some notes on installing DLC scripts for FO4 edition of CK.  However it does not seem to get a win for skyrim version of CK. So ignore that.

The solution I found was in the preferences dialog in CK.  Go to the scripts tab and in the "On double click" dropdown select "Edit Script".  Apply and try.

Now CK will correctly open the script when I click on a script on an object in the scene.  Now I can right click on a script and get to both the properties and to the script itself.  Nice.

The inconsistencies in the coding of the CK tooling is interesting. Having written a lot of quick and dirty tools over the years I can see the same problems.  Lack of consistency, no followup once it works  "good enough".  However, this is a code base they have been using as a production tool for a long time.  Would it kill them to put a few hours into polishing it?  Anyway... resouces again... See reddit for other rants on CK tool quality if you are bored. 

However, like other problems with skyrim, its the sort of thing you could solve by releasing feral power of the modding community on it as an open source project... probably some IP reason not to... anyway. There appers to be some open source attempts at replacing Creation Kit but its hard to gauge how complete they may be.  https://github.com/Open-CK

Debugging Lakeview Manor in Skyrim DLC

Ok, enough with getting lost with the tools.  My task is to actually debug some stuff. My primary objective is debugging some of the mess that I have encountered with the Hearthfire DLC.

Here is my tail of woe.

I purchased and built out my Heljarchen Hall house.  No problem.  Learned how to do it, had some fun, moved stuff around then wished I had built different wing options.

Then I picked up Winstad Manor and started working on that.  Critically, I was still building and adding furniture options, ran out of resources and went adventuring to get some more.

Then I picked up Lakeview Manor and started working on that.  Then the bugs started.  I could not get the door between the starter house and the main hall to appear at the Lakeview site.  None of the construction for the main hall foundations and wall would appear.  So I tried a bunch of stuff... then went back to Winstad for supplies and noticed that shit had gone wrong there.

Where Previously I had already completed the rooms in the main hall, it seemed to have reverted the 3D assets and I now had both walls with doors into the wings and walls without doors.  Almost like the walls I was trying to build at Lakeview had turned up at Winstad.... sound of penny dropping!

It seems to me that within the script to build the different buildings, there is a script that is assuming which location is being build, or a variable that is setting the location, is not being correctly updated when I fast travel between the two building sites.  And so, stuff that is being build is turning up on the wrong site and trashing my existing house rather than being added to the correct build site.

No problem thinks I... Just try to roll it back... however, it between figuring it all out, I had done a bunch of other stuff,  played about with my mod list, made a mess of my save files and generally lost track of where I was up to.  So trying to dig into the scripts of the BYOHhouse scripts seemed like a simpler solution... ROFL.  Or at least more interesting to the programmer in me... famous last words.

So, I have started poking around, learning where stuff is, how it works etc.  Hindsight suggests that what I am trying to do is fairly ambitious and potentially really really ignorant as it requires doing a lot of stuff backward... but such is the situation.

So here is what I have learned so far.

The assets for the house building are from the Hearthfires DLC.  The source for this DLC is included with the Creation Kit.  So download and install the Creation Kit from Steam... figure out where the source files are, unpack, navigate... start playing a guessing game based on a very random naming convention and very little in the way of comments.

Dig into TES5Edit and poke around in the Hearthfires esp.  See a whole bunch of the named objects but still get no clear idea of how all the bits interact.

Look around the internet and find that the main solution to this kind of tangle is to roll back to an earlier save and get on with your life.  Fair enough, as trying any other solution is probably futile for the average gamer.

So, in my playing, I have formed a rough idea of where all the bits lay. As with any game, we have three bits to potentially mess around with:

1) The static game files which, together with the engine will at runtime, create the dynamic game state.

2)  A running instance of the game with all the assets and such loaded.

3) A save file, which represents some subset of the dynamic state, from which a game instance can re-create enough of the dynamic state to keep the player amused.

While its interesting to get to understand how the bugs happened and the scripts involved, the only way to "fix" the issues without rolling back to a save where the issue does not exist, is to mess with dynamic state or the save file.

I spent some time playing around in the dynamic game tying to patch the bugs.  I tried resetting the quest for the lakeview manor build but could not get it to "reset" the building site in any way.  It may be possible but I figure that there is already substantial "stuff" at the building site that cannot be deleted by any of the build scripts.  So resetting the quests seemed to be a useless idea.  I tried using console commands to delete some of the assets, but could not figure out how to target them, or if it would be possible to reset the build quest in such a way as to "re-start" the build process.

The save file starts looking like a potentially better angle of attack.  Surely someone must have figured out how to unpack the save?

So there is documentation on the save file structure,  however, there is no way I want to try to to a low tech hack on a binary file of about 20MB size.

https://en.uesp.net/wiki/Tes5Mod:Save_File_Format
http://www.uesp.net/wiki/Tes5Mod:File_Format_Conventions

So, look for tools or roll my own.  Also a huge job.  There are a number of save file editors:

Random bits on Github
https://github.com/navinparray/tesv-skyrim-save-reader
https://github.com/OpenNingia/tesvsavedump

There are also short notes and clues around the web about how to use TES5Edit to manipulate save games. 
https://www.reddit.com/r/skyrimmods/comments/2w31vh/tesv_ess_editor_is_great_and_fixed_a_broken_save/

Tools on Nexus

Savegame Script Scalpel (Also called Papyrus Data Transfer or PDT)
Save Game Script Cleaner (Also called SaveTool) (Bit minimal but may be useful.  Does not work with Crash Fixes mod)
FallrimTools - Script cleaner and more  (Also called ReSaver)

FallrimTools seems to be the most up to date and is still supported. (Fallrim Docs

Youtube Resources

PDT resources on youtube (Savegame Script Scalpel)
https://www.nexusmods.com/skyrim/videos/428/?  (No audio so hard to guess whats going on)


FallrimTools/ReSaver resources on youtube
Modding Short Tutorials #3 Using Resaver fixes a glitched quest  (Virtually no useful information)
UPDATE VIDEO BLOG 5 - FALLRIM TOOLS  (how to install. No useage information)
Skyrim Special Edition Script Cleaner Tutorial ( Fallrim/ReSaver Basic usage tutorial but dated...)
Fallrim Check  (Current 2019 with some interesting hints...)
Skyrim How To Clean Your Saves Tutorial (Fairly current, more basic usage)

Save Game Script Cleaner resources on youtube
Skyrim Modding Tutorial - Save Cleaning  (Basic Usage)
Skyrim | Save Game Script Cleaner (SGSC) Tutorial  (Usage information once he gets through some blather)
Skyrim Mod of the Day - Episode 260: Save Game Script Cleaner ( Fairly comprehensive usage)
Skyrim Save Game Script Cleaner Review!  (2014  Basic Usage )


A possible solution is to use Save Game Script Cleaner to reset the whole of hearthfire.esm.  This will trash all my houses and all their content, adoptions, etc etc.  Pretty much a sledgehammer to kill a mosquito.

Debugging Tools for Skyrim

This is a set of notes on the methods and mechanics for debugging skyrim mods that I have collected.  These notes are about working on the PC.  If you are on another platform or console, your mileage may vary... greatly.

Who is this for?  What is it for?

Myself mostly, but at a more general level, these notes are written for programmers.  The stuff that I'm attempting to debug is in the guts of other peoples mods, core scripts or if I get really bored, SKSE.  Debugging the skyrim engine is beyond what is humanly possible without access to the source code and a solid education in game engine design, debugging C++ and some serious hours.  Trying to do it via disassembly or in a real time debugger is a monstrous task. Patching it would also be the stuff of nightmares.

This is a list of helpful notes so I can remember where I am up to and what I have learned.  It's not intended to be comprehensive or a tutorial on anything in particular.  Its just useful information, collected in one place that I can look over when I need to get some perspective.
My problem is that I find bits of information scattered around the web and there are times when its hard to bring it back to mind or find the answer to a question as I can't remember how I phrased it the last time I stumbled accross some vital detail.  Hence,  writing it down as I go.  That's all this is.

Tools 

Console Commands

The console commands inside skyrim are some of the first debugging, bug avoidance tools you will need to be familiar with.  If you are in a state where the game is still responsive, but you are jammed up or experiencing some unexpected behaviour, then you may be able to "hack" you way around the issue with some creative console work. 

All Skyrim Console Commands on Fandom

All Skyrim Console Commands on UESP

- Stuck in walls or between static elements

Use the console command tcl to toggle clipping off,  close the console and move to a safe area (slightly above the ground/floor) then use the console and the command tcl to turn clipping back on.  Close the console.  You will then gently float back to the ground.  

- Bugged Quests

There are all manner of quests that can get into an unplayable state.  This is due to two factors.  The first is that the original quest authors did not check for all possible cases in their quest scripts (and handle them).  The second is that with modding, the number of edge cases can change. As such, there are lots of ways that quests can get jammed.  

I have tried resetting quest variables and skipping ahead in some quests to try to get back on track.  However, this is a lot of hit and miss and there are some quests that I just cannot get to work. Mostly because I can't figure out what has gone wrong from the limited information that I can find.  

- Unfindable stuff

There are times in the game when its just hard to locate something and it starts getting un-fun to keep looking.  Lack of time and patience means that being able to use a console command to add a particular item is less immersion breaking than giving up and leaving the task unfinished.  Because you know that in a game as big as skyrim + mods, its possible you will never find your way back to this situation if you walk away.  So, find the items FormID via the help command or on the web  

-  player.additem *FormID* *quantityToAdd*

Mod Organiser 2

Apart from being a generally awesome tool for managing the chaos that is modding.. MO has a number of handy features that provide information and methods for debugging.  

 - Tools to manage the mods

Simply keeping on top of which mods are installed, what version they are, if updates are available, where to get the updates, where the notes for the mod are... etc.  This provides a lot of very valuable information about what is going into your skyrim game and allows you to fiddle with it very easily. 

- Profiles

The ability to mix and match mods on demand is nice, but the ability to run multiple profiles with different mixes turns this up to 11.  As a debugging tool, the ability to do comparative analysis is very very valuable. 

TES5Edit

Understanding what mods are doing to the base game and each other is really handy.  This is a kind of static analysis for the properties and variables that the mods mess around with.  It does not seem to provide any ability to see into the papyrus code that the mods run, but for trying to see which mods are stepping on each others toes, it does give you a view all the way down to the metal of the script engine.

- Cleaning dirty mods.

I recommend the auto clean function, tedious as it is.  (Would a batch method be too much to ask for?)   Keeping track of which mods need cleaning, which ones should never be cleaned and what other chaos you can self-inflict is all part of the TES5Edit experience.
This could be made simpler.

TES5Edit - Fixing Errors and merging mods

How to Make a Patch with TES5edit

zEdit

This is a kind of fork of xEdit.  https://z-edit.github.io/#/docs?t=Overview
 Looks like it has a similar feature set but with a different skin.  Still filling out the existing feature set.  Mostly targeted at working with mods and merging them.  Nothing obvious about hacking save files.  

Creation Kit

Still Exploring its debugging capacity. Certainly good for getting to grips with what the mods are doing and trying to understand all the guts of skyrim and how the mods function.  Those with any source code anyway.  Otherwise can just see some of the assets.

Creation Kit Source Files

Good for figuring out what the hellstorm is going on in the DLC's and general experience about how mods are built, styles of coding in the scripts and ways to actually do anything in Skyrim.

LOOT (Or BOSS)

These tools provide a way to manage load order dependencies.  LOOT also contains notes on various mods that need cleaning and some hints on resolving errors with some mods.
Resolving the dependency graph is a very useful process that you will need to perform each time you fiddle with your list of mods.

Papyrus Log / Skyrim Logging

The log from the scripting engine is ... a dumping ground for messages from scripts.  Its marginally useful,  contains a mess of false positives, warnings and spurious errors.  Once you mod list gets a bit long or you are trying to debug anything... it will start to fill up with a spew of useless information.  There is no way to address most of the issues that will appear here... so I am still unconvinced that its worth much.  However, you may see more value...

First you will need to turn your log on:

- Find your skyrim.ini file.
If you are using Mod Organiser then its under the Tools button > INI Editor
Otherwise, its in your skyrim folder.

- Turn on logging

Change the papyrus section to read:

[Papyrus]
fPostLoadUpdateTimeMS=500.0
bEnableLogging=1
bEnableTrace=1

bLoadDebugInformation=1


- Finding the log files
Once you have logging enabled, you will find the log files in your profile something like

C:\users\\My Games\Skyrim\Logs

Other people have reported that the logs will appear in the skyrim\logs folder.  However this should not be possible if skyrim is installed in a vanilla location as it should be write protected... so you may need to check for the logs in a couple of places.

Save File Cleaners

There are a couple of save file cleaners on nexus.  I have cracked open a file or two but not done anything substantial with them beyond reading my save files.  As the interaction between the save file and the game is a bit of a black box.  Currently trying to manually edit the save is only trial and error.

Save game script cleaner https://www.nexusmods.com/skyrim/mods/52363
Savegame script scalpel - Disassembler - Diagnostic Tool - Papyrus Data Transfer
https://www.nexusmods.com/skyrim/mods/53045

Tutorial - How to edit script variables inside a savegame

https://www.nexusmods.com/skyrim/videos/428/?

WRYEBASH

Another interesting and complex tool.  Still Exploring.  Good for packing mods and patches.  Similar to TES5Edit in that respect.

BSA Unpacker

Being able to unpack files from mod BSA packages.  Useful... but not essential yet.

Wix toolkit finance

To bitch or not to bitch, that is the question...

Hmmm where to start?

Half a decade ago, microsoft essentially abandoned setup tools... for their own operating system.  There was a bit of a zombie effort in Visual Studio with the old installer tools being included then dropped then kind of returned... but it was walking dead.  The politics were against it and they really seemed to want people to move on and only create software for their new app store thing.  So why not kill off another of the essential tools that real people depened on to support their customers?

Anyway,  see the history of my rants on this blog if you are into badly written screeds of bile about how that has poisoned my projects.

Fastforward... abit.  The anointed successor has been the WiX toolset.  Essentially, microsoft have dumped the responsibility onto the open source community to support one of their core platform tools.  Be that as it may... in half a decade, the WiX folk have produced a toolset of command line tools that work.  It handles the insane complexity of the task and integrates with the insane number of toolsets for building windows apps... and the insane number of platform changes that microsoft keep vomiting forth... all for gratis. 

Now, yet again, I am trying to work with WiX and yet again, I have run into the problem that existed every other time I have had the mis-fortune to run into it.... documentation.

It's monumental.  (as in a monument to something that you really wish happened to someone else).  Everything you need to know about the WiX toolset is in there... somewhere but there is no coherency to the knowledge it contains.  You need to self assemble that.

Consider this,  for the best part of a decade, the WiX folk have been hammering away at this problem and their best efforts is documentation of the toolset and schema, one pretty sketchy tutorial, stack overflow discussions and three books.   

Books are an increasingly pointless solution to the problem of knowledge about technical systems.  They are increasingly difficult to publish, useless to update, the quality of publications is falling and the trust that they contain the knowledge you are seeking is falling.  Technical publishers have done this to themselves, but its an emergent phenomenon of the speed of change vs the latency of publishing.  As such, books seem less and less useful.

I think we have passed the point where a static book is the best technology we have for communication of knowledge.  I think this inflection point occurred about 7-10 years ago.  Probably about the time I stopped buying technical books seriously.  I certainly purchased a lot after that... but it was a tapering off period that I didn't yet understand.  Now I look at the prospect of buying and using a book with distaste.  I'm seriously considering disposing of my collection of technical books because of how little use they are to me.  Everything is dated, many are out of date and none are searchable or accessible in the way I currently seek knowledge.   

Am I sad about this?  Perhaps.  Certainly a bit nostalgic,  but the replacements for books are just so much better.  Online, dynamic knowledge bases are simply better for most of the day to day question and answer stuff.  The one thing they usually fail at is what I call, curated knowledge. 

This is very much the core business of academics.  The collection, organisation, dissemination and curation of knowledge about a specific topic.  Where this is done well using accessible digital tools, its brilliant.  The ability to access, update, search, cross reference, annotate and version knowledge in a good system like a wiki or a well structured knowledge base is brilliant.  But its still a massive job. 

I have helped build a number of knowledge bases.  None of them has been simple and none have been financially rewarding.  They are run on passion.  Wikipedia has my respect but I also see its flaws. Good writers can't eat passion. 

I think the problem winds back to the lack of viable mechanisms to identify and reward high quality documentation as part of open source (or any source) tooling. 

I see that FireGiant has started trying to sell a companion product to the WiX toolset. Will this help with the documentation? Kind of burns when you're used to the "free" access model... but the reality is that free is a pretty expensive service to provide.

I know this has been said any number of times before... but there has to be a better way.

Having stated the problem...again.  What's the possible solutions? 

Patreon? Kickstarter? Micropayments? Freemium? Ad-supported? Slave writers?  All of the above?

Everyone seems to be re-inventing the same wheel for similar problems. 

Automation in Software development tools

Just reading and kicking around ideas about the impact of AI on software development.

The ideas I am synthesising are:

An article on creating occular prosthesis I saw on tv this morning,

This article on Brute force Proofs for Math problems:
https://motherboard.vice.com/en_us/article/padnvm/200-terabyte-proof-demonstrates-the-potential-of-brute-force-math

And an article on programmers as craftsmen that I read somewhere back in the past:
Might have been this one: http://manifesto.softwarecraftsmanship.org/


I think that there is a cusp point at which an industry transitions from human skill powered to automation.  This could be pure mechanical automation/reproduction or in future AI driven systems.  This has happened to any and all manufacturing over the past century.  Start with Armoury practice or the "American Method" in weapons manufacturing.  This was a simple move from craftsman driven industry to an industry built around a component based design where each component was produced by a specialist.  Then the specialist was replaced by a cheaper duplication process.  This could involve humans, but it was reduced to a series of steps that could be done on machines.   The craftsman who once had to know all the different aspects of the production process for a weapon, then faded into irrelevance.  (Until the process was re-discovered and turned into a you-tube series)

Why did this happen?  Economics?  Yes, the production line is more "efficient" at producing a volume of similar items.  There are also emergent phenomena that this production model created that were not possible in the craftsman model.

The invention of the production line model allowed many people to be part of the production that were not previously able to be a "one stop shop" craftsman.  People could be good at woodwork but not metal work.  They could be specialist at creating screws or making barrels.  They did not have to be masters of making barrels, triggers, stocks etc.  By decomposing the whole item into components, it allows more people to specialist.  Some of these specialties were boring and repetitive and would not be "enough" to keep a person who was a "total" craftsman interested, but it opened up an opportunity for people with varied ability and shorter educations who could then be part of a production line for high quality items. It also allows deeper "specialization" to occur.  A person who spends all day producing one type of thing, has the opportunity to get very good at it. If they produce many similar things, such as screws or triggers,  they can understand them at a level that is not availible to a craftsman who is trying to be good at all the areas of the design and the skills and tools required to produce it.  I call this the "specialization limit".

The second emergent phenomena is the scale of the item design.  There are only so many skills, techniques and materials that any one craftsman could learn and invest in.  This is what I am calling the "scale limits".  But a production line can bring the skills of many specialists to a much larger design.  Imagine trying to build a jet engine using the craftsman method.  One person learns the skills required to operate a huge machine shop and produces all the parts and components to construct one massive engine.  Its possible but it would be a singularly unique person who would pursue the education, experience and focus required to craft this kind of thing with the precision required. 

There are lots of other phenomena, such as replacement of people on the production line.  Even though we do not like the thought of being replaced,  its easier to replace and train a person on a single component production, than it is to replace a craftsman who is doing a whole bunch of different skills and processes.  This is often the work of a lifetime.

One more phenomenon that I want to make a point of:  The phenomenon of the product complexity growing beyond one persons capacity to either design or make.  If you look at the growth of complexity from a craftsman made flintlock rifle, through to something like the M1 Garand, I think the craftsman who made the flintlock could have looked at the Garand and "seen" how it worked and given some time and modern tools, probably made a reasonable copy.... but there is a point where the evolution complexity of a family of products exceeds most craftspeople.  Weapons are probably not a good subject to try to make this point as the complexity has not started to grow exponentially.  The most complex weapons I can think of off the top of my head could probably still be taken apart and rebuilt by a very competent modern fitter/machinist/electrical engineer... basically because they still do all the repair and service on these systems.  Perhaps I should call this the "serviceability limit" (since I'm making shit up...).   The point being, that without the servicability limit, the complexity of a system can easily grow to exceed any one persons or teams capacity.  Look at operating systems if you want an example.  They have evolved way beyond the capacity of anyone to service them.

With modern tooling and processes, there are less and less roles for humans in the production line.  Only in factories where they are doing small run or are too poor to afford robots do we still find people doing lots of the roles in production.  We currently have the robotic technology to replace just about everyone in a production line, however, there are still a range of "hard" bits that no one has bothered to automate.  If you look at a current generation vehicle assembly plant (watch any of the "mega-factory" type documentaries,  any of them could be fully automated.  However, its currently cheaper to use humans to do the "hard" bits than it is to finally automate the whole thing.  They main role for humans is still the "creative" bit.  Where the product is designed and problems (created by the human users, service agents and consumers of the product) are solved.  The funny thing is that most of these creative solutions are pretty common and could be automated in part or in full already.  Think about a car assembly plant.  The design of a car is not that tricky.  They are all essentially variations on a theme.  The main variance is the "problem" that they are solving.  "I need a small car to run around town and take the kids to school".  Not that hard to solve apparently.  The whole design could be automated with enough effort, and we could produce thousands of variants of the "small car" design.  (With 3D printing of components this has got to be even closer. I'm waiting for the kiosk where I can order a custom car and it will be delivered from some mega-factory that is essentially just a giant car printer and assembler robot. )

The last phenomena is what I call the "forgetting of skills".  Once a skill set no longer has a commercial purpose, it becomes a "hobby" at best or is simply forgotten by the majority of people.  When I went to school, woodwork and metalwork were still part of the curriculum.  Even though there were virtually no manufacturing jobs in the town that would have employed me with them.  There is still carpentry in building houses and lots of small fabrication shops doing repairs for all sorts of equipment... but the reality is that these are more foundational skills or hobby skills.  I really enjoy wood and metal work but I respect that they are less and less involved in the economy around me.  I have to face the fact that I should not encourage my children to study these subjects at school as they are probably irrelevant to them being able to survive.  My point is, once a skill set, such as crafting a flintlock rifle is no longer economically valuable,  everyone moves on, and the unique set of skills and knowledge that was previously encoded in a single group of craftspeople dies out. 


Anyway, to bring this back to my point....

Software development easily falls into the craftsman model at the moment.  Even on the big teams where they have compartmentalized design, programming, testing, deployment etc... its only the first step away from "one programmer shops" producing application packages. (Yes, I 'm a one programmer shop and I'm in the dark ages.... and often feel far from being a craftsman... different rant)

My point is, that there are components (clearly I use SDKs and UI controls and frameworks etc) that mean I am not re-inventing the wheel, but each of them has been hand crafted by one or more people.  There is very little that could be described as automated in software development.  I have used a couple of code generators (shout out to XSD xml parser generator) that are great at doing one thing well.  These are the future.   Reminds me of an automated UI generator that turned up in a news feed a month or two ago.  (Can't remember the reference at the moment)  but that is the way of the future. I think we are the last generation of craftspeople programmers.

I think there will be a point not too far in the future where round trip design/programming/compiling/services systems take over software production.  Humans will cease to be up to their elbows in every line of code and we will at least be able to describe the interface for a module and any side effects and not only have it written auto magically, but it proven to be correct.  This may include a suite of unit tests and any other tests, but these are human artifacts to test for human errors.  I think the tests will move to a higher level of abstraction to prove that the design layer is robust, as that is where the creativity will remain for a while.  Once the AI is suitable to replace the creative limit of the humans, then the whole process will be automated and software will be a dead art from the olden-days.

So, lets recap:

The "Specialization limit", the "serviceability limit" and the "scale limit".  These are all human limits imposed on software systems.

I would argue that the "servicability limit" is very loose in software, as a great deal of it is both blackboxed and so poorly debugged that its hard to argue that any of it is servicable. (Even the stuff that I try to write) Once its compiled and running in the wild on operating systems that have been updated beyond what it was tested on... all bets are off.  There is virtually nothing the user can do to "service" the item.  It either works or it doesn't.

My ability to service it is still viable but getting harder with the proliferation of platforms.  I am working on an old C++ application at the moment.  It has fairly clearly defined platform targets that evolve relatively slowly (I say relatively... whole other rant) in comparison with a javascript app that I was working on previously that is just a madhouse of platform variations and patches and just...

At a certain point, it exceeds any one persons ability to service this kind of product.  I have a fighting chance as the original designer and builder, but for many legacy systems, the complexity rapidly exceeds the skills of anyone who inherits the system.  Keeping up to date with platform movements and maintaining a heritage skill set and knowledge base to understand legacy codebases gets pretty ugly.  Just in my own work, I am now looking backward over more than a decade of development using a slew of languages, frameworks and sdks, along with a whole pile of bad design choices and trying to decide if its worth trying to modernise the program. (not that it had a large user base but it represents a big chunk of my life that I am not sure I want to say goodbye to yet...)

The "specialisation limit" is also starting to kick in.  I actively work in a range of languages across a few platforms and there is a point beyond which I cannot work in any more.  Trying to pick up something that I used to be good at and get back into it, takes some work.  I notice the cost.  Maybe it would have been easier when I was young and my head was empty, but its filled with all sorts of knowledge that is no longer relevant for all sorts of systems that no one cares about anymore.  (I think I need to burn my bookshelf.... )
Watching the library throw out books that I have read is a pretty shocking thing.... especially when you look inside the cover and see that it was checked out exactly 5 times in its lifespan.  But the point is, these skill sets are dead.  No one cares anymore.  They are not economically valuable and my children will never learn them.

The "scale limit" is an interesting issue with software, in that without automated tools, there is a point beyond which we cannot go with a codebase.  Its just too complex to comprehend in its entirety.  (This is without taking into account the hidden complexity of all the SDK's and the platform code that its running on...) Some of the round trip type tools allow you to deal with larger amounts of complexity, but at a certain point, it will exceed your limits, simply because we all have limits.  Automated systems are theoretically limited but the reality is that they can run with much larger capacity than any single human... and at that point we are irrelevant. We will no longer be ecconomically viable.

Programing will be a hobby at best. Our children will order software from automated kiosks that present customized packages assembled by software robots. The company with the best robots will win.

It's interesting to look at the evolution of machine tools along with the evolution of the production line.  Mechanical tools, Electric tools, CNC tools, lazer tools, 3D printers, design software, welding robots etc.

Looking at a woodworking shop (cause I can watch youtube..) you can contrast hand tools, planes, chisels, saws etc. with power tools, power plane, thicknesses, joiner, mortiser, biscuit joiner, table saw etc. The advantage of the power tools is speed and reproduce-ability.  The material is the same and the construction is the same, just faster.
Once you move up to CNC driven cutters and routers or 3D printers, you are no longer working the same material or construction methods.  Essentially, the materials have had to evolve to suit the tools, and the products that are produced are no longer the same.  Take a wooden box made with traditional techniques vs a wooden box made of MDF cut out on a CNC router, assembled by a robot and finished in a spray booth.  The first has been crafted from peices of natural wood, with grain and form.  Finished in a pleasing way by the maker.  If the maker used hand tools, it may be a little uneven or not, the difference between handtools used with care and power tools is not that great. The difference between that product and the one produced by the cutting robot however is more distinct.

Both are functionally similar, in that they are a box, but the second is really defined by the process and materials. This is because the production line for robotic wooded products is still only in about the second or third generation.  The next generation will be robotics that works with natural materials and reproduces a wooden box that appears to be made by a craftsman.  Initially there will be all sorts of cheats to make it easier for the robot to handle the components and to cut the joints... but these are not hard to overcome.  It's quite easy to visualize a production line of robots manufacturing wooden boxes that are not easy to differentiate from one made by a human craftsman.

All these tools are working in the production space.  None have yet moved into the "solving a problem" meta space.  A robot has not yet been developed that can identify the need for a wooden box, build it (or commission it) and supply it/install it and then finally service it through its functional life.  This would be a vertically integrated system. 

If we looked at software developers tools... its hard to see that we are really that far along.

Text editors are pretty nice, but once you start to list all the "power tools" that can be used, the list is quiet short and is only implemented in a few languages.

• Indenting / formatting tools - pretty print the file, de-mimify
• Spell checkers - syntax checkers and variable/function name checkers
• Cross referencing tools - quickly jump between files, declarations, definitions etc
• Re-factoring tools - lots

In the rest of the toolchain, we have testing tools, compliers, linkers, packaging macros, build tools, installer builders etc. Let's not belabor the point... 

Every one of these is still just duplicating what you could do by hand.  (Yeah, I realize that compiling a program by hand is beyond me...) but that's the point.  These tools are simply doing what is humanly possible... faster.  Essentially, "powertools" in the above woodwork analogy. None of them is doing anything not humanly possible... the materials are still the same.  There is little integration between the systems (except the compiler and linker... again not the point)

I think the next step will be when the materials are modified to be more machine friendly than human friendly.  Looking at all the "intermediate code" and java bytecode that is around, you can see the process is clearly in process.  While we could kind of work with bytecode, its not economically viable.  There is little point in trying and so the skills are already being forgotten (or not taught to the next generation).  I find it hard to imagine that I would encourage my kids to learn assembler.  It's just not a good use of their time.

When the software design tools do not bother to produce human readable code, even as an intermediate step, programmers are irrelevant.  When the design tools take a rough problem description, then software architects are irrelevant.  When bugs are automatically removed, then software serviceability will be solved.

I think that this will happen when we have a high enough level language to describe a problem and its boundaries and a compiler/software robot can translate that into a solution and customize it to our desire.

With the current crop of AI, I think the question is whether this will be a human readable language or simply a pattern recognition neural net that can interpret our vague problem definitions and produce a software solution on demand.  At which point, there will be little point in evolving operating systems by human hands anymore.  Machines do not need any of the graphical user interface crap that has bloated most of our OS's. 

Anyway, the vertical integration process will continue.

It's interesting that consumer goods like the above "wooden box" example are still so far from being vertically integrated and human free.

There is no robot or chain of robots that can harvest a wild grown tree, mill it, dry it, cut, fit and assemble it from real wood into a box and deliver it to me.  Note, wild grown tree and real wood.

There are sections of that production chain that exist and have been integrated, but they are still little autonomous units that are struggling to connect. 

There are a few more production lines that are able to consume wood chips, turn them into MDF/plywood etc. and cut, assemble and package for delivery, ship it and deliver it... but sill not even close to vertically integrated if you look at the "needs" end of the chain.

For me vertically integrated would be to start with a design (or brain fart) from me in a natural language including the type of wood, figure, finish etc (or even have that automagically recommended (not hard based on my preferences)) all the customization with my name, personal carving styles, size, inserts etc, go out, source the materials (cut on demand or stockpiled by robots) manufacturing the box, pack it, ship it to me and present it in a useful time frame. 

Bringing all that together will be the work of the next few generations.  I can see it happening with engineered materials in the production section of the chain.  But the problem definition is still rudimentary with our current natural language processing, the automated design should be pretty straght forward, but isnt yet.  The packaging, shipping and delivery is still at the whim of retailers who are struggling to produce a pair of customised pants.  Once you introduce the additional problems of non-engineered materials, its not hugely more complex, but the robotics need to be a lot better.

It would be awesome to see a tree farm managed by robots.  Maybe not "managed" but certainly operated.... I predict in the next 20 years. 

Having finished this brain fart I then read... https://www.microsoft.com/en-us/research/blog/program-repairs-programs-achieve-78-3-percent-precision-automated-program-repair/