Sloppy Code Article

http://journal.stuffwithstuff.com/2010/11/26/the-biology-of-sloppy-code/

I just read an article about Sloppy Code. The seed idea is not the gem here its the explanation of how to fit it into the mindset of "programmers" and all the issues surrounding the evolution of both the craft and the environment in which we are all working. I found the article very deeply resonated with a bunch of half formed ideas that have been slowly orbiting my conscious and unconscious mind for some time now. This article not only articulated it, but did it with grace and clarity. The linkage with the abstraction levels among the sciences was wonderfully illustrative. It just resonated.

The best aspect was the optimistic spin. I have been reading articles about change in various industries and environments recently and the common thread has been the fear and uncertainty communicated by the authors which ended up with a common negative taint being attached to change in any form. ( I understand that change generally means loss and dispossession by many... so its fair... but still there have to some who see the upside)
Anyway, I found this article strangely uplifting. It has a lot of parallels with what I find myself doing more and more. While I still occasionally get a job that I can break out C++ and hack against some low level library from Apache or Boost, more often I am writing loose VBA or scripts to drive high level objects or automate whole executable through some high level API. This gets stuff done and is often quite satisfying to get it done quickly, but it lacks some of the fundamental satisfaction of having constructed it from elemental primitives.

I guess thats why I still get so much satisfaction from going back to raw materials in the shed. I would rather build a lathe out of fundamental components, weld them together, cut and shape and slowly assemble them rather than buy one and use it.  But on the other hand I have enough experience with turn-key packages that I also enjoy getting something that "just works" and getting stuff done with it. Different levels of abstraction.

The next major building block I am wrestling with is AI. There are enough low level libraries around to build various simple constructs but you can still see the bare metal through the library. The question becomes whether to use a library that has rough edges or to build it yourself. There is not a strong enough value proposition to use them as higher level black boxes and glue something on top because they are not really high level. They are still just first generation collections of tools and routines.

I want a library that I can instantiate a functional AI from in a single line of code, be it a Neural Net or Agent game actor or some other variant that is already done. I can then just build the rest of the experiment rather than having to go back to almost bare metal and make all the decisions and construct it slowly.

Now I think about it... I guess I am moving further away from the metal in a number of threads. The attraction of building another 300KLOC program just to get something high enough to run a couple of stepper motors as an abstract unit within its own work envelope is just depressing. Maybe its just fatigue. Having re-invented the wheel a few times and worked with so many packages that have done the same thing, over and over again, I am just tired. There is a certain point at which the idea of inventing the same wheel in yet another immature language becomes down right depressing. Trying to map the concepts that I have spent countless hours of bloody minded effort learning onto a simpler faster way of doing it.... almost seems a step backward. The time spent lerning basic, Pascal, VB, Assembly, then C code and learning C++, OOP, Managed Code, VBA, Perl, Python, Lua, RegEx, various libraries and windowing toolkits, Generic Programming, Functional Programming, Logic Programming, Scripting Languages, Embedded Languages, all the IDE's, debuggers, profilers, Static Analysis tools, patterns, refactoring, Testing Frameworks, Graphics Libraries, Game Engines, Encryption Libraries, AI Engines, Physics Engines, and now GPU languages, Network stacks, Databases, Servers, OS's, Parallel Programming, Threads, Memory Models....  the hours and hours spent looking for solutions that you know must be there but will not turn up in a search no matter how you rack your brain to describe it.... all fading into irrelevance.... Now I can barely perceive the metal through the layers of code. Working in VBA over the objects in Office is a totally different model. So much of what you know is useless. You can do it the easy "Office" way or you can try to torture the system by mapping your own ideas over it and quickly find the limitations of just what is possible. Its not really OO, its not really any of the techniques you may have known, its impure, its unpleasant, its still possible to have control over some things but not an even level of control, you can still manage lifetimes but not easily.... in the back of your mind the darkness grows and you start doing things the "Office" way... and then the .NET way... and before you know it you are on the slippery slope to being a competent Access Developer. No longer battling against the restrictions but hacking fast and loose and getting stuff done.... not worrying about creating a Q&D object to encapsulate some code and putting dirty switch logic in that you would be embarrased to write in C++. It just works. Its not something that will come back to haunt you because it will get replaced in the next round of refactoring and massaging. In fact, adding the overheads of "Quality" just makes the code that little bit more rigid and expensive to change when it needs to. My feel is that the value proposition has been reconfigured with the lighter more dynamic systems that combine  high order abstractions with loose glue languages. There is much less value in building comprehensive code that is robust and complete because the very nature of these systems is fluid. The quality has been pushed from the code you write into the objects you trust. We are delegating at a much higher level. Not only are you delegating responsibility for functionality but also high level error handling and self management.

Suddenly COM has come into itself. Web API's are next. The objects are not only a black box, they are a black wall. With a tiny window in it. You can talk through that window and the rest totally and I mean totally takes care of itself. This promotes the ideas of loose coupling in a way that I had not fully appreciated before. Having some sort of abstract communication mechanism between your glue code ( or Sloppy Code) and the API of a service provider forces you to keep it at such a long arms length that many of your assumptions are obviously violated in ways that force you to stop making them. Communicating via JSON or XML or some non native intermediate language stops you depending on things that are too easy to depend on when you are passing longs across to an API of a dll that you have loaded by hand into memory. There is so much less and more to trust in the relationship. The illusion of control has been stripped away that little bit more and you need to be a little more accepting of how little you know or can depend on in the exchange.

I think pulling data from a web service in a stateless transaction is a cleansing experience. Much the same as I once heard John Carmack quoted as saying that working on driver code is good for a programmers soul; so to is working through a flimsy API via a crappy intermediate language with a service hosted on a computer who knows where running an unknown OS, maintained by someone across a public network of wildly fluctuating service availability and with no illusion of control. Its humbling, frustrating and simple while being ugly and primal at the same time.

The idea that you can profile the service and get into the guts of the code and fix stuff... just goes away. You need to be more accepting of your limits and realistic about choices that you can make and the cost of making them. Because the choices are real simple. You can't wait for an updated version of the library or compile one from source. You can't look for alternative options or roll your own... the whole system has become about the data in the system rather than the framework of code through which it flows no matter how the abstractions within the code facilitate reuse or change... its just crap to either work or get out of the way.

Moving on.... I want high level API's (essentially an AI that I can talk to in natural language that will then go and do stuff in an organic way) and I want it now ... end of post.