Mature Coding Environment

It’s a day for bitching about programming tools again. I have just finished reading a couple of fanboy posts and articles about various programming languages/IDE's etc that are new and hot and just the secret sauce and I am tired of it.

These systems are IMMATURE. They are not only not worthy of production code but they are not worthy of hobby coding. They are to use a technical term... toys.  They are a good idea that is under development and will not be ready for productive use until they have matured.

Why do I bother making what should be a bleedingly obvious comment? Firstly, cause I am sick of the endless spew of new coding stuff and secondly because its clearly not obvious to others. Oh and I'm forcing myself to articulate it because it helps crystallize my thoughts.

So what do I mean by mature? OK. I'll state it up front and then support it rather than trying to build it via subtle and cleaver arguments.... (Never really works anyway)

*** A MATURE PROGRAMMING ENVIRONMENT HELPS THE PROGRAMMER BE EFFECTIVE ***

Yeah? And.... like how?

1. A stable language feature set

2. A state of the art IDE or language support in your favorite text editor

3. Comprehensive RICH support for the major aspects of development (develop, test, deploy, maintain)

4. Has tools to extend the programmer via automation. (Debuggers, Static Analysis, formatters, document extraction, profiling, code generators, macros, Refactoring, GUI Designers, test frameworks and runners)

5. Has high level tools for managing complexity (various file/text/logic/structure views, flowcharts, models etc)

6. Integration with a STABLE ecosystem of components (databases, media libraries, GUI systems)

7. Has a rich knowledge base with both breadth and depth on both core competencies and application to more exotic projects.

8. Has systems of support. (No man is an island... shoulders of Giants etc) Forums, discussions, communities where solutions can be found in a timely fashion.

9. Comprehensive documentation for the tools.

10. Is integrated ( or can be integrated ) into a robust workflow that can be packed up for storage and reopened when needed.

Without getting into stupid methodology arguments, I think these aspects make for an environment that gives the working programmer the best chance of getting from A to B with a project in the most effective way. I'm not talking about the self enrichment that comes from tackling something unknown or hard or only working in assembler or any other side effect of programming. I'm talking about getting a job done effectively so you can get on to the next thing/problem/job/stage whatever. I get that many people enjoy programming and fiddling and wish it would never end. I have those moments and its easy to waste time for little real progress. (Refactoring should be stored in a locked cabinet with the other dangerous drugs...)  I'm talking about getting a specific, bounded job "done" efficiently. (I may be strange in that I am rarely working on a single project at a time, usually there are half a dozen or more in the air at any time in a couple of languages with their own domain issues, so I see patterns develop.)

This then becomes a "how long is a piece of string" discussion about what "efficiently" means.  Well it means the system works, satisfies the criteria I have agreed too, will not embarrass me when its reviewed and will not be a pain in the ass when I inevitably have to come back to it and make some changes.

So what’s with the MATURE shit? Well it goes like this... a software system is an investment... it has costs to create and hopefully will return value to someone in its use, not just its creation. It has a life cycle and some sort of diminishing returns curve. It also potentially has ways to extend its value (maintenance).  If you can conceptualise of software this way, then the economics of the tool set used to create it are an important variable in the value model. Not that hard if you're used to building any financial models or doing basic cost benefit analysis.

So the initial outlay cost of the tools is a fixed amount, but the cost of using the tools is a variable amount, dependent upon both the task its being used for (difficulty) and the time the task takes.  These two will probably have an exponential relationship; simply meaning that the harder the job, the longer it takes. However time is a linear constant, so its will probably not vary too much unless you have an unlimited number of people to throw at the job...(Mythical man month anyone?)

These two variable costs are the ones that make this model suck hardest; but also identify the issues to attack for the maximum gains. (Has anyone profiled the actual activity of programming? I 've certainly tried.)

The time variable can be fiddled with a little but has a ceiling. There is only so much work you can get out of someone in a given time period and adding more people has a diminishing return... so it has some pretty hard limits from the people side of the equation. However... if the tools have an effect on time... then.... ah... you see the point... if you tools are throttling the activity of the people then you are tool bound.... better tools could potentially allow them to reach their maximum productivity.

The other variable is difficulty, which can manifest in all sorts of ways. Difficulty from the platform, from weird API's, from crappy third party support, from ugly documentation, from missing language features, from having to write and test the same boilerplate code over and over again... The point however is that within the activity of programming itself there are only two states... the first is where you are making forward progress toward the project goals(State A) and the other where you are stalled or going backward (State B).  In state B, you are burning time and resources for no gain. These situations can't always be avoided... or could they?
There can be a bit of grey between these two states but forcing myself to make a call between state A or state B can clarify the situation in my own head.

Anyway, so after a brief look at my personal economics philosophy I get back to mature programming environments and how it all relates.

The key point is that a mature programming environment has been optimised to reduce the cost in time and the multiplying effect of complexity/difficulty. This optimising never quite ends but there is a distinction between a well developed, mature environment and the solutions available for a "new" language/tool set.

The thought exercise I always use is to conceptualise a couple of projects, the first is a one-off throw away data solution for a project for a single user of maybe 1kloc, then a simple experiment package of say about 10kloc for a couple of researchers, the next is a more developed multi-part tool set for working with motion capture data of about 70kloc for both internal and external users, the last is a larger package with more history, a huge library of resources, multiple generations in service at once and large body of users of about 750kloc. I mentally try to apply a prospective tool set/language to each of these projects and see if I can imagine using that tool set productively on the projects.

Honestly, most of the languages and tool sets I see talked up fail before the first hurdle. They're not even worth considering for a tiny throw-away project. Why not?  Because their initial setup and investment in the tools is massive in comparison to the time spent on the productive project work!  It takes time to find and assemble all the bits and update to the latest builds and scrounge enough information to build a GUI and you need to hand code everything without any useful samples... etc. Why bother?  For larger projects the initial cost is much less significant, but the other issues start to come into play. How well integrated is the tool set? Will it build with a single button click, will it deploy iteratively, can I build various types of tests (unit, integration, GUI?)  Etc.  How does the tool chain scale? How does it manage complexity and extend the limits of the human brain? Can it graphically express code structures, does it support static analysis tools, are there profilers, debuggers, code formatters, documentation systems, code generators and an API for building your own tools against the exiting toolset. Is there a macro system for the tools?

These are basic features that a working programmer should expect. But so often are lacking.

As such, there are very few systems that can conceivably be described as mature. There are a lot that are moving in that direction and there are an infinite slew that is much closer to toys....
I'm just tired of the illiterate fanboy’s who get lost in the excitement of a shiny new toy without realizing that its got such a long hard way to go before its grown up enough to be a serious contender for anything.  That probably makes me seem quite dated.....

I must build a table of all the contenders one day... Wikipedia maybe....

Sloppy Code Article

http://journal.stuffwithstuff.com/2010/11/26/the-biology-of-sloppy-code/

I just read an article about Sloppy Code. The seed idea is not the gem here its the explanation of how to fit it into the mindset of "programmers" and all the issues surrounding the evolution of both the craft and the environment in which we are all working. I found the article very deeply resonated with a bunch of half formed ideas that have been slowly orbiting my conscious and unconscious mind for some time now. This article not only articulated it, but did it with grace and clarity. The linkage with the abstraction levels among the sciences was wonderfully illustrative. It just resonated.

The best aspect was the optimistic spin. I have been reading articles about change in various industries and environments recently and the common thread has been the fear and uncertainty communicated by the authors which ended up with a common negative taint being attached to change in any form. ( I understand that change generally means loss and dispossession by many... so its fair... but still there have to some who see the upside)
Anyway, I found this article strangely uplifting. It has a lot of parallels with what I find myself doing more and more. While I still occasionally get a job that I can break out C++ and hack against some low level library from Apache or Boost, more often I am writing loose VBA or scripts to drive high level objects or automate whole executable through some high level API. This gets stuff done and is often quite satisfying to get it done quickly, but it lacks some of the fundamental satisfaction of having constructed it from elemental primitives.

I guess thats why I still get so much satisfaction from going back to raw materials in the shed. I would rather build a lathe out of fundamental components, weld them together, cut and shape and slowly assemble them rather than buy one and use it.  But on the other hand I have enough experience with turn-key packages that I also enjoy getting something that "just works" and getting stuff done with it. Different levels of abstraction.

The next major building block I am wrestling with is AI. There are enough low level libraries around to build various simple constructs but you can still see the bare metal through the library. The question becomes whether to use a library that has rough edges or to build it yourself. There is not a strong enough value proposition to use them as higher level black boxes and glue something on top because they are not really high level. They are still just first generation collections of tools and routines.

I want a library that I can instantiate a functional AI from in a single line of code, be it a Neural Net or Agent game actor or some other variant that is already done. I can then just build the rest of the experiment rather than having to go back to almost bare metal and make all the decisions and construct it slowly.

Now I think about it... I guess I am moving further away from the metal in a number of threads. The attraction of building another 300KLOC program just to get something high enough to run a couple of stepper motors as an abstract unit within its own work envelope is just depressing. Maybe its just fatigue. Having re-invented the wheel a few times and worked with so many packages that have done the same thing, over and over again, I am just tired. There is a certain point at which the idea of inventing the same wheel in yet another immature language becomes down right depressing. Trying to map the concepts that I have spent countless hours of bloody minded effort learning onto a simpler faster way of doing it.... almost seems a step backward. The time spent lerning basic, Pascal, VB, Assembly, then C code and learning C++, OOP, Managed Code, VBA, Perl, Python, Lua, RegEx, various libraries and windowing toolkits, Generic Programming, Functional Programming, Logic Programming, Scripting Languages, Embedded Languages, all the IDE's, debuggers, profilers, Static Analysis tools, patterns, refactoring, Testing Frameworks, Graphics Libraries, Game Engines, Encryption Libraries, AI Engines, Physics Engines, and now GPU languages, Network stacks, Databases, Servers, OS's, Parallel Programming, Threads, Memory Models....  the hours and hours spent looking for solutions that you know must be there but will not turn up in a search no matter how you rack your brain to describe it.... all fading into irrelevance.... Now I can barely perceive the metal through the layers of code. Working in VBA over the objects in Office is a totally different model. So much of what you know is useless. You can do it the easy "Office" way or you can try to torture the system by mapping your own ideas over it and quickly find the limitations of just what is possible. Its not really OO, its not really any of the techniques you may have known, its impure, its unpleasant, its still possible to have control over some things but not an even level of control, you can still manage lifetimes but not easily.... in the back of your mind the darkness grows and you start doing things the "Office" way... and then the .NET way... and before you know it you are on the slippery slope to being a competent Access Developer. No longer battling against the restrictions but hacking fast and loose and getting stuff done.... not worrying about creating a Q&D object to encapsulate some code and putting dirty switch logic in that you would be embarrased to write in C++. It just works. Its not something that will come back to haunt you because it will get replaced in the next round of refactoring and massaging. In fact, adding the overheads of "Quality" just makes the code that little bit more rigid and expensive to change when it needs to. My feel is that the value proposition has been reconfigured with the lighter more dynamic systems that combine  high order abstractions with loose glue languages. There is much less value in building comprehensive code that is robust and complete because the very nature of these systems is fluid. The quality has been pushed from the code you write into the objects you trust. We are delegating at a much higher level. Not only are you delegating responsibility for functionality but also high level error handling and self management.

Suddenly COM has come into itself. Web API's are next. The objects are not only a black box, they are a black wall. With a tiny window in it. You can talk through that window and the rest totally and I mean totally takes care of itself. This promotes the ideas of loose coupling in a way that I had not fully appreciated before. Having some sort of abstract communication mechanism between your glue code ( or Sloppy Code) and the API of a service provider forces you to keep it at such a long arms length that many of your assumptions are obviously violated in ways that force you to stop making them. Communicating via JSON or XML or some non native intermediate language stops you depending on things that are too easy to depend on when you are passing longs across to an API of a dll that you have loaded by hand into memory. There is so much less and more to trust in the relationship. The illusion of control has been stripped away that little bit more and you need to be a little more accepting of how little you know or can depend on in the exchange.

I think pulling data from a web service in a stateless transaction is a cleansing experience. Much the same as I once heard John Carmack quoted as saying that working on driver code is good for a programmers soul; so to is working through a flimsy API via a crappy intermediate language with a service hosted on a computer who knows where running an unknown OS, maintained by someone across a public network of wildly fluctuating service availability and with no illusion of control. Its humbling, frustrating and simple while being ugly and primal at the same time.

The idea that you can profile the service and get into the guts of the code and fix stuff... just goes away. You need to be more accepting of your limits and realistic about choices that you can make and the cost of making them. Because the choices are real simple. You can't wait for an updated version of the library or compile one from source. You can't look for alternative options or roll your own... the whole system has become about the data in the system rather than the framework of code through which it flows no matter how the abstractions within the code facilitate reuse or change... its just crap to either work or get out of the way.

Moving on.... I want high level API's (essentially an AI that I can talk to in natural language that will then go and do stuff in an organic way) and I want it now ... end of post.

Natural Language Processing

http://nlpwp.org/book/chap-words.xhtml

Good resource both on Haskell and NLP.  Nothing that can't be done in another language. I feel much more comfortable mapping this problem space to C++ than to Haskell, simply because I'm more fluent. I also have a sneaking suspicion that the myths about Functional Programming are less about any intrinsic properties of the language and more about the person holding the hammer....

I have actually done some of this in VBA, which if anyone is interested, is painful due to the ugly data structures and having to build everything yourself. I know I can use .NET containers etc but its still ugly because its not my favorite hammer and its dog slow, hangs unpredictably and ... well its just dog slow(very slow dog... ignore obvious edge cases in this metaphor) I can make any tool work given enough time and energy but some just make the job harder in ways that the word "harder" does not express completely or gracefully. Think of VBA as a plastic toy hammer with an asthmatic squeaker .... C++ on the other hand is a bit like an antique 2T power hammer with a loose linkage in the return spring and a lumpy anvil but damn it can hit the problem hard....

Imposter Syndrome

http://blog.asmartbear.com/self-doubt-fraud.html

I like this post. Its well constructed, short and punchy and speaks on a topic that resonates with me (and some other people I can think of). In the contextual abstract its a beautiful post. (Although the first post in the comments is a total fanboy tag) anyhoo... as for the actual content, its slightly disturbing. I would suggest that while I identify with the Imposter Syndrome, in reality I would probably conclude that not only have I been there but I have taken the escape hatch route. I did not go to the happy place, I have gone to the safe place.

This line of thought becomes a complicated tangle of self doubt, supposed objective analysis, excuses, rationalisations and unfulfilled dreams until reality crashes in, gets dismissed as excuses, exits stage left in a huff and proceeds to play devils advocate from the wings dressed in the guise of a "Gandalf-ian" old wise man getup. 

Am I confusing the pressure of unreasonable expectations and workload with the fear of failure/discovery issues that were discussed in the post? Perhaps

Am I pushing myself to learn new things or am I cruising at a safe altitude? Nope. New things every damn day.

Am I building something bigger than myself? Nope. Its just a job that will exist long after I have gone.

Am I endlessly passionate about what I am doing? Bits of it. A great deal is politics and ephemeral bullshit... but I find value in it all. It stretches me in other ways that are not always comfortable or pleasant... so I would phrase it as: "Its a challenge every day".

On the topic of being in a safe place vs on the scary bleeding edge... I think as a parent and provider for children, the safety and security aspects trump the bleeding edge thrill stuff.  Its irresponsible to expose your children to risk and stress. Full stop. There is no "but..." arguments after this point. I can think up any number of rationalisations and guilt trips about how I'm wasting my talent etc but its not about me and I accept that I may look like a cop-out, or I'm hiding behind my children so I don't have to play with the big boys... but providing a safe and low stress environment for the children comes first, second and third otherwise I'm a failure as a parent. So while every guilt trip still lands and hurts, I suck it up and do what I know is the right thing. I find ways to enjoy a job that is not thrilling every minute of the day, I humbly accept pay that is far from the bracket that I was aiming at, I disappoint the dreams of various people who are looking for "great things" from me ( or great money...), I trade all this for a reliable pay packet, a quiet neighborhood with trees and dogs and schools and beaches where my children can grow up and learn without having to worry about all the other stuff.

Do I still dream? Is the hungry urge to beat the world at something still there? Do I still want to do amazing things and create fantastic tools that I can just about see how to do? Hell yessssss! But I take care of business first.  Walk carefully through the minefield. Make safe choices, reduce risk, manage money, balance the budget, keep the cart on the tracks, put one foot after another, ignore temptation, be the money cop, ignore opportunities, stay focused, play the long game, avoid regular fixed costs, reduce debit, stay with what you know, save for a rainy day, don't explore shady service providers(phone companies, banks, internet providers etc),  be the dull dependable safe sunscreen wearing boring person that puts food on the table, a roof over their heads and endless stimulation in their minds.

Oh and don't ride motorbikes. I miss my bike every day. Its was 300kg of big, black crazy risk taking behavior. It reminds me not only of what I have given up but inversly what I have given it up for. Better than a tattoo because the pain does not fade. (even for a huge battle cruiser, it could hit 180kmh at red line in 5th... so I hear)

Now I have the thrill of debugging spreadsheets and labeling equipment.... no comparison really.

Tenacious C IDE

http://tenaciousc.com/

This looks like an interesting product. Need to investigate more at a later date.

RESTful services model

http://martinfowler.com/articles/richardsonMaturityModel.html

This is an interesting article on the architecture of a RESTful service. I've been running into this term for a while now and have not really investigated it before. So this was a very useful read. Funny now that I've pasted the link and looked at where the article was actually hosted it make sense why it was so polished.  Shows how little I'm actually reading page headers and such....

Curating a blog

Every so often I have the need to go back and clean up old blog posts. It's, strangely enough about this time of year, most years. The research students are done and its time to clean up, round up the assets, restock the supplies in the labs, archive the tools and software, catch my thoughts, and generally look at the year in review. But back to the point....

Curating a blog. I guess I'm not having any new ideas that have not been had by others who maintained diaries or any other sort of longitudinal writing. It becomes a body of work and patterns start to emerge.
I wonder if by cleaning it up and curating it I'm actually destroying some of the historical issues that would otherwise have been interesting to me in the future. Bad spelling which was a result of starting with a blog package without spell checking (and being too lazy to manually copy/paste into a spell checker). This in itself is valuable insight into why I was blogging at the time. It was about cathartic train of though writing. Just hacking something onto the page without too much thought about why or who or what. That is being lost a little at a time as I go back over old writing, see it with current eyes and standards and "clean it up" to suit my current frame of reference. Is this a good thing?

Does anyone care or will they ever care?

In this respect paper has more authenticity. I certainly have some piles of paper with random writing on it and directories on old computers full of random writing. However its less structured. There is little sense of a timeline because so much of it has been moved around from backups and disks that its lost its original time stamps, or its actually been re-edited at some point. The other side of the coin is that the paper writing is also dateless so its not that different I guess. Mostly just random snippets of bits and pieces. Every so often I try to find the time and energy to digitize it and extract all the supposed pearls but like all piece of string projects, the value is pretty trivial compared to the work to transcribe it all.
Its not like I actually believe there are any great insights lost in it. Mostly its just childish fragments and scenarios that were meaningful to me at the time.
Midlife crises suck.

The boy is starting to get quite verbal now. He's putting good meaningful sentences together and its possible to get a bit of a conversation going. Two or three sentences anyway but he's expressing himself a little more than just frustrated screams or inarticulate noises. Every day its a bit clearer.

The girl and I have successfully built our second robot together. This one is a OWI 5DOF arm with a USB interface. It's from a kit so its not as big a challenge for her. I wanted something that would work once we has put it together rather than a home brew system that would have taken endless tweaking and frustrating debugging. She still only has an attention span of a couple of minutes at a time so we need fast results and a predictable outcome. Its been a great little project so far.
The firs robot we built was about 6 months ago. It was another kit system using Capsela components for a simple walking robot with pose-able arms. Very trivial but a good first project. We had results in about 10 minutes without any tools. Like most Capsela systems its not featured on any websites and seems to have never existed. I found it in a little toyshop an it seemed to be the only one they had.  Anyway....
Now I have to find the next challenge.