Pointless philosophical debate - Disengagment, Debugging and Futility

I listened to an Honours presentation a few years ago called:  "Knowing When To Quit: Self-Focused Attention Enables Disengagement, Positive Affect Predicts Reengagement Tendencies. (Tristan Hazelwood)".

It was on one of those aspects of psychology that I had never considered before.  That of "task disengagement"... or the ability to quit doing something. I found this quite personally intriguing becasue I had always seen "quitting" as a failure state and even though I have quit a lot of things in my life... it was always with this sense of personal failure.

Tristans experiment looked at how long people will continue to attend to a "futile" task before stopping doing it.  There were some manipulations as well... which are not relevant.

I had never really asked myself this question or the extension questions.

How long do I keep doing futile tasks? 
How do I know a task is futile?
What if a task seems futile but is ultimatly not?
What if the effort required to complete the task is disproportionate in comparison to the reward?

Anyway,  this gets back to my today issue:  Fixing bugs in other peoples code. 

I have just spent way too long trying to debug why the wrong favicon was being displayed in a particular browser (Firefox).  This required me to:

• Verify my assumptions about the local copy of the site files (Still broken)
• Verify my assumptions about the server copy of the site files (Still broken)
• Reload everything in Firefox
• Test in multiple other browsers
• Read up on how Firefox handles FavIcons
• Break into the sqllite db used by Firefox to cache the favicons
• Clear the cache, reload, test, clear again, clear a different way... test-fiddle cycle until I got the combination correct and finally expunged the old favicon from the chain of server-browser-cache rubbish.

In hindsight, knowing what I know now, this was a worthless task.  It was a combination of a stupid caching policy/bug in firefox and my uploading "placeholder" images before I had the final design for the favicons. The problem would not have been visible to the client because it was in my local browser cache.  But at the time all I could see was an error!

When is a debugging task "worthwhile"?

I have a long history of tenacity with debugging... and in hindsight many of the more intractable bugs now seem to be examples of time wasted and needless distractions.  Others were important and meaningful battles that I needed to fight to get projects over the line. All were bugs.... are there any clues to differentiate between valuable battles and needless distractions?

When the bug is in someone elses code... 

I think the first possible clue is where the bug lies. I think debugging is often a loosing proposition, even if you do pin it down.  I have spent way too long proving, documenting and reporting bugs in products that I use in the hope that they will be fixed.  Some have been... but I think that the majority of them resulted in a discussion with the product programmer/support minion that not only wasted more time, but resulted in them not resolving the bug in any way that resulted in me getting a fix when I needed it. (I.e very soon after identifying it)
I would suggest that while these activities were probably noble they were essentially of no value to me or the work I was trying to get done. (Except where they resulted in me clearly identifying that the bug was not in my code)

My point?  Stop debugging other peoples code where possible.  There is just not enough time in my life to fix all the broken software (and get a usable result) Once I see that the bug is not in my code... give up.  Find a work-around, hack it, abandon the tool/framework/etc... but move on.

When there is more than one way to acheive a result...

The difference between "the right way to code" and the "expeditious way to code" can sometimes be clear.... however, I find there are usually lots of ways to get to the same result.  Unfortunatly, I often find that the "recomended" way in the docs/tutorials/book/forum post/back-of-the-toilet-door is either out-of-date, incomplete, partially thought out or just plain wrong.  The time I have spent trying to get stuff working that I found on the internet... that eventually turned out to be rubbish, is painful to recount.  (This is mostly my own fault for trying to work in too many languages/toolkits/frameworks etc without taking the time to be an expert first)
I think that I need to be a bit more willing to abandon broken code a bit faster than I currently am. This often comes back to some time pressure, that I'm trying to get something working quickly rather than taking a bit more time to learn the docs first.  False ecconomy.

That being said, I would say that more than 50% of the time when I have gone to the docs for whatever I'm working on, the docs are wrong in part or whole.  So, I'm not sure if there is any actual win with reading the docs first... but it might help in some cases.   To paraphrase... go back to the source Neo...

My advice?  Don't hold onto any ideas of the right/clean/ideologically sound/best practice way of doing something.  Get it working as best you can and clean it up later if it proves to be a problem.  Future proofing is a never ending piece of string argument.... you cannot win, so don't fight the battle.

Usually I find that the next time I try to solve the same problem, I inevitably have a different approch anyway... so I tend to end up rewriting rather than fixing... but thats me.  The key point is to let go of something that is not working no matter how ideologically "right" it may be.

The curiosity trap...

Debugging to figure out how something works is a really painful way to learn.  There are probably lots of scenarios where its the only way (security work, reverse engineering malware, deep bugs in critical systems etc)  but for most of the desktop stuff I do, there is just no reason to be trying to learn that far down the levels of abstraction. It's just too slow.

My point?  I think when its curiosity driving the activity and you find yourself trolling through code... "you're doing it wrong".  Go feed your brain on wikipedia or get a pet or play Sudoku.... the urge to solve puzzles can lead programmers into some really pointless time wasting.

Neat and tidy.... 

The urge to "finish" something or make it "tidy" is a seductive beasty!  It can lead you to make beautiful things... or take you to crazy places.

I find that neatness in coding is important so you can stay on top of the project... but there is a slippery slope beyond that point where neatness for the sake of it becomes a procrastination exercise (he types on a blog....laughing ironically as the keystrokes land...)

Basically, there is no limit to how far you can take neatness in coding.... I think the best advice is to go the other way... encourage messiness to the point you cannot function... then take a tiny step back. Minimize how much extra time you need to spend on the housekeeping...

This ties in with our human pattern recognition systems so it can be a bit of a two edged sword ( triple edged swords just mess up this metaphore...so just don't...)  in that it can be very valuable but also lead to the darkside.

Neatness in the code allows us to scan repeating structures in ways that don't involve fully concious "thought".  This can be a really valuable behaviour; that personnally has spotted issues more times than I could guess.  However... the other side of this problem is that it only works with repetition.  To get maximum value, you need repetative patterns.  Once these patterns get larger... you start to run into the bad 'code smell' of  DRY.  (Don't repeat yourself).   So, I think its probably best used for "small stuff". Such as how I order operators or how I use whitespace in a statement.  The ways that brackets are laid out (this may be why code formatting is such a universal issue for code jockeys)... but once I start to see repeating blocks of statements... its time to refactor or shop for a red lightsaber. 

I think there is a bit more to think about in this area of pattern recognition in coding, but thats another post for another day.

In summation your honour... 

Estimating is always easier in hindsight.  Knowing when to let go of a task is an np hard problem.  Having written all these ideas down, its still not clear that there is any generalisable wisdom in any of it.  But, I feel a bit clearer having articulated some of the issues, which is really the point of writing it down....

Unit Testing to find bugs???

http://architects.dzone.com/articles/unit-tests-dont-find-bugs

I just read the above article and a couple of the comments.  The author seems to think that Unit Tests have something to do with finding bugs.

What bugs? Unit testing cannot "find" bugs.  A developer can use Unit tests to detect violations of expected behaviour (assuming those unit tests exist).  Bugs are conceptual issues that may exist in the developesr expected functionality, the users expected functionality, unexpected functionality, future functionality... etc.  

1) Unit Tests can be used to verify that a piece of code "does what it is designed to do".  I.e, that it has the expected behaviour or functionality as the designer understands it.

2) Unit Tests can't verify that code "does other things that it was not designed to do" as this is an open ended list.  Simple design, encapsulation and composition are better tools to manage this kind of complexity.

3) Unit Tests can be used to verify that code "does not do a specific thing that it was designed not to do" but only once the designer/developer knows enough about the undesirable behaviour to write a specific test for it.  This then becomes an instance of the first clause above.  However, it does not guarantee that this behaviour cannot occur in an unexpected way.  It can only verify that the behaviour does not occur in the way expected by the test. 

A developer can use a Unit Test to assert something explicitly about the behaviour of a unit of code and then Verify that Assertion now and in the future.

A developer cannot use Unit Tests to infer something about the behaviour of the code. (Well they could, but it will bite them sooner or later)


A Unit Test is an attempt to treat a piece of code as a deterministic machine.

Sample Data In - > Code Under Test -> Expected Behaviour Out.

This system involves no variables.  It's a simple mapping from the input to the output. It has no state except what you inflict on it. This is all a Unit Test can be.  Even if you get clever and provide an exhaustive list of sample data and run it all through the code, its still the same case.  You cannot assert anything about the behaviour of the code except under the conditions you have tested.  You can infer about the behaviour of the code under conditions you have not tested... but its guessing.  Not even educated guessing. Certainly not SWAG.  It's just WAG.

You CANNOT predict all the conditions that the code could, should, might, maybe run under, now or in the future.  You can't come close to guessing what tiny modifications might be done to the code... you can't know if that code will be transcoded to another system, language or hardware platform.... it may be compiled, cross-compiled, JIT compiled, interpretted, it might be "optimised" by hand or tool... you code can be so thoughly changed that you can not reason about it.  The language standard could evolve, your code could be compiled by better or worse tools, linted, scrubbed, re-ordered, have other code injected, be mutated at runtime, suffer a stack fault, have its memory corrupted.... malware may interfere, API's change, security software not even invented yet may completely stuff it up.  Its even possible that your code could run in a Virtual Machine or some other system that allows pause and restart.  You code could have its state saved and reloaded or modified and reloaded... or reloaded on a completely different machine in a cluster... or a different data center on a different continent..  there is just nothing that cannot be munged up given the wrong scenario. 

But in every case above, you can still assert intended behaviour and verify if the code satisifes that assertion or not.

Just for fun, imagine the worse possible scenario for the piece of code that you can, combining all the worst bits above. 

The Code is compiled to intermiary bytecode using a compiler you have never seen, run on a custom VM, split across unsyncronised virtual threads, saved and reloaded with different time clocks on different hardware, then recompiled using a JIT to machine code, is reordered to optimise it to run on a different hardware architecture, then the threads are stopped and started a couple of times just for fun.... then its all brought back together, the results are rounded, types are dynamically converted and the results passed through a couple of API's using subtly different type sizes (say, Lua binding then through a C# wrapper and back to some native C++ and out to a Python script) before being thrown against the wall, wrapped into JSON and passed to a grafted-on GUI running a mixture of Javascript, ASP and Actionscript....on an old handheld game console that has been jail-broken and has an emulator running a stripped down version of  Windows 95 a 1/10 expected clock speed and a hand patched copy of IE6 ... and enough Malware to make your eyes bleed.

Can you verify that the code does not do anything that it should not?  Can you find "Bugs" using a Unit Test in this scenario?  Can you even guess what a "bug" might be in this scenario?  The only thing you can do is test for expected functionality and verify if it exists or not. 

Welcome to Unit Testing.

Software Testing Stack

http://www.matthewbussa.com/2011/01/evolution-of-automated-testing.html

This is a good article that summarizes the software testing stack as I visualize it.  The problem I keep coming back to is that there are no turn key systems to implement this kind of stack for most developers. You can find fragments of the solution in some languages. Others have much better sets of tools, but they are still not integrated or complete.

As a developer I just want a testing suite that I can turn on, build tests in and run in various obvious ways (selectively, by group, by layer etc), I want a back end reporting system that will give both simple and rich stats, coverage metrics, etc.  And most importantly, I want the system to support my favorite languages, libraries and frameworks.

I figure it should be illegal to use a language/framework/library etc that is not able to be comprehensibly tested to using an open set of tests on a known testing framework. Things like this are just not business ready.

Seems like the .NET testing environment is maturing quickly but its still a bit of a "roll-your-own" mentality which makes portability of the tests between developers less easy.