Unit Testing to find bugs???

http://architects.dzone.com/articles/unit-tests-dont-find-bugs

I just read the above article and a couple of the comments.  The author seems to think that Unit Tests have something to do with finding bugs.

What bugs? Unit testing cannot "find" bugs.  A developer can use Unit tests to detect violations of expected behaviour (assuming those unit tests exist).  Bugs are conceptual issues that may exist in the developesr expected functionality, the users expected functionality, unexpected functionality, future functionality... etc.  

1) Unit Tests can be used to verify that a piece of code "does what it is designed to do".  I.e, that it has the expected behaviour or functionality as the designer understands it.

2) Unit Tests can't verify that code "does other things that it was not designed to do" as this is an open ended list.  Simple design, encapsulation and composition are better tools to manage this kind of complexity.

3) Unit Tests can be used to verify that code "does not do a specific thing that it was designed not to do" but only once the designer/developer knows enough about the undesirable behaviour to write a specific test for it.  This then becomes an instance of the first clause above.  However, it does not guarantee that this behaviour cannot occur in an unexpected way.  It can only verify that the behaviour does not occur in the way expected by the test. 

A developer can use a Unit Test to assert something explicitly about the behaviour of a unit of code and then Verify that Assertion now and in the future.

A developer cannot use Unit Tests to infer something about the behaviour of the code. (Well they could, but it will bite them sooner or later)


A Unit Test is an attempt to treat a piece of code as a deterministic machine.

Sample Data In - > Code Under Test -> Expected Behaviour Out.

This system involves no variables.  It's a simple mapping from the input to the output. It has no state except what you inflict on it. This is all a Unit Test can be.  Even if you get clever and provide an exhaustive list of sample data and run it all through the code, its still the same case.  You cannot assert anything about the behaviour of the code except under the conditions you have tested.  You can infer about the behaviour of the code under conditions you have not tested... but its guessing.  Not even educated guessing. Certainly not SWAG.  It's just WAG.

You CANNOT predict all the conditions that the code could, should, might, maybe run under, now or in the future.  You can't come close to guessing what tiny modifications might be done to the code... you can't know if that code will be transcoded to another system, language or hardware platform.... it may be compiled, cross-compiled, JIT compiled, interpretted, it might be "optimised" by hand or tool... you code can be so thoughly changed that you can not reason about it.  The language standard could evolve, your code could be compiled by better or worse tools, linted, scrubbed, re-ordered, have other code injected, be mutated at runtime, suffer a stack fault, have its memory corrupted.... malware may interfere, API's change, security software not even invented yet may completely stuff it up.  Its even possible that your code could run in a Virtual Machine or some other system that allows pause and restart.  You code could have its state saved and reloaded or modified and reloaded... or reloaded on a completely different machine in a cluster... or a different data center on a different continent..  there is just nothing that cannot be munged up given the wrong scenario. 

But in every case above, you can still assert intended behaviour and verify if the code satisifes that assertion or not.

Just for fun, imagine the worse possible scenario for the piece of code that you can, combining all the worst bits above. 

The Code is compiled to intermiary bytecode using a compiler you have never seen, run on a custom VM, split across unsyncronised virtual threads, saved and reloaded with different time clocks on different hardware, then recompiled using a JIT to machine code, is reordered to optimise it to run on a different hardware architecture, then the threads are stopped and started a couple of times just for fun.... then its all brought back together, the results are rounded, types are dynamically converted and the results passed through a couple of API's using subtly different type sizes (say, Lua binding then through a C# wrapper and back to some native C++ and out to a Python script) before being thrown against the wall, wrapped into JSON and passed to a grafted-on GUI running a mixture of Javascript, ASP and Actionscript....on an old handheld game console that has been jail-broken and has an emulator running a stripped down version of  Windows 95 a 1/10 expected clock speed and a hand patched copy of IE6 ... and enough Malware to make your eyes bleed.

Can you verify that the code does not do anything that it should not?  Can you find "Bugs" using a Unit Test in this scenario?  Can you even guess what a "bug" might be in this scenario?  The only thing you can do is test for expected functionality and verify if it exists or not. 

Welcome to Unit Testing.