Folklore in software research and other Academic Misdemeanor

http://morendil.github.com/folklore.html

This is a good article to read on inquiry and critical review of publications. The fact that its about software engineering is almost irrelevant as it provides a nice summary of problems with citing non-existent references to support an argument. Should be a useful read for new research students.


http://www.johndcook.com/blog/2009/09/18/make-up-your-own-rules-of-probability/

This is a vaguely interesting article about bad stats in published papers. It alludes to both poor statistics and poor research habits.  Nothing new or particularly special here.  One of the interesting points is a comment about the Journal not correcting the mistake... Not sure what the comment is referring to but it got me thinking.

We bang on about academic misconduct and plagiarism but what about misconduct at the publishing end of the food chain? What happens when a Journal or other forum edits something or fails to correct a mistake? Or can they even correct it? Once the Journal has been distributed on dead trees its very difficult to announce a correction or recall the print run.  Obviously with online distribution the cost to change is much less. However the mechanism is still convoluted. Do you change the text with a note about the edit? Do you print a retraction in the next edition? Do you quietly update the only webpage and pretend it didn't happen? Who peer reviews the peer reviewers? Is there a "Media Watch" for Journals?

The actual substance of the article on creating your own rules of probability is I feel all too common among researchers who spend more time on their subject matter and not enough time on understanding the tools they use for their research. This is only getting worse with the explosion in software solutions and black box tools that researchers are using or trying to use to stitch together a solution to their problem without fully understanding many of the side effects embedded in the process.  This is a problem of complexity that has no easy solution except to reduce the complexity back to a level that the researcher can manage. Often that's paper and pencil.  I don't say that to be mean, its just that computers tend to add so many layers of complexity and hide so much of its that its impossible for any one person to consider them all when trying to produce a result that they are confident about. It's so easy to embed an assumption into part of a system and forget about it. It's even easier to embed assumptions that you are not even aware you hold. They are very hard to detect, let alone document or validate. But that's the game we play....

My new favorite catch phrase is "No one researches alone." This is similar to the "Standing on the shoulders of Giants"(Newton) philosophy that has been around for years although with a darker meaning.  The problem is that now we are not just standing on the shoulders of giants, we are holding hands with all sorts of crappy software, mysterious script files, graphing packages, hacked stats tools, time poor tech staff, half trained monkeys, reviewers with agendas, publishers going broke and researchers with attention split across teaching, administration, grant hunting, students, politics, ... etc etc.
Nothing you do is your work alone... for good and ill.  It's only sheer bloody-minded persistence that can get a piece of research published now. The number of obstacles and distractions that derail the publication of any science are growing exponentially.  There is also less capacity in the system to catch errors, reproduce studies and pinpoint where the errors came from.  Again, sheer bloody-minded OCD will get you there, but its hard to find people who have the right mix of personal attributes and topical interest to take on an increasingly hard process when the results and rewards seem to be diminishing every year.

We can train a student in the tools and techniques of research but I'm coming more and more to the opinion that research training is simply a culling processes. Unless you have the personality and motivation before you get here.... there's no amount of knowledge that will shape you into a researcher. 

I think its probably possible to predict researchers and hackers in primary school.  These people are born and nurtured rather than being made. There is probably a whole rant on the education system from this topic but I will avoid it.  I'm done here.

Later....