Dialog Game Mechanics

http://www.escapistmagazine.com/articles/view/editorials/reviews/8898-L-A-Noire-Review

Just read a review of LA Noir and the same problem has again shown up.  How to run character-player dialog that can follow the narrative while being convincing and functional. This problem has been around since CRPG's were first cracked out of the shrink wrap.

You want to talk to a character about something... so the current state of the art is a canned dialog tree or some sort of horrible natural language parser.  Both of which suck for very real and insurmountable reasons.

The canned dialog tree is incredibly restrictive and breaks the players immersion immediately because its never a good fit for what they actually want to say/ask/scream.  The natural language parser systems I have used always turn into a game of twenty questions just to find something the character will respond to.  Hardly a "conversation"; in reality a kind of torture that I just could not find entertainment in.  

My solution would be the same as that used for the light gem in Thief-the dark project. (Being too cryptic am I? You should read my honors thesis.) Its simply an abstract representation of otherwise impossible to aquire game knowledge.  The player gets a narrative artifice that, once they suspend disbelief, allows them access to information that cannot otherwise be communicated from the third person POV.  It also provides a way to project the knowledge back to the players space.  So in essence it actually engages the player even more.  There is a great deal to learn from this simple interface design exercise.

The question about dialog is... does is have to be concrete language passing between the player and their avatar and from the Avatar to the Character.  The answer is no and yes.So how would I design an immersive dialog system suitable for a free flowing game vs one with a strong narrative? 

Firstly a free-flowing game does not have the requirements for structured, topical dialog that a narrative game has... but the point is still valid.  I feel that I have a solution for both situations.  In general though the free flowing games simply has less objective to a dialog situation so there are less ways to evaluate success or failure.  (Makes it easier to be right.)  CRPG's that allow the player to wander around and talk to anyone (Morrowind et al) turn dialog into more of a random treasure hunt. If you are on a mission then the same rules apply as in a strongly narrative game.
A strong narrative game usually has structure and intent in a dialog.  Its not that dialog tree's didn't contain information, its just that the mechanic of picking a dialog option from a list is horribly crude.

Seems like an easy problem to solve. But that's me.  Now the question is to turn it into a PhD or go commercial with it?  Decisions decisions