http://pastie.org/private/evhyiniur0li6oqnaroa

I know full well that if one is to write a compiler or script interpreter, or any application that needs to process text, it is best to use an already crafted lexer/parsing technology because it is the most convenient and optimal. My project, however, does not have this purpose; for the sheer fun of implementing my own lexical analyzer, and it's later use as my own despite the benefits of a ready-made technology, I am trying to implement a dynamic lexer, that is, a lexer whose states, conditions, and actions are generated a runtime from given keywords, symbols, and general token types.

So far I have had the most success implementing the state machine itself and its components. I have also succeeded in implementing simple classes that represent the structure of a system of states and transitions that facilitate generating the actual objects aforementioned. My problem is not with the most basic layer, the state machine, its states, conditions, and actions, nor with the secondary layer, the tree structure that then generates them, but with another layer that is responsible for taking keywords and generating paths of such nodes, which end before symbols and spaces and the end of the text, and which enable the state machine to distinguish between predetermined keywords and just "identifiers".

In "LexerTreeBuilder.cs" I have a class with a method that helps me generate these keyword node paths, but when using it I construct a state machine with a semicolon symbol and three keywords, "folk, folly and app", varying tokenizing behavior results from it. When I type in "folk" by itself, it does not recognize folk as the keyword "folk" but rather as an identifier. When I put a semicolon or space after it, however, it does recognize it as the keyword. Can anyone help me with this. I know the whole project is a lot of code, but it should be very easy for you as opposed to me, because even though I have programmed here and there for years, and I am still learning, and my mind is not as advanced as most experienced programmers.

So thank you, and I just ask that someone please help with this. It's been frustrating me for a while now.