I created a new TokenMaker for my custom language (which is very similar to Java). Using the JavaTokenMaker.jflex as the starting point, I was surprised how easy it was to get a basic version going. Documentation is quite good (compared to what I've seen elsewhere...).
Here's a note for using JFlex: In the docs it is stated that the generated files are hand-massaged and trimmed. I used JFlex's skeleton feature to apply the changes in the skeleton file, so now I get the mentioned changes automatically for each build and do not have to manually change the generated file (unless there are more edits than documented). This I find very convenient and it lends itself for automatic builds.
The one thing I did not yet manage to achieve is to create a function token for functions whose name is not known in advance, i.e. user-defined functions. The virtual rule would be
- Code: Select all
i.e. an identifier followed by an opening round bracket, where the token should only comprise the identifier part, not the bracket, and parsing after detecting this should continue again at the opening bracket. Essentially, this is some kind of lookahead.
From what I see in the examples, functions are always explicitly listed, and not generically recognized. Is there some recipe on how to tackle this problem?