TokenMakerMaker vs. Flex-conversion

[schulz]

Hi,

I tried to make use of the TokenMakerMaker to get a first idea of how TokenMakers look alike for RSTA.
One problem I encountered is that the TMM seems not to handle $ in all cases. I tried to define a function, which in our syntax is introduced by a $-sign, but the generation process fails because the Matcher does a special treatment for $-signs identifying RegEx groups. I tried escaping using \$, but it did not work. Escaping worked in the multi comment definition, though.
In the end, I think, TMM does not suffice for our templating language, which is non-C, non-XML. On the other hand, it is quite XML-like, as, e.g., operations have begin- and end-"tags". As we have a Flex-definition already, is there any (nice) way to convert such an existing definition to a TokenMaker?

Thanks,
Stefan

[robert]

Creating new TokenMakers has always been somewhat of a barrier of entry for folks with custom languages, because they either aren't yet familiar with Flex, or are turned off by the manual tweaking needed for RSTA. If you're familiar with Flex, then it shouldn't be hard to make the changes needed for JFlex to create a TokenMaker.

The manual steps are defined in the documentation comment for each built-in TokenMaker class, see for example XmlTokenMaker.flex.

The process is also outlined in the start of this forum thread.

The annoying thing about creating a TokenMaker is that you don't return a Token each time you identify one. Rather, since it's more efficient for RSTA to identify tokens one line at a time, your TokenMaker is given one line of text to lex at a time, and your lexer creates a linked list of Tokens to return. So you can't use a flex specification verbatim, but some mild tweaking should make it work.

Try taking a look at XmlTokenMaker.flex, if a markup language is a good starting point, and creating your own based on it. Feel free to ask questions if you have any!