“TokenMakerMaker” – Easy creation of custom syntax highlighters for RSTA

I’ve gotten several requests asking for the ability to add syntax highlighting for custom languages to RSyntaxTextArea.  Because of the way it is designed, RSTA doesn’t lend itself well to this type of thing.  Creating a highlighter that just read through a properties file (or XML) and highlighted stuff via regular expressions would probably not perform well enough for most people.  But the learning curve for how to hack at JFlex-generated output so that RSTA is happy with it is also rather high.

So I’m working on a new tool, called, for lack of a better name, “TokenMakerMaker” (name suggestions are welcome!).  RSTA has uses a TokenMaker class to know how to highlight for a given language, such as Perl or Java.  TokenMakerMaker is a GUI program that allows you to enter the specifications for your language – keywords, comment delimiters, data types, etc.  It can take that input and generate the JFlex input file for your TokenMaker, as well as the actual Java source for your TokenMaker (it’s hooked into JFlex).  Further, if you’re running TokenMakerMaker with a JDK, as opposed to a JRE, it can compile the Java source generated and allow you to test it out in an RSyntaxTextArea instance, at design time!

This tool is nice because the generated scanners will be extremely fast (optimized by JFlex).  In fact, they should be every bit as fast as the scanners that come built into RSTA.  The key will be making the UI give as many options as possible so that what a generated scanner can do isn’t too rigid.

To test it out, I’m using it to generate a (simplified) TokenMaker for Java.  How is it looking so far?  Well…

Here’s a screenshot of the panel allowing you to define the comment delimiters:

Defining the comment delimiters for Java in TMM

Defining the comment delimiters for Java in TMM

And the panel allowing you to define the data types:

Defining the data types for Java

Defining the data types for Java

And here’s the testing window you get after clicking “Generate”:

Testing our generated Java TokenMaker scanner

Testing our generated Java TokenMaker scanner

It’s still early on but I hope to have a few people test it out soon.  Here’s what’s left to be done:

  1. Flesh out the UI
  2. Add panel for defining standard library functions
  3. Add more configurability for Strings (escapes allowed or not, etc.)
  4. Allow saving TokenMaker definition to XML.  Create batch tool to generate .flex/.java directly from XML to facilitate automation of the process

If you’d like to try this tool out early and provide feedback, let me know!

5 Responses to ““TokenMakerMaker” – Easy creation of custom syntax highlighters for RSTA”

  1. James says:

    I’d definitely like to try this out and provide feedback!

  2. ryan says:

    Been using your library for a bit now and I love the simplicity to implement it. I am planning on upgrading my app to take advantage of autocomplete and regex find replace and would love to try out “TokenmakerMaker” at the same time and give you some feedback.

  3. [...] Fifesoft Blog Another boring blog about software « “TokenMakerMaker” – Easy creation of custom syntax highlighters for RSTA [...]

  4. [...] development has unfortunately come to a crawl, a first beta of TokenMakerMaker is available for testing.  Contact me if you’re interested in testing it out.  It will be a [...]

  5. [...] “TokenMakerMaker” application that I blogged about oh-so-long-ago finally has a home in Subversion, in the [...]