TokenMakerMaker finally in Subversion

The “TokenMakerMaker” application that I blogged about oh-so-long-ago finally has a home in Subversion, in the parent RSyntaxTextArea repository.  I’ve been getting some more questions about it lately so I figured it might force me to give it a little more attention if it were “officially” public.  You can check it out here:

http://svn.fifesoft.com/svn/RSyntaxTextArea/TokenMakerMaker/trunk

The usual caveats:

  1. I haven’t really worked on it since I last blogged about it, so it may have bugs.  It definitely has bugs dealing with the “multiline” string option.
  2. The code isn’t pretty.
  3. There will be no “official” releases for awhile, unlike RSyntaxTextArea, AutoComplete and SpellChecker.  If you want it, you’ll have to grab it from SVN.
  4. Unlike all other RSyntaxTextArea-based projects, TokenMakerMaker requires Java 6 to run.  Since TMM is a developer tool and not a library, I don’t really see this as an issue, especially since even Java 5 has reached EOL.

While the project itself is open-source (probably LGPL, but I haven’t decided on a license just yet), any TokenMakers you generate from it are yours to keep – you can license them however you wish.  Do keep in mind however, that TokenMakers are only usable via RSyntaxTextArea, which is LGPL.

The TokenMakerMaker repository contains an Eclipse project.  To get it building error-free, you’ll also need to check out the RSyntaxTextArea project, and add it as a build dependency to your TMM project.  This is because TMM always uses the very latest bits of RSyntaxTextArea to keep them in-sync.

With both of the projects checked out, you can run TokenMakerMaker directly from Eclipse by launching org.fife.tmm.Main as a standard Java application.

Alternatively, you can build TokenMakerMaker, and launch it from the resulting executable jar.  To do so, you’ll need to first run the main target in the RSTA build.xml Ant script (“make-jar”).  This creates dist/rsyntaxtextarea.jar in the RSyntaxTextArea project, which is used by TokenMakerMaker.  Then you need to run the main target in the TMM build.xml script, which builds TokenMakerMaker.  The result is a stand-alone TokenMakerMaker application in TokenMakerMaker/dist.  You can run this application as follows:

cd rsta-projects-root/TokenMakerMaker/dist
java -jar tmm.jar

Here are some basic notes about using the UI:

  • You can save and load your progress via Ctrl+O/Ctrl+S.  Your TokenMaker spec. is saved in an XML file.  This way you can come back and work on it later if you want, without having to dig into the generated flex.
  • Once the app starts, the first thing you’ll want to do is go to File -> Options.  In the “General” panel you can specify a “working directory.”  This is the location that generated *.flex, and their corresponding *.java files, will be placed.  Change this to wherever you want.  I believe old files with the same name will be overwritten without prompting you if it’s okay, so be warned!  =)
  • On the “General” tab, the difference between “AbstractJFlexTokenMaker” and “AbstractJFlexCTokenMaker” is that the latter will cause your TokenMaker to auto-indent after lines ending in ‘{‘, as well as auto-align closing ‘}’ chars when they are typed (assuming auto-indent is enabled in the RSyntaxTextArea it’s running in).  It’s supposed to be the option you want to pick if the language you are creating derives syntax from C, and uses curly braces to denote code blocks.
  • The app is supposed to be smart, and if you don’t enter a value for a field that is required, you shouldn’t be able to generate anything without an error prompt.  But there may be issues I haven’t discovered yet.
  • TMM assumes that the Java install running it is a JDK.  If it isn’t, TMM will still generate the .flex and .java source files for you, but it will not be able to compile the resulting .java file and allow you to test it in-proecess, which is one of its really cool features.  For this reason, I’d suggest running it with a JDK as opposed to a JRE.

Hopefully this is enough to get people started.  If you have any questions or issues, feel free to add a comment to this blog entry, or better yet, use the forums.

Comments are closed.