Archive for the ‘TokenMakerMaker’ Category

RSyntaxTextArea 2.5.3 Released!

Sunday, June 29th, 2014

RSyntaxTextArea 2.5.3 was just released on GitHub!  Here’s what’s new:


  1. Improvements to PHP syntax highlighting.
  2. In HTML and XML, “mark occurrences” should match tag names even when the caret is in an attribute, not just when it is in the tag name itself.
  3. Java syntax highlighting updated for Java 8 classes, interfaces, and enums.
  4. SQL: matching opening/closing parens are now highlighted.
  5. Fixed several bugs.


  • Only minor changes to support/stay in sync with RSyntaxTextArea 2.5.3.


  • Only minor changes to support/stay in sync with RSyntaxTextArea 2.5.3.


  1. Improvements to CSS code completion.
  2. Improvements to PHP code completion.  Documentation for many functions is now included.
  3. JavaScriptOutlineTree now understands basic JavaScript OO paradigms much better.


  • Only minor changes to support/stay in sync with RSyntaxTextArea 2.5.3.


  • Added “Recent Files” sub-menu to the File menu.
  • Added an option to install a theme on the RSTA editor when previewing changes to a TokenMaker.
  • Bumped version number in the About dialog to “1.0” to match that in the documentation.



RSyntaxTextArea repositories now on GitHub, require Java 5

Sunday, August 11th, 2013

Two big things going on with RSyntaxTextArea!

First, the source for RSTA ia being moved from our personal SVN server to GitHub.  Since GitHub is the place to host open source projects these days, hopefully this new home will provide RSTA with more exposure and accessibility.  AutoComplete, SpellChecker, and the other sister libraries are moving over as well.  I haven’t quite decided yet what will happen to the current SVN server; my current idea is to keep it up and available, but to edit the readme files and Ant scripts to talk about the new location (and not actually build anything in the case of the Ant scripts, to force people to notice!).  But I can’t decide whether it’s better to just remove the repository entirely – that will force users to see it’s no longer there, causing them to go to the site and realize the source is now on Github.  While it’s a little more harsh, it will keep the project from appearing “dead” to people who simply monitor the SVN and see no updates being made.

The second big change is that, starting with the next release, RSyntaxTextArea will require Java 5.  Yes, I’m finally retiring support for Java 1.4.  If you still need to run on such an ancient JVM, you can continue to use 2.0.7 (or even fork it on Github!).  If any huge issues are found I’ll be happy to create a maintenance branch based off 2.0.7 for you Java 1.4 folks, but I seriously doubt that’ll happen.

Migrating to Java 5 doesn’t do much for the library itself, but it does modernize the code base, and fixes a couple of odd issues here and there.  You’ll find “java5” branches in each RSTA project now; that’s where all the action is.  I’m trying to embrace git’s painless branching and all.  🙂  When the work is finally done (should be in a day or two) I’ll merge these branches back into master.

Finally, I’ll be adding all known issues into the GitHub issue tracker and any other bits and pieces left remaining.  As an aside, I’ve found GitHub’s interface to be painless, fast, and intuitive.  Much better than the clunky old SourceForge interface.

TokenMakerMaker Updates

Saturday, April 14th, 2012

It’s been awhile since I’ve written about TokenMakerMaker, but it’s a great tool and it’s been getting some love lately, so I’ll talk about some of the recent change here.

TokenMakerMaker is a tool for folks who want to add support for custom languages to RSyntaxTextArea, but who don’t want to spend time learning JFlex (the most common way to add support for a new language).  TMM provides a simple GUI interface that lets you specify the makeup of your language (keywords, comment styles, etc.) and generates a TokenMaker (the class that defines syntax highlighting for a language) for you.  The best part is that it is using JFlex behind the scnes to generate your TokenMaker, so it should be every bit as fast as if you wrote a JFlex specification “by hand.”  Please see my original post on TokenMakerMaker for more information on running it.

So what’s new?  First off, number literals can now be specified in the “Numbers” tab.  You can specify whether, and in what format, integer, float, and hex literals are specified in your language:

Numbers Options

Numbers Options

Second, the TokenMaker class generated by TMM now properly registers your line comment style with the “toggle line comment” action (Ctrl+/, Cmd+/ on Macs).

TMM has also been updated for recent changes in RSyntaxTextArea’s token types.  For example, the General tab contains a check box allowing you to specify whether your language allows the boolean literals “true” and “false”, and has a new tab allowing you to specify keywords that should be highlighted as “secondary” keywords (commonly used for control flow keywords such as “return”).

Boolean Literal Highlighting

Boolean Literal Highlighting

A new option has been added to allow the customizing of the theme installed on test RSyntaxTextAreas when testing out your TokenMaker:

RSTA Theming Support

RSTA Theming Support

Finally, and perhaps most importantly, a new “Output” tab shows you a log of what’s going on when you click “Generate” to create your TokenMaker.  This will allow you to easily spot any errors that occur (errors are highlighted in red).  No more having to watch the console for this information.

Output Panel

Output Panel

Give it a shot and see what you think!  I plan to keep evolving this tool.  Likely updates will include:

  1. Verbatim strings support
  2. A way to specify tokens to highlight as variables (languages such as Perl, with easily-identifiable variables would benefit from this)
  3. Specification of keywords & functions via regex as well as plain text

I’m also actively working on the Help system in the application, to hopefully help out first-time users.


TokenMakerMaker finally in Subversion

Tuesday, April 20th, 2010

The “TokenMakerMaker” application that I blogged about oh-so-long-ago finally has a home in Subversion, in the parent RSyntaxTextArea repository.  I’ve been getting some more questions about it lately so I figured it might force me to give it a little more attention if it were “officially” public.  You can check it out here:

The usual caveats:

  1. I haven’t really worked on it since I last blogged about it, so it may have bugs.  It definitely has bugs dealing with the “multiline” string option.
  2. The code isn’t pretty.
  3. There will be no “official” releases for awhile, unlike RSyntaxTextArea, AutoComplete and SpellChecker.  If you want it, you’ll have to grab it from SVN.
  4. Unlike all other RSyntaxTextArea-based projects, TokenMakerMaker requires Java 6 to run.  Since TMM is a developer tool and not a library, I don’t really see this as an issue, especially since even Java 5 has reached EOL.

While the project itself is open-source (probably LGPL, but I haven’t decided on a license just yet), any TokenMakers you generate from it are yours to keep – you can license them however you wish.  Do keep in mind however, that TokenMakers are only usable via RSyntaxTextArea, which is LGPL.

The TokenMakerMaker repository contains an Eclipse project.  To get it building error-free, you’ll also need to check out the RSyntaxTextArea project, and add it as a build dependency to your TMM project.  This is because TMM always uses the very latest bits of RSyntaxTextArea to keep them in-sync.

With both of the projects checked out, you can run TokenMakerMaker directly from Eclipse by launching org.fife.tmm.Main as a standard Java application.

Alternatively, you can build TokenMakerMaker, and launch it from the resulting executable jar.  To do so, you’ll need to first run the main target in the RSTA build.xml Ant script (“make-jar”).  This creates dist/rsyntaxtextarea.jar in the RSyntaxTextArea project, which is used by TokenMakerMaker.  Then you need to run the main target in the TMM build.xml script, which builds TokenMakerMaker.  The result is a stand-alone TokenMakerMaker application in TokenMakerMaker/dist.  You can run this application as follows:

cd rsta-projects-root/TokenMakerMaker/dist
java -jar tmm.jar

Here are some basic notes about using the UI:

  • You can save and load your progress via Ctrl+O/Ctrl+S.  Your TokenMaker spec. is saved in an XML file.  This way you can come back and work on it later if you want, without having to dig into the generated flex.
  • Once the app starts, the first thing you’ll want to do is go to File -> Options.  In the “General” panel you can specify a “working directory.”  This is the location that generated *.flex, and their corresponding *.java files, will be placed.  Change this to wherever you want.  I believe old files with the same name will be overwritten without prompting you if it’s okay, so be warned!  =)
  • On the “General” tab, the difference between “AbstractJFlexTokenMaker” and “AbstractJFlexCTokenMaker” is that the latter will cause your TokenMaker to auto-indent after lines ending in ‘{‘, as well as auto-align closing ‘}’ chars when they are typed (assuming auto-indent is enabled in the RSyntaxTextArea it’s running in).  It’s supposed to be the option you want to pick if the language you are creating derives syntax from C, and uses curly braces to denote code blocks.
  • The app is supposed to be smart, and if you don’t enter a value for a field that is required, you shouldn’t be able to generate anything without an error prompt.  But there may be issues I haven’t discovered yet.
  • TMM assumes that the Java install running it is a JDK.  If it isn’t, TMM will still generate the .flex and .java source files for you, but it will not be able to compile the resulting .java file and allow you to test it in-proecess, which is one of its really cool features.  For this reason, I’d suggest running it with a JDK as opposed to a JRE.

Hopefully this is enough to get people started.  If you have any questions or issues, feel free to add a comment to this blog entry, or better yet, use the forums.