Archive for April, 2010

TokenMakerMaker finally in Subversion

Tuesday, April 20th, 2010

The “TokenMakerMaker” application that I blogged about oh-so-long-ago finally has a home in Subversion, in the parent RSyntaxTextArea repository.  I’ve been getting some more questions about it lately so I figured it might force me to give it a little more attention if it were “officially” public.  You can check it out here:

http://svn.fifesoft.com/svn/RSyntaxTextArea/TokenMakerMaker/trunk

The usual caveats:

  1. I haven’t really worked on it since I last blogged about it, so it may have bugs.  It definitely has bugs dealing with the “multiline” string option.
  2. The code isn’t pretty.
  3. There will be no “official” releases for awhile, unlike RSyntaxTextArea, AutoComplete and SpellChecker.  If you want it, you’ll have to grab it from SVN.
  4. Unlike all other RSyntaxTextArea-based projects, TokenMakerMaker requires Java 6 to run.  Since TMM is a developer tool and not a library, I don’t really see this as an issue, especially since even Java 5 has reached EOL.

While the project itself is open-source (probably LGPL, but I haven’t decided on a license just yet), any TokenMakers you generate from it are yours to keep – you can license them however you wish.  Do keep in mind however, that TokenMakers are only usable via RSyntaxTextArea, which is LGPL.

The TokenMakerMaker repository contains an Eclipse project.  To get it building error-free, you’ll also need to check out the RSyntaxTextArea project, and add it as a build dependency to your TMM project.  This is because TMM always uses the very latest bits of RSyntaxTextArea to keep them in-sync.

With both of the projects checked out, you can run TokenMakerMaker directly from Eclipse by launching org.fife.tmm.Main as a standard Java application.

Alternatively, you can build TokenMakerMaker, and launch it from the resulting executable jar.  To do so, you’ll need to first run the main target in the RSTA build.xml Ant script (“make-jar”).  This creates dist/rsyntaxtextarea.jar in the RSyntaxTextArea project, which is used by TokenMakerMaker.  Then you need to run the main target in the TMM build.xml script, which builds TokenMakerMaker.  The result is a stand-alone TokenMakerMaker application in TokenMakerMaker/dist.  You can run this application as follows:

cd rsta-projects-root/TokenMakerMaker/dist
java -jar tmm.jar

Here are some basic notes about using the UI:

  • You can save and load your progress via Ctrl+O/Ctrl+S.  Your TokenMaker spec. is saved in an XML file.  This way you can come back and work on it later if you want, without having to dig into the generated flex.
  • Once the app starts, the first thing you’ll want to do is go to File -> Options.  In the “General” panel you can specify a “working directory.”  This is the location that generated *.flex, and their corresponding *.java files, will be placed.  Change this to wherever you want.  I believe old files with the same name will be overwritten without prompting you if it’s okay, so be warned!  =)
  • On the “General” tab, the difference between “AbstractJFlexTokenMaker” and “AbstractJFlexCTokenMaker” is that the latter will cause your TokenMaker to auto-indent after lines ending in ‘{‘, as well as auto-align closing ‘}’ chars when they are typed (assuming auto-indent is enabled in the RSyntaxTextArea it’s running in).  It’s supposed to be the option you want to pick if the language you are creating derives syntax from C, and uses curly braces to denote code blocks.
  • The app is supposed to be smart, and if you don’t enter a value for a field that is required, you shouldn’t be able to generate anything without an error prompt.  But there may be issues I haven’t discovered yet.
  • TMM assumes that the Java install running it is a JDK.  If it isn’t, TMM will still generate the .flex and .java source files for you, but it will not be able to compile the resulting .java file and allow you to test it in-proecess, which is one of its really cool features.  For this reason, I’d suggest running it with a JDK as opposed to a JRE.

Hopefully this is enough to get people started.  If you have any questions or issues, feel free to add a comment to this blog entry, or better yet, use the forums.

Javadoc assistance for Java Code Completion

Sunday, April 11th, 2010

I’ve added the ability to add “source attachments” to jars on the “class path” or “build path” (not sure what to call that, Eclipse uses “build path,” but RSTA isn’t compiling anything, since it’s just an editor…).  You can now get Javadoc in the description window alongside code completion choices.  In the example below, I’m editing java/lang/String.java, and I’ve attached the src.zip that comes with the JDK.  You’ll see that the Javadoc for the relevant String method is shown:

Source Attachments working

Source Attachments working

Beyond that, I’ve been working on the API, using RText as the test application.

Also, not the source outline tree on the left.  I’m working on getting this component simple and usable as well.  I’m tempted to shove it into the RSTALanguageSupport library, since it would only be used with RSTA instances using the library, but since it isn’t specifically related to the editor, it may just end up in RText itself.  It’s really nice – it displays the outline of the source as parsed by the code completion parser, and clicking on an item in the tree jumps to that item in the source code.  However, it is Java-specific and is not currently meant to be used for outlining just any language.

Java Code Completion Status

Thursday, April 1st, 2010

The code completion library for standard programming languages for RSyntaxTextArea is almost ready for a public repository.  I plan on calling this library “RSTALanguageSupport” (nice name, huh?), as it contains support for editing common programming languages with RSyntaxTextArea.

In getting this library tidied up, I’ve picked up the Java code completion yet again, since it’s by far the most complex.  I thought I’d give a status update now so people don’t have unrealistic expectations about the library when it’s first released.

Assuming you’ve added any jars you want on your “build path” (to use Eclipse terminology), the Java code completion support can handle:

  1. Class fields and methods:

    Completion of fields and methods

    Completion of fields and methods

  2. Local variables in scope in the current method:

    Completion for local variables

    Completion for local variables

  3. Methods and static fields of non-primitive fields and local variables:

    More local variable completion

    More local variable completion

  4. Common Javadoc completions that I personally find extremely useful in Eclipse:

    Javadoc completions

    Javadoc completions

It does not currently handle completions for return types of methods:

Can't do this just yet

Can't do this just yet

This is the single issue that will take the longest to resolve, as it’ll take a re-hashing of how the code completion is parsing the code.  Other features and bug fixes will appear before this one is fixed.

Also, please note that the Java code completion should not be considered “good design.”  I am (poorly) self-taught in lexing/parsing, and so what I have now is certainly not optimized, easy to read, or using the latest design techniques.  It is what it is.  :)  I consider this to be a learning experience, and that the library will improve over time.

That said, it seems reasonably performant (only a pause when it has to initially load completions for many new classes, such as multiple imports of large packages, such as “javax.swing.*”), and is more than usable on the machines I’ve tested on.  Memory usage could certainly be improved (stuff isn’t shared between multiple editors implementing Java completion that should be) but I plan to fix that soon.

As a side note – as you can see from the first screenshot, I’m also working on tying Javadoc into the description window.  Notice that the Javadoc for the “str” field is nicely formatted in the tool tip-style side window?  That happens for all fields and methods in the current source file.  I’m also working on the ability to add “source attachments” for jars on the “classpath,” just like Eclipse, which will be used for assistance with libraries.  So for example, you can point to the src.zip included with your JDK, and get a nice description of each class, field and method in the JDK, just like in your favorite real IDE!

Integration of this library into RText as a plugin is already underway.  As always, RText will be a showcase of all the cool stuff you can do with RSyntaxTextArea.