Own syntax schemes

[omadawn]

Sorry to wake up an old post but I have been through this thread and a number of others and it is still quite unclear how I do this.

I am looking for a concrete example on how to add additional reserved words so that they are highlighted?

I have found examples on how to create a custom token maker and I have tried that but I realize I haven't found any directions on how to make my rSyntaxTextArea use those tokens.

rSyntaxTextArea.setSyntaxEditingStyle(SyntaxConstants.SYNTAX_STYLE_TCL);

[Kyrremann]

What do you mean by adding additional words?

As far as I understand you have to make your own language, or expand the language-file for the chosen language.

[robert]

Hi omadawn,

Kyrremann's right, I never got around to adding "official" support for adding keywords to existing languages. It didn't seem like the requirement would come up very often, and it feels lopsided allowing users to easily add keywords but not providing a means to easily delete existing ones.

I could modify the existing TokenMakers to make it easier to do these things, but that would impact performance (perhaps marginally, but it would hit languages with large amounts of keywords/functions/etc. harder than those with smaller numbers, where the impact would be negligible).

You probably could add keywords quite easily yourself though. You'd just need to extend the existing TokenMaker you're interested in, and override the right addToken() overload to check for your added tokens and change the token type if necessary. To do this, you'd do something similar to what's done in UnixShellTokenMaker.

First you'd create an org.fife.ui.rsyntaxtextarea.TokenMap, and load it up with your tokens to add. Then, in your subclass of e.g. TclTokenMaker, override the following addToken() overload to check for your extra tokens:

java code:

public void addToken(char[] array, int start, int end, int tokenType, int startOffset) {
   if (tokenType==Token.IDENTIFIER) { // Assuming all of your new highlighted words are identifiers
      int value = myWordsToHighlight.get(array, start, end);
      if (value != -1) {
         tokenType = value;
      }
   }
   super.addToken(array, start,end, tokenType, startOffset);
}

You're basically intercepting tokens that would be painted as plain identifiers, and deciding whether they should be something different. You can change them to be keywords, functions, or any other token type supported by RSTA.

Finally, register your TclTokenMaker extension class as described by Hookahey back on page 1 of this thread:

java code:

AbstractTokenMakerFactory atmf = (AbstractTokenMakerFactory) TokenMakerFactory.getDefaultInstance();
atmf.putMapping("a key string denoting your style", "fully.qualified.classNameOfYourSyntaxStyle");
textArea.setSyntaxEditingStyle("a key string denoting your style");

You can of course use SyntaxConstants.SYNTAX_STYLE_TCL if you wanted to "override" the existing TCL highlighting.

Does this work for you?

[omadawn]

The thread went back and forth a little so I had come to the conclusion that a simple way to add tokens to an existing interface had not been developed but then there were several points where overriding an existing token maker was discussed but only by folks who seemed to already understand the process. ^_^

I'm fine overriding one I just couldn't figure out how. Let's see if I understand it...

In the sample overridden addToken() method myWordsToHighlight is the instance of org.fife.ui.rsyntaxtextarea.TokenMap that I have created? That looks reasonably straight forward put(stringName, type).



When you say 'You can of course use SyntaxConstants.SYNTAX_STYLE_TCL if you wanted to "override"' you mean register it like so:
AbstractTokenMakerFactory atmf = (AbstractTokenMakerFactory) TokenMakerFactory.getDefaultInstance();
atmf.putMapping("SyntaxConstants.SYNTAX_STYLE_TCL", "com.mydomain.mySyntaxWhichIsTCLPlusAFewKeyWords");
textArea.setSyntaxEditingStyle("SyntaxConstants.SYNTAX_STYLE_TCL");

Then continue to use
rSyntaxTextArea.setSyntaxEditingStyle(SyntaxConstants.SYNTAX_STYLE_TCL);

Or I register it with atmf.putMapping("Whatever darned name I want", <etc>
and use
rSyntaxTextArea.setSyntaxEditingStyle("Whatever darned name I want");




Thanks for the detailed reply. I'm sure I don't have much to add about how completely awesome rsyntaxtextarea is. The code I'm developing is an odd little side project that I started mostly to learn how to use the API my company has to interact with our devices. We have a custom language called iRules which runs on them and is derived very closely from TCL so all I really need to do is add the new keywords. I honestly didn't expect the project to produce anything very useful. I figured I could do reserved word highlighting. Looping through a list and wrapping a highlight block around words in a jEditPane didn't look that difficult. I figured all the other stuff that really makes good code editors useful like paren matching, comment highlighting, etc. would be more work than I was willing to put in.

Then I stumbled across this package and my crappy little editor went from notepad to eclipse Jr. in 11 lines of code. Literally, I copied and paste 8 lines of code out of your sample then modified 3 others.

[robert]

Thanks for the kind words, I'm glad you find RSTA so useful!

When you say 'You can of course use SyntaxConstants.SYNTAX_STYLE_TCL if you wanted to "override"' you mean register it like so:
AbstractTokenMakerFactory atmf = (AbstractTokenMakerFactory) TokenMakerFactory.getDefaultInstance();
atmf.putMapping("SyntaxConstants.SYNTAX_STYLE_TCL", "com.mydomain.mySyntaxWhichIsTCLPlusAFewKeyWords");
textArea.setSyntaxEditingStyle("SyntaxConstants.SYNTAX_STYLE_TCL");

Then continue to use
rSyntaxTextArea.setSyntaxEditingStyle(SyntaxConstants.SYNTAX_STYLE_TCL);

Or I register it with atmf.putMapping("Whatever darned name I want", <etc>
and use
rSyntaxTextArea.setSyntaxEditingStyle("Whatever darned name I want");

Yes, either one will work (except when passing a SyntaxConstants value to setSyntaxEditingStyle, don't wrap it in quotes!). You can use the former if you don't ever need the "standard" TCL support that ships with RSyntaxTextArea. Otherwise there's no difference between the two.

[omadawn]

Ah yeah, not sure where I got the quotes from, thanks for pointing that out.

I won't ever need the standard TCL syntax. Not in this editor but I'll probably make a new one anyway just for clarity. It _should_ make the code easier to maintain. My code is sketchy enough without obfuscating it by referencing things by different names. ^_^

[omadawn]

Fantastic, it works!

Well almost.

I seem to have the syntax down. I'm using Function right now because whatever syntax is defaulting IDENTIFIER to the same color as normal text. I know there's a separate way to specify colors. I'll have to figure that out later. But here's my issue. most of the 'words' I need to add all have a double semicolon in them. For example in my test I did the following.

myWordsToHighlight.put("thisword", Token.FUNCTION);
myWordsToHighlight.put("thatword", Token.FUNCTION);
myWordsToHighlight.put("HTTP::respond", Token.FUNCTION);


thisword and thatword both get colorized. HTTP::respond however does not.

[omadawn]

I spent some time poking around at the code in the debugger and while I'm no closer to getting this to work I have a slightly better understanding of it.

It seems that the problem isn't that HTTP::respond hasn't been created as a token to highlight but rather that RSTA isn't handing it to the syntax checker to verify if it _should_ be highlighted. Looking at what gets passed I get 'HTTP' ':' 'respond'

So I assume that what I need to do is somehow modify the text editors idea of what a 'word' is? I've been poking around the javadocs and have not found a way to do so. I found an article on your blog stating that XML syntax had been updated so it recognizes colons and dashes in words. Is that something I can enable or is that enabled globally and it's the fact that there are two that is screwing me up?

[robert]

Yes, the TCL token maker assumes a word starts with a letter, underscore, or '$' sign, followed by any of those character types again or digits. Unfortunately there's no way to override that without modifying the TclTokenMaker.flex file and regenerating it.

I have no idea what constitutes a valid TCL token. If that type of syntax ("Something::else") is valid in TCL, I'll be happy to update the built-in TclTokenMaker for you, after which you'll be able to do what you're doing without issue. Otherwise, if you wish to do this yourself, look at the class comment at the top of TclTokenMaker.flex and follow its instructions - make your changes, regenerate via JFlex, and make the few required changes by hand on the generated .java file (which I really should get around to automating...).

[omadawn]

It seems like it is based on this list of tcl commands http://www.tcl.tk/man/tcl8.4/TclCmd/contents.htm (see http://www.tcl.tk/man/tcl8.4/TclCmd/packagens.htm for a specific example)
Though it is a little uncommon. More common are commands with underscores in them.

I'm not terribly well versed in TCL specifically myself. ^_^ But the syntax is quite common in iRules.

I'll take a look at the flex file and try and rebuild that with an updated list for my own usage. It seems like this is another way to extend the list of keywords and since I'm doing this I could probably get rid of the extended tokenMaker class I have created to get this far? In fact I might be able to get very specific. iRule commands all use a common UPPERCASE::lowercase format that I could probably check for.

Not sure when I'll get a chance to work on this again I'm pretty busy this week. I'll test and post back as soon as I can.