On-the-fly token adding?

Questions on using RSyntaxTextArea should go here.

Moderator: robert

On-the-fly token adding?

Postby majenko » Sun Jul 14, 2013 11:06 am

Any sign of on-the-fly token adding yet?

I am working on using RSTA to replace a rather ropey text editor in an existing system, and that system has libraries you can add to it (basically C++ classes). Those libraries have a text file with keywords (aka tokens) in them, and it would be fantastic if I could parse those text files and add them to the relevant tokens sections for syntax highlighting.

I saw in pasisng in another post that this had been thought about, but has anything happened yet? It would really be a fantastic addition to what is already a fantastic component.
majenko
 
Posts: 7
Joined: Sun Jul 14, 2013 11:01 am

Re: On-the-fly token adding?

Postby robert » Tue Jul 16, 2013 2:19 am

Nope, this has not happened yet. If you plan on using an existing syntax highlighter, such as CPlusPlusTokenMaker, it should be straightforward to role your own though. Simply subclass, and override the right addToken() overload to check for your extra keywords.

java code:

public class MyTokenMaker extends CPlusPlusTokenMaker {

private TokenMap extraTokens;

public MyTokenMaker() {
extraTokens = createExtraTokens();
}

// ...

@Override
public void addToken(char[] array, int start, int end, int tokenType, int startOffset, boolean hyperlink) {
// This assumes all of your extra tokens would normally be scanned as IDENTIFIER.
if (tokenType == TokenTypes.IDENTIFIER) {
int newType = extraTokens.get(array, start, end);
if (newType>-1) {
tokenType = newType;
}
}
super.addToken(array, start, end, tokenType, startOffset, hyperlink);
}

}
User avatar
robert
 
Posts: 760
Joined: Sat May 10, 2008 5:16 pm

Re: On-the-fly token adding?

Postby majenko » Tue Jul 16, 2013 9:23 am

So I'd populate extraTokens with something like
Code: Select all
extraTokens.put("SPOON", TokenTypes.IDENTIFIER);

(I'm pretty new to Java)

For multiple token types (I have reserved words, identifiers, macros, etc) would I have multiple TokenMaps, or just use the one TokenMap with different types? How would that affect the addToken() method?
majenko
 
Posts: 7
Joined: Sun Jul 14, 2013 11:01 am

Re: On-the-fly token adding?

Postby robert » Tue Jul 16, 2013 12:39 pm

Yes, you're right. Specify the token to be highlighted, and what token type you want to highlight it with. Note that you can technically use any value from TokenTypes, as they have no special meaning, but typically you'll stick to whatever types make the most logical sense. You can add multiple tokens of different types all to a single TokenMap; that's how it was designed to operate. i.e.

java code:

TokenMap extraTokens = new TokenMap();
extraTokens.put("foreach", TokenTypes.RESERVED_WORD);
extraTokens.put("unless", TokenTypes.RESERVED_WORD);
extraTokens.put("doTaxes", TokenTypes.FUNCTION);
extraTokens.put("fillGradient", TokenTypes.FUNCTION);


The addToken() method you override is unchanged from the example I gave. It's just assuming that all tokens you add to the TokenMap are parsed as IDENTIFIERs by the subclassed TokenMaker (e.g., typically starting with a letter or underscore, followed by 0 or more letters, underscores, or digits, but this depends on the language you've subclassed).
User avatar
robert
 
Posts: 760
Joined: Sat May 10, 2008 5:16 pm

Re: On-the-fly token adding?

Postby majenko » Tue Jul 16, 2013 1:04 pm

Hmmm Getting there :)

Now... can I access the token maker within the RSTA instance? So I can do something like (from another class):

Code: Select all
((MyTokenMaker) myTextArea.getTokenMaker()).addNewToken("spoon", TokenTypes.FUNCTION);


?
majenko
 
Posts: 7
Joined: Sun Jul 14, 2013 11:01 am

Re: On-the-fly token adding?

Postby majenko » Tue Jul 16, 2013 2:34 pm

Looking at it, what's needed is RSyntaxDocument.getSyntaxStyle():

Code: Select all
    public TokenMaker getSyntaxStyle() {
        return tokenMaker;
    }
majenko
 
Posts: 7
Joined: Sun Jul 14, 2013 11:01 am

Re: On-the-fly token adding?

Postby majenko » Tue Jul 16, 2013 3:49 pm

Well, whaddayaknow? With that small mod to the RSyntaxDocument class I have it working. I now load my extra keywords at RSTA creation time, and the highlighting is right :) The highlighting even changes correctly when the available keywords change as you select a different library set!

Beautiful!

I really love RSTA :)
majenko
 
Posts: 7
Joined: Sun Jul 14, 2013 11:01 am

Re: On-the-fly token adding?

Postby majenko » Sun Jul 21, 2013 8:59 pm

One problem I have just noticed...

Extended the C++ class, but there is no function folding. The popup menu has folding greyed out.

It works on the pure C++ class, but not my inherited version.

Is there something I need to do to make it work?

Here's my whole extension:
Code: Select all
package org.fife.ui.rsyntaxtextarea.modes;

import java.io.*;
import javax.swing.text.Segment;

import org.fife.ui.rsyntaxtextarea.*;

public class ArduinoTokenMaker extends CPlusPlusTokenMaker {
    private TokenMap extraTokens;

    public ArduinoTokenMaker() {
        extraTokens = new TokenMap();
    }

    @Override
    public void addToken(char[] array, int start, int end, int tokenType, int startOffset, boolean hyperlink) {
        // This assumes all of your extra tokens would normally be scanned as IDENTIFIER.
        if (tokenType == TokenTypes.IDENTIFIER) {
            int newType = extraTokens.get(array, start, end);
            if (newType>-1) {
                tokenType = newType;
            }
        }
        super.addToken(array, start, end, tokenType, startOffset, hyperlink);
    }

    public void addKeyword(String keyword, int type) {
        extraTokens.put(keyword, type);
    }

    public void clear() {
        extraTokens = new TokenMap();
    }
}
majenko
 
Posts: 7
Joined: Sun Jul 14, 2013 11:01 am

Re: On-the-fly token adding?

Postby majenko » Sun Jul 21, 2013 9:22 pm

Actually, ignore that - I just found the FoldParserManager ;)
majenko
 
Posts: 7
Joined: Sun Jul 14, 2013 11:01 am


Return to Help

Who is online

Users browsing this forum: No registered users and 7 guests

cron