TokenTokenMaker and hex in assembly

[Vicne]

Hi,

Let me first thank you for your outstanding components. I really love them.

I am using TokenTokenMaker to customize the syntax highlighting for the assembly language I am working with, and I have two questions. I hope it's the best place to ask...

1) First, I'm not sure I'm using the best mapping between keyword, datatype and function.
For example, in the instruction
000401D8 C001 LDI:8 #0x00,R1
I guess # and , must be declared as operators, but would you declare the instruction (LDI:8) a keyword or a function ? What is the impact of choosing one or the other ?
Similarly, what about the register (R1) : keyword or data type ?

2) What about addresses and hex representation of the instruction, that are present at the beginning of each line, such as :
000401D8 C001 LDI:8 #0x00,R1
For now, it seems they are considered numbers if they only have decimal digits, but literal (other color) as soon as they include a hex letter. Is there a way to highlight them the same way ?

Best regards,

Vicne

[robert]

Hi Vince,

1. The token types you choose to colorize different things in your language are actually completely arbitrary. You could use the "keyword" type to highlight number literals for example, and nothing "bad" would happen. The token types have names simply to try to provide some level of consistency in syntax highlighting between languages. In reality, there are many languages with lexical tokens that don't have an exact token type defined in RSTA; in those cases I simply use a different token type that isn't being used elsewhere in the language.

In short - there's no impact, use whatever you want.

2. I believe you want to add the ability to parse hex literals and colorize them as numbers, correct? This could be added if so, although without some sort of prefix, i.e. "0x...", such a scanner might miss things like "C001" since it would be difficult to decide between it being an identifier and a hex literal. If it's positional like your situation (I think), a hand-made TokenMaker would likely have to be used to take care of all cases. Although thinking about it, you *might* be able to get away with a regex, such as "^[0-9a-zA-Z]\s+[0-9a-zA-Z]" representing a number literal, right? Again, this would be another new feature for TMM though...

TokenMakerMaker does need to be updated, it hasn't been modified in awhile and is behind where it could be with respect to RSTA's new features.

[Vicne]

Hi, and thanks for your reply

1. The token types you choose to colorize different things in your language are actually completely arbitrary.
... In short - there's no impact, use whatever you want.

Great.

2. I believe you want to add the ability to parse hex literals and colorize them as numbers, correct?

Yes. In fact, all "numbers" are hex in this file.

This could be added if so, although without some sort of prefix, i.e. "0x...", such a scanner might miss things like "C001" since it would be difficult to decide between it being an identifier and a hex literal. If it's positional like your situation (I think), a hand-made TokenMaker would likely have to be used to take care of all cases.

OK, I tried starting from the resulting .flex and merging the result with the AssemblerX86TokenMaker you provide as part of the RSyntaxTextArea.
I then succeeded in creating the Java class with JFlex and removing the duplicate methods etc.
It's starting to look good.

Although thinking about it, you *might* be able to get away with a regex, such as "^[0-9a-zA-Z]\s+[0-9a-zA-Z]" representing a number literal, right?

Well, yes, I thought about that for the address, which moreover has a fixed length, but I have a problem. When I define :

Address = (^[0-9a-fA-F]\s+[0-9a-fA-F])

I'm getting a Syntax Error on the caret sign in JFlex (without it, it's ok)... No idea why...

Again, this would be another new feature for TMM though...
TokenMakerMaker does need to be updated, it hasn't been modified in awhile and is behind where it could be with respect to RSTA's new features.

OK. But I understand assembly is not a typical structured language so maybe it's out of scope for TMM.

Thanks very much for your help,

Best regards,

Vicne

[Guest]

In short - there's no impact, use whatever you want.

One more question : I'm using setMarkOccurrencesColor(true) but it seems only some expressions react to it. Is that linked to the tokens I define in the flex file ?

Best regards,

Vicne

[robert]

You'll need to override the AbstractTokenMaker.getMarkOccurrencesOfTokenType(int) method to return true for all token types you want "marked." There's no direct support for it in TokenMakerMaker but you should be able to edit the generated .flex or .java file with your override. Yet another thing to add to TokenMakerMaker!

[Vicne]

Hi, Robert,

Thanks for your help and sorry for the delay.
Overriding AbstractTokenMaker.getMarkOccurrencesOfTokenType(int) worked indeed, but it doesn't highlight tokens cross-type. In other words, if one clicks on an expression in a comment, that expression doesn't get highlighted when it's used as an argument. Not a big deal, but I'd rather have highlights not be limited to one type.

Thanks again,

Vicne

[robert]

Yes, it's done on a per-type basis. In reality, marking occurrences is usually semantics-based; that is, clicking on a variable named "foo" will only mark occurrences of "foo" that are actually that variable's name. RSTA is just cheating and highlighting all identifiers (or tokens of the same type) as the "foo" you click on.

It sounds like what you really want is "mark all" perhaps? It would be a little legwork, but you could implement this yourself with a CaretListener and these methods (on RTextArea):

Code: Select all

int markAll(String toMark, boolean matchCase, boolean wholeWord, boolean regex);
void clearMarkAllHighlights();
void setMarkAllHighlightColor(Color color);


You'd basically want to clone the org.fife.ui.rsyntaxtextarea.MarkOccurrencesSupport class, since it handles imposing a delay with no caret changes before changing the highlights. You'd just change the actionPerformed() method to be something like this:

Code: Select all

public void actionPerformed(ActionEvent e) {

   textArea.clearMarkAllHighlights();
   if (c.getDot()!=c.getMark()) {
      return;
   }

   try {
      int start = Utilities.getWordStart(textArea, c.getDot());
      int end = Utilities.getWordEnd(textArea, c.getEnd());
      String word = textArea.getText(start, end); // May need to be end+1, not sure
      textArea.markAll(word, false, true, false);
   } catch (BadLocationException ble) {
      ble.printStackTrace();
   }

}


Does that sound like it'll fit your needs?

[Vicne]

It sounds like what you really want is "mark all" perhaps?

Yes, that would do it. Sometimes, a value is loaded in a register that will be used in a jump. So in this case, clicking on the operand and highlighting the matching address would make sense. Assembly is very - very - loosely typed

You'd basically want to clone the org.fife.ui.rsyntaxtextarea.MarkOccurrencesSupport class, since it handles imposing a delay with no caret changes before changing the highlights. You'd just change the actionPerformed() method to be something like this: (...)
Does that sound like it'll fit your needs?

Yes, exactly.
I cloned it that way (named it MarkAllOccurrencesSupport) and made it extend MarkOccurrencesSupport, but I missed a "setMarkOccurrencesSupport(MarkOccurrencesSupport);" method on RSyntaxTextArea.
Had to subclass RSyntaxTextArea to add such a method but MarkOccurrencesSupport is private so I couldn't set it from a subclass.
I ended up copying all occurrence-related methods from RSyntaxTextArea and it builds...
... but it crashes on init because as MarkAllOccurrencesSupport extend MarkOccurrencesSupport, the super constructor gets called (still don't understand why as I don't call super() explicitely) and crashes because it calls in turn :

p = new MarkOccurrencesHighlightPainter(); // This line addresses the private "p" of the superclass
setColor(color); // This line calls the overridden method that works on a separate "p" instance which is not yet initialized, so it crashes with a NPE.



Well, I could start hacking the original RSyntaxTextArea classes, but I'd rather be able to simply update the version in the future and subclassing seemed cleaner to me. Have to think about it more...

Thanks again for your insightful help.

Vicne

[robert]

Rather than subclassing MarkOccurrencesSupport (I see now that you need access to the private field "p"), you could make a copy of it and name it MarkAllOccurrencesSupport, and change just the actionPerformed() method. Alternatively you could change the source of your local copy of MarkOccurrencesSupport so that either p is protected, or you can add a protected getPainter() method so your subclass can access the field.

Then, rather than modifying RSyntaxTextArea so its MarkOccurrencesSupport is your copy, simply install yours from the outside:

Code: Select all

MarkAllOccurrencesSupport support = new MarkAllOccurrencesSupport();
support.install(textArea);


This is all RSTA really does anyway when you call setMarkOccurrences(true) (besides firing a property so anybody who wants to get notified about it does get notified).

[Vicne]

Rather than subclassing MarkOccurrencesSupport (I see now that you need access to the private field "p"), you could make a copy of it and name it MarkAllOccurrencesSupport, and change just the actionPerformed() method.

That's exactly what I had done.

Then, rather than modifying RSyntaxTextArea so its MarkOccurrencesSupport is your copy, simply install yours from the outside:

Code: Select all

MarkAllOccurrencesSupport support = new MarkAllOccurrencesSupport();
support.install(textArea);


Oh, yes. That way ! I was looking for a method on the area to add the support, not the other way round. Excellent.

OK. That's done.

Now, the behaviour is quite strange : not only do I have the highlights matching the term I clicked on (green), but I also have weird blocks highlighted in orange when I set the focus on some words :


(Please note the light yellow line is the one which has the caret on, while the blue highlight is set to indicate which line the debugger is on)

Is that a known side effect ?

Sorry if I'm asking too many questions. Don't hesitate to tell me if it's the case.

Best regards,

Vicne