Custom Folding, MLC, and RSTA code insights

[Kuriel]

Hi,

Being my first post, I'd like to say that this creation of yours is excellent. I'm glad it off-shot from RText as I needed something very much like this, started coding from scratch, got a good deal ahead, but, unfortunately, ran out of time. Finding this was a blessing

I've spent a day putting together a custom flex file for a personal script language, processed it, and integrated it into my project. It's working very well out of the box, thank you. However, since I'm pressed for time, I can't devote as much of it as I'd like to learning the intricacies any further. The remaining things I want to do is:

1. To incorporate custom folding, specifically starting with "@@" and ending with "!!"; while at the same time, integrating the html folding that's available in RSTA. I tried starting from the Curly fold parser and then doing another take with the HTML folder, but I'm missing something since it's not working. What I did so far was to :

a. Make a custom fold parser. I tried using the property FOLD_TYPE_USER_DEFINED_MIN but it wasn't catching
b. Inserted the constructor for this parser into the FoldParserManager class for my custom syntax style
c. Added a Token property for the FOLD_TYPE_USER_DEFINED_MIN property
d. wrote into the custom flex the regex for this property by doing:
d1. CustomFold = ("@@"|"!!")
d2. {CustomFold} {addToken(Token.FOLD_TYPE_USER_DEFINED_MIN)}

Having only had a day of experience with RSTA and JFlex, I'm pretty sure with all those steps, I botched it completely If you could outline what I need to do, it'd be most helpful. Ideally, I'd like to have the html folding working with it, but if I can just get folding for lines between "@@" and "!!", I'll be happy. I don't mind starting from scratch

2. Make the single and double quote characters the beginnings and ends of a multi-line comment/doc comment, , behaving like the ones Java uses (i.e. "/* */", "/** */) . Also, making the string "<!--" as the start of a line comment which ends with "-->".

3. Find out where I can inject a document filter (or something similar) to replace a bracket with a pair of brackets (i.e. [] for [ or ], {} for { or } etc) as it is inputted. I tried adding it to the TextEditorPane class but something is preventing it from triggering. My original work used a JTextPane and it worked there; could it be because RSTA uses a JTextArea that this is not working? Out of curiousity, what would happen to RSTA if I changed its base class to a JTextPane?

4. Make small changes to the bracket matching code in order to highlight both partner brackets instead of just the remote one. Also, it seems that the bracket partner search only starts when a bracket is found to precede the caret. I'd like to change this to include a search when the caret succeeds a bracket and is not preceded by another bracket. I'll likely be looking for any caret listener RSTA has in its code (if any), but any insight would be welcome

Thanks again for this wonderful project and for any help you can afford me.

[robert]

Hi Kuriel,

1. To incorporate custom folding, specifically starting with "@@" and ending with "!!"; while at the same time, integrating the html folding that's available in RSTA. I tried starting from the Curly fold parser and then doing another take with the HTML folder, but I'm missing something since it's not working. What I did so far was to :

This should be doable. It looks like you're trying to identify "@@" and "!!" as tokens, but not necessarily color what's between them a certain color. Is that right? If so then you're on the right track. For step d2, instead of using Token.FOLD_TYPE_USER_DEFINED_MIN, you must use an existing token type. RSTA will only render tokens that are one of the built-in token types. At first, I'd suggest you pick something like Token.SEPARATOR so you can easily identify them as being identified correctly by your TokenMaker.

Then, as to your FoldParser, I'm not sure where this fold region is valid (outside of HTML tags, inside them, anywhere, etc.). I'm going to assume they must be outside of HTML tags. Without digging too deep, I think you'd just add some extra logic for your case. What that logic is, and where it goes in the file, depends on the things above, as well as whether HTML tags are allowed inside the new fold regions as well as outside of them. Can you provide more details?

2. Make the single and double quote characters the beginnings and ends of a multi-line comment/doc comment, , behaving like the ones Java uses (i.e. "/* */", "/** */) . Also, making the string "<!--" as the start of a line comment which ends with "-->".

So standard HTML <!-- ... --> comments would now only be single-line comments? If one spanned more than one line, that would be an error token? As for the string-style MLC's, I'm assuming those would only be outside of tags, to avoid clashing with attributes, correct?

I'm starting to wonder whether you should start with the HTML syntax highlighting & folding, or would need to write your own entirely...

3. Find out where I can inject a document filter (or something similar) to replace a bracket with a pair of brackets (i.e. [] for [ or ], {} for { or } etc) as it is inputted. I tried adding it to the TextEditorPane class but something is preventing it from triggering. My original work used a JTextPane and it worked there; could it be because RSTA uses a JTextArea that this is not working?

I would think adding a DocumentFilter should work, I'd have to look into it to find out why it doesn't. One alternative (and probably cleaner than a DocumentFilter) is to create custom Actions to fire when inserting '[' and '{' characters, and add entries to the text area's InputMap and ActionMap to have your actions fire when the user types those characters.

Out of curiousity, what would happen to RSTA if I changed its base class to a JTextPane?

I imagine it'd be more work than first thought. At a minimum, JTextArea provides some utility methods that JTextPane does not, that you'd have to implement. Feel free to try it though and let me know how it goes.

4. Make small changes to the bracket matching code in order to highlight both partner brackets instead of just the remote one.

This can be done out of the box, as of 2.0.5. Just call setPaintMatchedBracketPair().

Also, it seems that the bracket partner search only starts when a bracket is found to precede the caret. I'd like to change this to include a search when the caret succeeds a bracket and is not preceded by another bracket. I'll likely be looking for any caret listener RSTA has in its code (if any), but any insight would be welcome

I'd like to see this feature as well. Feel free to add a Feature Request if you'd like to see this added, so it can be tracked. I'll look into it. If you want to look at it yourself, the code to do bracket matching is nasty, you'll want to look at RSyntaxUtilities.getMatchingBracketPosition(RSyntaxTextArea). I think it would be straightforward to implement; you'd just want to factor out the "first part" (that which finds whether you're on a matchable character at all) into its own method, and first call it with the "preceding" char, then call it with the "next" char if the preceding char wasn't a matching-bracket character.

[Guest]

Thank you for the great response, robert. I appreciate you taking the time to look in on this.

This should be doable. It looks like you're trying to identify "@@" and "!!" as tokens, but not necessarily color what's between them a certain color. Is that right?

Yes, that's correct. I'm hoping though that the single "@" would still trigger the annotate coloring?

If so then you're on the right track. For step d2, instead of using Token.FOLD_TYPE_USER_DEFINED_MIN, you must use an existing token type. RSTA will only render tokens that are one of the built-in token types. At first, I'd suggest you pick something like Token.SEPARATOR so you can easily identify them as being identified correctly by your TokenMaker.

Nice. I'll do that. Thanks for the insight.

Then, as to your FoldParser, I'm not sure where this fold region is valid (outside of HTML tags, inside them, anywhere, etc.). I'm going to assume they must be outside of HTML tags. Without digging too deep, I think you'd just add some extra logic for your case. What that logic is, and where it goes in the file, depends on the things above, as well as whether HTML tags are allowed inside the new fold regions as well as outside of them. Can you provide more details?

I believe that it must be inside the tags? Would folding these regions inside tags, mean that the enclosing tags remain visible? Overall, everything between @@ and !! should get folded. For example:

@@
[line for script]
[line 2 for script]
<html>
foldable region
</html>
!!

So standard HTML <!-- ... --> comments would now only be single-line comments? If one spanned more than one line, that would be an error token? As for the string-style MLC's, I'm assuming those would only be outside of tags, to avoid clashing with attributes, correct?

I'm starting to wonder whether you should start with the HTML syntax highlighting & folding, or would need to write your own entirely...

I hope I wouldn't need to write one from scratch. Actually, in code execution, the html comment still functions as an MLC. I just want to use the toggle-comment functionality to use the html comment, which I've been able to do in my application of RSTA.

As for the string-style MLC, I'm after the multiline coloration; as it is, the quotes only color a single line, not the span of what they enclose if, like say, the line is broken by a carriage return/line wrap. It's all to catch an un-closed string so the user can immediately see they're missing a quote.

I would think adding a DocumentFilter should work, I'd have to look into it to find out why it doesn't. One alternative (and probably cleaner than a DocumentFilter) is to create custom Actions to fire when inserting '[' and '{' characters, and add entries to the text area's InputMap and ActionMap to have your actions fire when the user types those characters.

That can work, thanks for the idea. It's just that, on my own attempt at something like RSTA before discovering it, I wrote the logic for a DocumentFilter that worked in tandem with a DocumentListener, and it worked as it should. This was to have a place to quickly add document style and behavior. So when I was working with RSTA, I just thought "Let's try to inject that here...", to save time.

I imagine it'd be more work than first thought. At a minimum, JTextArea provides some utility methods that JTextPane does not, that you'd have to implement. Feel free to try it though and let me know how it goes.

I figured as much. It's only due to the previous item that I asked. I suspect that parts of what I wrote do not function, but without error, in a JTextArea. IIRC, JTextPane and parent class JEditorPane follow a different architectural path from the same JTextComponent parent JTextArea shares. I'm not sure though, I skimmed the surface on info about that subject.

This can be done out of the box, as of 2.0.5. Just call setPaintMatchedBracketPair().

Awesome! I did see a thread where you mentioned it but I wasn't sure if it was exactly what I was after.

I'd like to see this feature as well. Feel free to add a Feature Request if you'd like to see this added, so it can be tracked. I'll look into it. If you want to look at it yourself, the code to do bracket matching is nasty, you'll want to look at RSyntaxUtilities.getMatchingBracketPosition(RSyntaxTextArea). I think it would be straightforward to implement; you'd just want to factor out the "first part" (that which finds whether you're on a matchable character at all) into its own method, and first call it with the "preceding" char, then call it with the "next" char if the preceding char wasn't a matching-bracket character.

Great. I'll do that. But if I have time in a few days, I'll also look at the class you mentioned, and do the extraction and coding myself. Perhaps PM you the results? If you get to it sooner though, I'll be happier with that as well

Thanks again.

[robert]

I actually have even more questions after your last post, but I'll just tackle 'em one at a time.

This should be doable. It looks like you're trying to identify "@@" and "!!" as tokens, but not necessarily color what's between them a certain color. Is that right?

Yes, that's correct. I'm hoping though that the single "@" would still trigger the annotate coloring?

The HTML syntax highlighter doesn't highlight annotations, the only language that does that currently is Java. You'd have to add annotation highlighting to your TokenMaker flex file (which would be easy, just copy it from JavaTokenMaker.flex). And yes, it's easy to highlight both.

I believe that it must be inside the tags? Would folding these regions inside tags, mean that the enclosing tags remain visible? Overall, everything between @@ and !! should get folded. For example:

@@
[line for script]
[line 2 for script]
<html>
foldable region
</html>
!!

That helps a little. My question was basically, is the example below *also* valid?

Code: Select all

[line for script]
<html>
[line for script]
@@
[line for script]
[line for script]
!!
</html>


And if so, what's considered a fold region? I'm assuming the html tag's contents, with a nested one for the @@/!! pair?

I hope I wouldn't need to write one from scratch. Actually, in code execution, the html comment still functions as an MLC. I just want to use the toggle-comment functionality to use the html comment, which I've been able to do in my application of RSTA.

That will happen as long as your TokenMaker's getLineCommentStartAndEnd() method returns { "<!--", "-->" }. HTMLTokenMaker does this out of the box.

As for the string-style MLC, I'm after the multiline coloration; as it is, the quotes only color a single line, not the span of what they enclose if, like say, the line is broken by a carriage return/line wrap. It's all to catch an un-closed string so the user can immediately see they're missing a quote.

HTML already does this, at least, for HTML attributes. Are you asking for this with a string type outside of HTML tag attributes? This is doable by copying the implementation that's already there.

Great. I'll do that. But if I have time in a few days, I'll also look at the class you mentioned, and do the extraction and coding myself. Perhaps PM you the results? If you get to it sooner though, I'll be happier with that as well

I hope to do it soon (should be easy), but I've been having trouble finding time to devote to the project lately, unfortunately. But I believe I should be able to get to it in the next few days.

[Guest]

Thanks again for the quick reply, robert.

I actually have even more questions after your last post, but I'll just tackle 'em one at a time.

I can imagine. Sorry...

The HTML syntax highlighter doesn't highlight annotations, the only language that does that currently is Java. You'd have to add annotation highlighting to your TokenMaker flex file (which would be easy, just copy it from JavaTokenMaker.flex). And yes, it's easy to highlight both.

Nice. I think I did that already but haven't compiled it into the library my application uses.

That helps a little. My question was basically, is the example below *also* valid?

Code: Select all

[line for script]
<html>
[line for script]
@@
[line for script]
[line for script]
!!
</html>


Hmm, it wouldn't be I'm afraid. The reason for encapsulating stuff between @@/!! is because users have been using a user defined highlighting scheme for Notepad++. The actual application didn't have anything resembling a good editor, so someone came up with a process to extract a script's contents, grab the properties of its origin and used it as string properties/annotations coming after @@. This was to distinguish one script from another since his "plug-in" allowed for loading of multiple scripts at once. So the lines for @@ and !! aren't actually part of the script's syntax, they're just there to divide code blocks from different scripts/macros from each other on N++. Here's a sample of how it looks (sorry it's a bit long):

Code: Select all

@@ @Frame: Combat
@PROPS@ fontColor=black ; autoExecute=true ; fontSize=1.00em ; sortBy= ; color=default ; playerEditable=false ; applyToSelected=false ; group=Windows ; tooltip= ; minWidth= ;
[h, code: {
[if(isFrameVisible("Proxy Window")): closeFrame("Proxy Window")]
[current = getName(getInitiativeToken())]
[jsonChar = getLibProperty(current, "Lib: Properties")]
[stance = json.get(jsonChar, "stance")]
[charSkills = json.get(jsonChar, "charSkills")]
[charSkillKeys = json.get(jsonChar, "charSkillKeys")]
[magic = !json.isEmpty(json.get(jsonChar, "spells"))]
[kata = !json.isEmpty(json.get(jsonChar, "kata"))]
[kiho = !json.isEmpty(json.get(jsonChar, "kiho"))]
[maho = !json.isEmpty(json.get(jsonChar, "maho"))]
[tattoo = !json.isEmpty(json.get(jsonChar, "tattoo"))]
[guard = !json.contains(json.get(jsonChar, "expirationQueue"), "Guard")]
[classCheck = json.get(jsonChar, "characterClass")]
[iaijutsuCheck = json.contains(charSkills, "Iaijutsu")]
[actualVoid = json.get(jsonChar, "actualVoid")]
[checkGrapple()]
[grappleType = json.get(macro.return, "type")]
[grappleCheck = if(actualVoid > 0 || grappleType, 1, 0)]
[handsCheck = json.get(jsonChar, "hands")]
[largeWeaponInHand = 0]
[proneCheck = 0]
[attman = 0]
[hands = !json.isEmpty(handsCheck)]
[if(hands): combine = json.merge(handsCheck), invCheck); combine = json.get(jsonChar, "inventory")]
[swordCheck = if(!json.isEmpty(getItemDetails(combine, json.append('[]', "Katana", "Wakizashi"), current)), 1, 0)]
[swordSpellCheck = 0]
[if(classCheck == "Shugenja"), code:{
[swordSpellCheck = json.contains(json.get(jsonChar, "memorizedSpells"), "Katana of Fire")]
[swordCheck = if(swordCheck || swordSpellCheck, 1, 0)]
}]
}]

[h, if(hands), code: {
[getItemDetails(handsCheck, json.append('[]', "Weapon"), current)]
[foreach(item, macro.return), code: {
[weaponCheck = json.contains(json.get(item, "keyWords"), "Large")]
[if(weaponCheck && largeWeaponinHand == 0): largeWeaponInHand = 1]
[skill = json.get(item, "skill")]
[if(skill != ""): attman = if(json.contains(charSkills, skill), 1, 0); '<!-- -->']
}]
};
{
[attman = if(json.contains(charSkills, "Jiujitsu"), 1, 0)]
}]

[h, code: {
[token(current), if(getState("Prone") && largeWeaponInHand): proneCheck = 1; proneCheck = 0]
[fullhands = json.length(handsCheck)]
[pickup = !json.isEmpty(json.get(jsonChar, "pickup"))]
[inventory = json.get(jsonChar, "inventory")]
[small = 0]
[medium = 0]
[large = 0]
[items = 0]
}]

[h, if(!json.isEmpty(inventory)), code: {
[getItemDetails(inventory, '[]', current)]
[readyItemsCheck = !json.isEmpty(macro.return)]
[if(readyItemsCheck): evalMacro('
[foreach(item, macro.return), code:{
[keyWords = json.get(item, "keyWords")]
[if(json.contains(keyWords, "Small") && small == 0): small = 1]
[if(json.contains(keyWords, "Medium") && medium == 0): medium = 1]
[if(json.contains(keyWords, "Large") && large == 0): large = 1]
}]
'); '<!-- -->']
[getItemDetails(handsCheck, json.append('[]', "Activated"), current)]
[items = !json.isEmpty(macro.return)]
};
{
'<!-- -->'
}]

[h, code: {
[ta = json.get(jsonChar, "totalActions")]
[rmLink=macroLinkText("Process: Combat Options@Lib: Scripts", "none")]
}]

[frame("Combat Options"): {
<html>
<head>
</head>
<body>
<form action="[r:rmLink]" method="json">
<table border="1">
<tr border="1" colspan="7" align="center"><th colspan="7" border="1" align="center" style="background-color:white"><b><u>General Actions</u></b></th></tr>
<tr border = "1" colspan="7" align="center">
[token(current), if(ta < 2 && getState("Prone")): '<input type="submit" name="btncombat" value= "Stand up">']
[if(ta == 0 && (stance == "Full Attack" || stance == "Attack") && !proneCheck): '<input type="submit" name="btncombat" value= "Attack">']
[if(ta == 0 && (stance == "Full Attack" || stance == "Attack") && attman  && !proneCheck): '<input type="submit" name="btncombat" value= "Attack + Maneuvers">']
[if(ta == 0 && grappleCheck && (stance == "Full Attack" || stance == "Attack") && !proneCheck): '<input type="submit" name="btncombat" value= "Grapple">']
[if(guard && ta < 2 && stance != "Full Attack"): '<input align="center" type="submit" name="btncombat" value= "Guard">']
[if((actualVoid > 0 || iaijutsuCheck) && swordCheck && !getState("Dazed", current) && ta == 0 && (stance == "Full Attack" || stance == "Attack") && !proneCheck):'<input type="submit" name="btncombat" value= "Iaijutsu Challenge">']
</tr>
<tr border="1" colspan="7" align="center"><th border="1" colspan ="7" align="center"style="background-color:white"><b><u>Special Abilities</u></b></th></tr>
<tr border="1" colspan="7" align="center">
[if(ta == 0 && magic && (stance != "Full Attack")): '<input type="submit" name="btncombat" value="Spellcasting">']
[if(ta <= 1 && kata): '<input type="submit" name="btncombat" value= "Kata">']
[if(ta <= 1 && kiho): '<input type="submit" name="btncombat" value= "Kiho">']
[if(ta == 0 && maho): '<input type="submit" name="btncombat" value= "Maho">']
[if(tattoo): '<input type="submit" name="btncombat" value= "Tattoos">']
</tr>
<tr border="1" colspan="7" align="center"><th colspan ="7" align="center"style="background-color:white"><b><u>Equipment Actions</u></b></th></tr>
<tr border="1" colspan="7" align="center">
[if(hands && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Drop Held Item">']
[if(ta <= 1 && pickup && fullhands < 2 && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Pickup Item">']
[if(ta <= 1 && large && fullhands == 0 && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Ready: Large Item">']
[if(ta <= 1 && medium && fullhands < 2 && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Ready: Medium Item">']
[if(small && fullhands < 2 && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Ready: Free or Small Item">']
[if(ta <= 1 && hands && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Stash Held Item">']
[if(ta <= 1 && items && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Use Item">']
</tr>
<tr border="1" colspan="7" align="center"><th colspan ="7" align="center"style="background-color:white" border = "1"><b><u>Misc. Actions</u></b></th></tr>
<tr border="1" colspan="7" align="center">

[h: length = json.length(getLibProperty("jsonDelay", "Lib: Properties"))]
[h, if(getCurrentInitiative() != initiativeSize() - 1): end = 0; end = 1]
[if(ta < 2 && length < initiativeSize() - getCurrentInitiative() - 1 && !end): '<input type="submit" name="btncombat" value= "Delay Turn">']
['<input type="submit" name="btncombat" value= "End Turn">']
[if(ta == 0 && stance != "Full Attack"): '<input type="submit" name="btncombat" value= "Use Skill">']
</tr>
</table>
</form>
</body>
</html>
}]

!!


Actually, html can also be found outside script lines bound by [] because the whole thing get's rendered/outputted on an HTML pane.

And if so, what's considered a fold region? I'm assuming the html tag's contents, with a nested one for the @@/!! pair?

So, essentiallly, it would be the other way around. Everything is bound within @@/!! and if there're html tags (not within []) then these would be nested. Again, what's important really is the folding between @@/!!. html folding would really be nice, but it's not something we have, even in the N++ UDL.

That will happen as long as your TokenMaker's getLineCommentStartAndEnd() method returns { "<!--", "-->" }. HTMLTokenMaker does this out of the box.

Yes, thank you. I got that part going at the start since it was so straightforward.

HTML already does this, at least, for HTML attributes. Are you asking for this with a string type outside of HTML tag attributes? This is doable by copying the implementation that's already there.

I'll take another crack at it. I was looking at the Java implementation, you see, since /** */ and /* */, was what I am used to.

I hope to do it soon (should be easy), but I've been having trouble finding time to devote to the project lately, unfortunately. But I believe I should be able to get to it in the next few days.

I understand completely. This project I'm working on is one of those things that are satisfying to do, but sadly do not bring home the bacon .

Thank you once again.

[Guest]

Well I got MLC to begin but my MLCEnd (-->) doesn't cut it off becaues during the flex processing, I always get the warning the the rule will never be matched. I've tried a lot of stuff but the need to process the flex, delete the extraneous functions and correct the one line that has a missing variable, is piling up a lot of head-to-table banging My folding code is a bit messed up too, maybe because of trying too much. Right now, I'll be happy with just folding stuff between @@ and !!.

[Kuriel]

I figured it out (apart from folding - haven't got around to it yet), thanks. I started from scratch using HTML instead of Java as base, and it was easier from that point on. I need to tweak the regex to reflect statements being enclosed in "[]" as, right now, the reserved/function word highlighting is lost when they're inside brackets.

Edit: I got the MLC working as well as reserved words working enclosed. Also nailed the folding for code between @@ and !!. I had to switch back to the Java model though I guess the most challenging step now would be folding html that's within the @@!! bounds.

Edit #2 : Is there a default behavior for the @ sign somewhere outside the generated java from flex? Even if I took all references within the flex to @ (e.g. annotation, url), it colors whatever comes before or after it to what I'm guessing as the one for URL (email?). Also, I have @@ as Token.SEPARATOR, and I'm trying to get @ to be LineCommentBegin. Well it sort of works, but I need a space after @ to get the line to color green, and when I don't have that space, it triggers that coloration I mentioned earlier (sort of gold-ish color).

Edit #3: Had a crack at the bracket matching but didn't finish it. Apparently, it needs little different handling than what's in place right now. The matching works fine after the change, it's the repaint that's not the same. So, when I saw that I needed to change some things over more than one class, I thought that it'd be better to have your solution instead, since I didn't want to re-apply what I did should RSTA get updated in the future, and more likely than not, your solution would be more elegant

Edit #4: For some reason, autocomplete's autoactivation isn't kicking in. I set it to true, made sure autocomplete is enabled and set the delay to 1200 to 2000ms over several tests. The popup just doesn't appear, though hitting the trigger key pulls up the popup windows just fine. Any ideas on what I'm doing wrong?

Edit #5: BTW, robert, your build.xml files had a typo on the debuglevel line. It said "var" instead of "vars". I saw it on both the RSTA and Autocomplete checkouts.

[robert]

Is there a default behavior for the @ sign somewhere outside the generated java from flex? Even if I took all references within the flex to @ (e.g. annotation, url), it colors whatever comes before or after it to what I'm guessing as the one for URL (email?). Also, I have @@ as Token.SEPARATOR, and I'm trying to get @ to be LineCommentBegin. Well it sort of works, but I need a space after @ to get the line to color green, and when I don't have that space, it triggers that coloration I mentioned earlier (sort of gold-ish color).

Sounds like you have some sort of "ErrorIdentifier" state that is matching tokens deemed to be invalid in your language. Some of the standard TokenMakers do this - the gold-ish color is the "error" color (which I know isn't very intuitive. At least one theme I made used identifier color with a light reddish/pinkish background, which is probably more reasonable. Although you could argue it's not the scanner's job to identify errors, but rather that of a parser, but anyway...). If two rules match a span of characters, JFlex always takes the one that matches a longer span, which in your case must be the error token state.

Assuming this is the issue, you can remove the error state, assuming you have the catch-all rule at the end of it, e.g.

Code: Select all

   .                { addToken(Token.ERROR_IDENTIFIER); }


A state like "ErrorIdentifier" is simply supposed to be an optimization for catching invalid/unexpected tokens or characters in a file, by letting JFlex match as many characters in one pass as it can. The rule above should work, but might produce many tokens if the user types lots of "invalid" characters for your source file (though this really still shouldn't produce any noticeable performance issues).

Note also if you're not comfortable marking *anything* you haven't explicitly scanned for as an "error" token, you can use the catch-all rule above and mark them as Token.IDENTIFIER instead of Token.ERROR_IDENTIFIER. Personally I flip-flop as to which I prefer. Usually it just depends on how comfortable I am with my knowledge of the syntax of a language, and how comprehensive my other scanning rules are.

Edit #3: Had a crack at the bracket matching but didn't finish it. Apparently, it needs little different handling than what's in place right now. The matching works fine after the change, it's the repaint that's not the same. So, when I saw that I needed to change some things over more than one class, I thought that it'd be better to have your solution instead, since I didn't want to re-apply what I did should RSTA get updated in the future, and more likely than not, your solution would be more elegant

Haven't had a chance to look into it yet, but I will soon...

Edit #4: For some reason, autocomplete's autoactivation isn't kicking in. I set it to true, made sure autocomplete is enabled and set the delay to 1200 to 2000ms over several tests. The popup just doesn't appear, though hitting the trigger key pulls up the popup windows just fine. Any ideas on what I'm doing wrong?

Have you called setAutoActivationEnabled(boolean)? I'll update the Javadoc so that the auto-activation related methods all @see one another, and mention this in the description.

Edit #5: BTW, robert, your build.xml files had a typo on the debuglevel line. It said "var" instead of "vars". I saw it on both the RSTA and Autocomplete checkouts.

Thanks, I recently discovered this as well. I think I fixed it in the 2.0.5 release, but it could have been afterwards. Funny enough, it doesn't seem to stop the JDK from compiling the projects on Windows, but it does on OS X. To be fair though, on Windows I use a genuine 1.4.2_18 JDK to compile, to avoid having to set the boot-classpath and keep 1.4 compliance. OS X Lion doesn't even support Java 1.4 or 1.5, so I have to build with the 1.6 JDK there. It could be that they've made later JDK's more strict with the debug parameter names, or maybe it's the different JDK's on different OSes. Who knows! In any case, it should be fixed now.

[Kuriel]

Thanks for the new info, I'll be sure to try those out to get my project to a finish!

I did do setAutoActivationEnabled to true but still no go. I followed the example on the project page as well. I look forward to the next update In the meantime, I'll work on your other suggestions. Thanks again.

Edit: The tip for Token.ERROR_IDENTIFIER was spot on. Thank you.

[Kuriel]

As for the string-style MLC, I'm after the multiline coloration; as it is, the quotes only color a single line, not the span of what they enclose if, like say, the line is broken by a carriage return/line wrap. It's all to catch an un-closed string so the user can immediately see they're missing a quote.

HTML already does this, at least, for HTML attributes. Are you asking for this with a string type outside of HTML tag attributes? This is doable by copying the implementation that's already there.

Happy holidays robert,

I studied and applied the implementation as best I could to work with single and double quotes but it's generating a serious error. If I type like say, "abcdef and hit Enter, the whole line get's replicated, and this happens each time I make a new line. I thought it was because it couldn't work without being confined like how HTML implementation triggers when a line is started with a <, but even when I tried an exact copy of it, the error still occurs.

If you don't mind, here's a stripped down .flex of what I've done so far. I took out the HTML stuff (but adding it in the future) and the reserved words and functions to shorten it a bit more. Could you point out what I've done wrong?

Code: Select all

/*
 * Generated on 12/24/12 5:39 PM
 */
package org.fife.ui.rsyntaxtextarea.modes;


import java.io.*;
import javax.swing.text.Segment;

import org.fife.ui.rsyntaxtextarea.*;


/**
 *
 */
%%

%public
%class CustomScriptTokenMaker
%extends AbstractJFlexCTokenMaker
%unicode
/* Case sensitive */
%type org.fife.ui.rsyntaxtextarea.Token


%{
   public static final int IN_MACRO_STRING                  = -1;
   public static final int IN_MACRO_CHAR                  = -2;
   public static final int INTERNAL_IN_MACRO               = -3;


   /**
    * Constructor.  This must be here because JFlex does not generate a
    * no-parameter constructor.
    */
   public CustomScriptTokenMaker() {
   }

   /**
    * Adds the token specified to the current linked list of tokens as an
    * "end token;" that is, at <code>zzMarkedPos</code>.
    *
    * @param tokenType The token's type.
    */
   private void addEndToken(int tokenType) {
      addToken(zzMarkedPos,zzMarkedPos, tokenType);
   }
   
   /**
    * Adds the token specified to the current linked list of tokens.
    *
    * @param tokenType The token's type.
    * @see #addToken(int, int, int)
    */
   private void addHyperlinkToken(int start, int end, int tokenType) {
      int so = start + offsetShift;
      addToken(zzBuffer, start,end, tokenType, so, true);
   }


   /**
    * Adds the token specified to the current linked list of tokens.
    *
    * @param tokenType The token's type.
    */
   private void addToken(int tokenType) {
      addToken(zzStartRead, zzMarkedPos-1, tokenType);
   }


   /**
    * Adds the token specified to the current linked list of tokens.
    *
    * @param tokenType The token's type.
    * @see #addHyperlinkToken(int, int, int)
    */
   private void addToken(int start, int end, int tokenType) {
      int so = start + offsetShift;
      addToken(zzBuffer, start,end, tokenType, so, false);
   }


   /**
    * Adds the token specified to the current linked list of tokens.
    *
    * @param array The character array.
    * @param start The starting offset in the array.
    * @param end The ending offset in the array.
    * @param tokenType The token's type.
    * @param startOffset The offset in the document at which this token
    *        occurs.
    * @param hyperlink Whether this token is a hyperlink.
    */
   public void addToken(char[] array, int start, int end, int tokenType,
                  int startOffset, boolean hyperlink) {
      super.addToken(array, start,end, tokenType, startOffset, hyperlink);
      zzStartRead = zzMarkedPos;
   }


   /**
    * Returns the text to place at the beginning and end of a
    * line to "comment" it in a this programming language.
    *
    * @return The start and end strings to add to a line to "comment"
    *         it out.
    */
   public String[] getLineCommentStartAndEnd() {
      return new String[] { "<!--", "-->" };
   }


   /**
    * Returns the first token in the linked list of tokens generated
    * from <code>text</code>.  This method must be implemented by
    * subclasses so they can correctly implement syntax highlighting.
    *
    * @param text The text from which to get tokens.
    * @param initialTokenType The token type we should start with.
    * @param startOffset The offset into the document at which
    *        <code>text</code> starts.
    * @return The first <code>Token</code> in a linked list representing
    *         the syntax highlighted text.
    */
   public Token getTokenList(Segment text, int initialTokenType, int startOffset) {

      resetTokenList();
      this.offsetShift = -text.offset + startOffset;

      // Start off in the proper state.
      int state = Token.NULL;
      switch (initialTokenType) {
         case Token.COMMENT_MULTILINE:
            state = MLC;
            start = text.offset;
            break;
         case INTERNAL_IN_MACRO:
            state = MACRO;
            start = text.offset;
            break;
         case IN_MACRO_STRING:
            state = STRING;
            break;
         case IN_MACRO_CHAR:
            state = CHAR;
            break;
         default:
            state = Token.NULL;
      }

      s = text;
      try {
         yyreset(zzReader);
         yybegin(state);
         return yylex();
      } catch (IOException ioe) {
         ioe.printStackTrace();
         return new DefaultToken();
      }

   }


   /**
    * Refills the input buffer.
    *
    * @return      <code>true</code> if EOF was reached, otherwise
    *              <code>false</code>.
    */
   private boolean zzRefill() {
      return zzCurrentPos>=s.offset+s.count;
   }


   /**
    * Resets the scanner to read from a new input stream.
    * Does not close the old reader.
    *
    * All internal variables are reset, the old input stream
    * <b>cannot</b> be reused (internal buffer is discarded and lost).
    * Lexical state is set to <tt>YY_INITIAL</tt>.
    *
    * @param reader   the new input stream
    */
   public final void yyreset(Reader reader) {
      // 's' has been updated.
      zzBuffer = s.array;
      /*
       * We replaced the line below with the two below it because zzRefill
       * no longer "refills" the buffer (since the way we do it, it's always
       * "full" the first time through, since it points to the segment's
       * array).  So, we assign zzEndRead here.
       */
      //zzStartRead = zzEndRead = s.offset;
      zzStartRead = s.offset;
      zzEndRead = zzStartRead + s.count - 1;
      zzCurrentPos = zzMarkedPos = zzPushbackPos = s.offset;
      zzLexicalState = YYINITIAL;
      zzReader = reader;
      zzAtBOL  = true;
      zzAtEOF  = false;
   }


%}

// Custom specifications
CustomFunctions         = (   "I took them out to shorten the sample")
CustomReservedWords      = ( "shortened_sample" )
CustomReservedWords2         = ( "shortened_sample2" )
Annotation               = ("@"{Identifier}?)
CustomOperators         = ("!"|"&&"|"*"|"+"|"-"|"/"|":"|"<"|"="|"=="|">"|"||")

Letter                     = [A-Za-z]
LetterOrUnderscore            = ({Letter}|"_")
NonzeroDigit                  = [1-9]
Digit                     = ("0"|{NonzeroDigit})
HexDigit                     = ({Digit}|[A-Fa-f])
EscapedSourceCharacter            = ("u"{HexDigit}{HexDigit}{HexDigit}{HexDigit})
NonSeparator                  = ([^\t\f\r\n\ \(\)\{\}\[\]\;\,\.\=\>\<\!\~\?\:\+\-\*\/\&\|\^\%\"\']|"#"|"\\")
IdentifierStart                  = ({LetterOrUnderscore}|"$")
IdentifierPart                  = ({IdentifierStart}|{Digit}|("\\"{EscapedSourceCharacter}))

LineTerminator            = (\n)
WhiteSpace               = ([ \t\f]+)

/* No char macros */
/* No string macros */

MLCBegin               = "<!--"
MLCEnd                  = "-->"

/* No documentation comments */
/* No line comments */

IntegerLiteral            = ({Digit}+)
HexLiteral               = (0x{HexDigit}+)
FloatLiteral            = (({Digit}+)("."{Digit}+)?(e[+-]?{Digit}+)? | ({Digit}+)?("."{Digit}+)(e[+-]?{Digit}+)?)
ErrorNumberFormat         = (({IntegerLiteral}|{HexLiteral}|{FloatLiteral}){NonSeparator}+)
BooleanLiteral            = ("true"|"false")

Separator               = ([\(\)\{\}\[\]])
Separator2               = ([\;,.])

Identifier               = ({IdentifierStart}{IdentifierPart}*)

URLGenDelim            = ([:\/\?#\[\]@])
URLSubDelim            = ([\!\$&'\(\)\*\+,;=])
URLUnreserved         = ({LetterOrUnderscore}|{Digit}|[\-\.\~])
URLCharacter         = ({URLGenDelim}|{URLSubDelim}|{URLUnreserved}|[%])
URLCharacters         = ({URLCharacter}*)
URLEndCharacter         = ([\/\$]|{Letter}|{Digit})
URL                  = (((https?|f(tp|ile))"://"|"www.")({URLCharacters}{URLEndCharacter})?)

%state MACRO
%state MACRO_IDENTIFIER_CHECK
%state STRING
%state CHAR
%state MLC
/* No documentation comment state */
/* No line comment state */

%%

<YYINITIAL> {
   "["                        { addToken(Token.MARKUP_TAG_DELIMITER); yybegin(MACRO); }
   "["({Letter}|{Digit})+         {
                             int count = yylength();
                             addToken(zzStartRead,zzStartRead, Token.MARKUP_TAG_DELIMITER);
                             zzMarkedPos -= (count-1); //yypushback(count-1);
                             yybegin(MACRO_IDENTIFIER_CHECK);
                           }
   /* Keywords */
   { CustomReservedWords }      { addToken(Token.RESERVED_WORD); }

   /* Keywords 2 (just an optional set of keywords colored differently) */
   { CustomReservedWords2 }      { addToken(Token.RESERVED_WORD_2); }

   /* Functions */
   { CustomFunctions }          { addToken(Token.FUNCTION); }

   {BooleanLiteral}            { addToken(Token.LITERAL_BOOLEAN); }
   
   {LineTerminator}            { addNullToken(); return firstToken; }

   {Identifier}               { addToken(Token.IDENTIFIER); }

   {WhiteSpace}               { addToken(Token.WHITESPACE); }

   /* Annotations. */
   {Annotation}               { addToken(Token.ANNOTATION); }

   /* String/Character literals.
   '                     { addToken(Token.LITERAL_CHAR); yybegin(CHAR); }
   \"                     { addToken(Token.LITERAL_STRING_DOUBLE_QUOTE); yybegin(STRING); } */

   /* Comment literals. */
   {MLCBegin}   { start = zzMarkedPos-4; yybegin(MLC); }

   /* Separators. */
   "@@" |
   "!!" |
   {Separator}               { addToken(Token.SEPARATOR); }
   {Separator2}            { addToken(Token.IDENTIFIER); }

   /* Operators. */
   {CustomOperators}            { addToken(Token.OPERATOR); }

   /* Numbers */
   {IntegerLiteral}            { addToken(Token.LITERAL_NUMBER_DECIMAL_INT); }
   {HexLiteral}               { addToken(Token.LITERAL_NUMBER_HEXADECIMAL); }
   {FloatLiteral}               { addToken(Token.LITERAL_NUMBER_FLOAT); }
   {ErrorNumberFormat}            { addToken(Token.ERROR_NUMBER_FORMAT); }

   /* Ended with a line not in a string or comment. */
   <<EOF>>                     { addNullToken(); return firstToken; }

   /* Catch any other (unhandled) characters. */
   .                        { addToken(Token.IDENTIFIER); }

}

<MACRO> {
   {Identifier}            { addToken(Token.MARKUP_TAG_ATTRIBUTE); }
   {WhiteSpace}            { addToken(Token.WHITESPACE); }
   [\"]                  { start = zzMarkedPos-1; yybegin(STRING); }
   [\']                  { start = zzMarkedPos-1; yybegin(CHAR); }
   "]"                     { yybegin(YYINITIAL); addToken(Token.MARKUP_TAG_DELIMITER); }
   <<EOF>>                  { addToken(zzMarkedPos,zzMarkedPos, INTERNAL_IN_MACRO); return firstToken; }
}

<MACRO_IDENTIFIER_CHECK> {
   {Identifier}         { yypushback(yylength()); yybegin(MACRO); }
   .                  { /* Shouldn't happen */ yypushback(1); yybegin(MACRO); }
   <<EOF>>               { addToken(zzMarkedPos,zzMarkedPos, INTERNAL_IN_MACRO); return firstToken; }
}

<CHAR> {
   [^\']*                  {}
   [\']                  { yybegin(MACRO); addToken(start,zzStartRead, Token.MARKUP_TAG_ATTRIBUTE_VALUE); }
   <<EOF>>                  { addToken(start,zzStartRead-1, Token.MARKUP_TAG_ATTRIBUTE_VALUE); addEndToken(IN_MACRO_CHAR); return firstToken; }
}


<STRING> {
   [^\"]*                  {}
   [\"]                  { yybegin(MACRO); addToken(start,zzStartRead, Token.MARKUP_TAG_ATTRIBUTE_VALUE); }
   <<EOF>>                  { addToken(start,zzStartRead-1, Token.MARKUP_TAG_ATTRIBUTE_VALUE); addEndToken(IN_MACRO_STRING); return firstToken; }
}


<MLC> {
   [^hwf\n-]+            {}
   {URL}               { int temp=zzStartRead; addToken(start,zzStartRead-1, Token.COMMENT_MULTILINE); addHyperlinkToken(temp,zzMarkedPos-1, Token.COMMENT_MULTILINE); start = zzMarkedPos; }
   [hwf]               {}
   \n                  { addToken(start,zzStartRead-1, Token.COMMENT_MULTILINE); return firstToken; }
   {MLCEnd}            { yybegin(YYINITIAL); addToken(start,zzStartRead+3-1, Token.COMMENT_MULTILINE); }
   "-"                  {}
   <<EOF>>               { addToken(start,zzStartRead-1, Token.COMMENT_MULTILINE); return firstToken; }

}


/* No documentation comment state */

/* No line comment state */


BTW, the MACRO_IDENTIFIER_CHECK part and parts I copied in later to try solving the problem but with or without it, the error still occurs.

About the autoActivationEnable for AutoComplete not working as it should, would it be that putting my reserved words/functions in as BasicCompletion entries have anything to do with it? As I've mentioned, I just used the example given at the site.

Thanks!