It might be cool if the TokenMaker has methods like isWhitespace(int type) and isSeparator(int type) to allow a little more flexibility.
Might be a good idea. For now, the latest update no longer requires Token.WHITESPACE to paint tab lines; it'll do so with any token type, so you now get them within multi-line comments, Token.IDENTIFIER (if you were to match all "plain text" and whitespace together for performance, for example), or whatever else you throw at it. If the token type isn't Token.WHITESPACE, it'll paint tab lines in the right places for all leading whitespace in the first token of a line, if any. If it IS Token.WHITESPACE though, it'll assume the token really is all spaces and tabs, and skip the scan for leading whitespace for performance reasons (all the built-in scanners use Token.WHITESPACE the "expected" way).
This does mean that, if you use your own token types and not RSTA's, if you have a type with the same value as Token.WHITESPACE, you might get tab lines painted where you wouldn't expect them to be (inside text, etc.). But this is somewhat unlikely, and the workaround would be to simply change that particular token type to a different value, so it would be an easy fix.
I can reproduce your rendering issue in your app, but not in mine with the latest changes (and using the same font as you). Can you update and see if that fixes things (not sure why it would as the actual rendering code hasn't changed, but just to make sure)? If it's still happening, I'll really buckle down and figure it out.
There also might be third round of updates, redoing how these lines are painted, since the current implementation doesn't paint them at all on lines with no tokens on them, which is different, and (albeit slightly) annoying compared to what Scite does.
