NullPointerException for ClojureTokenMaker

[jrose]

I am getting a NULL pointer error a lot when trying to use RSyntaxTextArea with Clojure files. This bug started after I set the line wrapping to true. Without line wrapping it doesn't happen. It appears to be an issue with the ClojureTokenMaker class.

There are actually a few other things I wanted to add to the Clojure flex file, is there somewhere I can see how to compile the flex file with JFlex? I have checked out the TokenMakerMaker but I was hoping to update some of the regex for some tokens.

Here is the error I am getting

#<EventDispatchThread Thread[AWT-EventQueue-0,6,main]>
java.lang.NullPointerException
at org.fife.ui.rsyntaxtextarea.TokenMakerBase.addToken(TokenMakerBase.java:114)
at org.fife.ui.rsyntaxtextarea.modes.ClojureTokenMaker.addToken(ClojureTokenMaker.java:1421)
at org.fife.ui.rsyntaxtextarea.modes.ClojureTokenMaker.addToken(ClojureTokenMaker.java:1404)
at org.fife.ui.rsyntaxtextarea.modes.ClojureTokenMaker.yylex(ClojureTokenMaker.java:1852)
at org.fife.ui.rsyntaxtextarea.modes.ClojureTokenMaker.getTokenList(ClojureTokenMaker.java:1478)
at org.fife.ui.rsyntaxtextarea.RSyntaxDocument.getTokenListForLine(RSyntaxDocument.java:369)
at org.fife.ui.rsyntaxtextarea.WrappedSyntaxView$WrappedLine.modelToView(WrappedSyntaxView.java:988)
at org.fife.ui.rsyntaxtextarea.WrappedSyntaxView.modelToView(WrappedSyntaxView.java:504)
at org.fife.ui.rsyntaxtextarea.WrappedSyntaxView.yForLineContaining(WrappedSyntaxView.java:835)
at org.fife.ui.rsyntaxtextarea.RSyntaxTextAreaUI.yForLineContaining(RSyntaxTextAreaUI.java:237)
at org.fife.ui.rtextarea.RTextAreaBase.yForLineContaining(RTextAreaBase.java:1137)
at org.fife.ui.rtextarea.LineNumberList$Listener.caretUpdate(LineNumberList.java:641)
at javax.swing.text.JTextComponent.fireCaretUpdate(JTextComponent.java:391)
at org.fife.ui.rtextarea.RTextArea.fireCaretUpdate(RTextArea.java:583)
at org.fife.ui.rsyntaxtextarea.RSyntaxTextArea.fireCaretUpdate(RSyntaxTextArea.java:732)
at org.fife.ui.rtextarea.RTextAreaBase.processComponentEvent(RTextAreaBase.java:775)
at java.awt.Component.processEvent(Component.java:6155)
at java.awt.Container.processEvent(Container.java:2083)
at java.awt.Component.dispatchEventImpl(Component.java:4737)
at java.awt.Container.dispatchEventImpl(Container.java:2141)
at java.awt.Component.dispatchEvent(Component.java:4565)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:684)
at java.awt.EventQueue.access$000(EventQueue.java:85)
at java.awt.EventQueue$1.run(EventQueue.java:643)
at java.awt.EventQueue$1.run(EventQueue.java:641)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
at java.awt.EventQueue$2.run(EventQueue.java:657)
at java.awt.EventQueue$2.run(EventQueue.java:655)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:654)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

[robert]

Thanks for the bug report, I'll take a look.

Unfortunately, you'll need JFlex to generate the *TokenMaker classes shipped with RSTA, and there's really no way around it. Were the features you wanted to add to the ClojureTokenMaker fixes that everyone using it would appreciate, or were they for a custom language (dialect?) or other use most people wouldn't need? If it's the former I'd be happy to add your changes to the shipped ClojureTokenMaker. I personally don't use Clojure so there may well be issues with it.

[jrose]

Thanks for the reply. I am psyched to hear you would be willing to update the ClojureTokenMaker class. The updates I am hoping for are definitely all for standard Clojure and would be beneficial for all Clojurians! I would be more than happy to put together a list of keywords, and forms etc., if it helps. I have noticed bugs with a few key things including strings, clojure char defs, and some common forms.

I have tried to add a few more Clojure files with mixed results. They include: (AutoCompletion) ClojureCompletionProvider, ClojureFunctionCompletion, ClojureAutoCompletion, and a ClojureFoldParser. The auto complete classes seem to working pretty well but my ClojureFoldParser does not. Truth be told, i just copied the CurlyFoldParser and tried to refactor it to work for parenthesis instead of curly's. I am bummed to say it doesn't work though...

Just let me know what info etc., I can provide to help update the Clojure features. It would be awesome to to help contribute to the RSTA Clojure features.

jon

[robert]

Any syntax issues - missed keywords, parsing errors, etc., would be great. I could probably look that stuff up myself, but it's much more likely to be done right if someone actually using it (like you ) verifies things for me.

Also, it's fantastic that you're working on code completion for Clojure! If you want to contribute any code you can just email it to me; I'll be happy to add it to the project (BSD-licensed). Or if you want to keep a different license or keep personal control over the source, I can simply link to your project or repository. Or if you want to keep it to yourself, that's fine too.

If the folding logic for Clojure is really as simple as parens vs. curlys, we should definitely be able to get that to work. If you can send me what you had (or post it) I can take a look. I'd imagine you should be able to just extend CurlyFoldParser and override isLeftCurly(Token) and isRightCurly(Token) to return true on left- and right-parens, respectively.

[jrose]

I would be more than happy to send you my Clojure completion code. Just let me know what your email is, I can't find it on the site or your profile.

All of the classes I created consist of minor modifications to previous AutoCompletion classes so there very well might be bugs but I haven't been having any issues. The only portion of the Clojure completions that is in Clojure is my clojure-completion-builder.clj class. Clojure has some really nice meta data functions, like Java reflection but more descriptive with stuff like function source, etc. The goal for my project is to basically to have each open Clojure project dynamically create it's own CompletionProvider containing only the vars and functions the project could use. I am hoping to not have the bloat that eclipse auto completion does by not including completions from outside of the project dependencies. Currently, this completion-builder class calls a function (all-ns) which returns all of the libraries (from jars) available in the current namespace. I am pretty sure that AutoCompletion is doing something similar for python completions from what I have read in the source etc. so let me know if this is something I should aim to include. Otherwise I could just generate the xml completions like the C completions have.

As for the curly parser, I tried something similar but realize now it was the wrong way to go. I think I tried to isLeft/RightParen to Token but with no success. Clojure forms just have everything inside of parens

(+ 1 1)
(println "foobar")

It seems that the CurlyFold parse would need to be updated to look for parens and the function name inside of the left parent instead of outside a curly. If you point me in the right direction I would be happy to look into updating this my self if it help

[jrose]

I have just added a new Clojure syntax wish list wiki page to my project's github repo. I also thought I would make a post on the Clojure mailing list for suggestions. I think a very up to date reference for keywords and forms would be the VimClojure syntax file https://bitbucket.org/kotarak/vimclojure/src/e85c389440a3/vim/src/main/vim/syntax/clojure.vim.

There are a few other key Clojure features that would be amazing to have in RSTA that I could provide more info on. Just let me know how I can help organize syntax info, forms, etc., to make it is easy as possible for you.

[robert]

Great, this is a lot of information to look at! I'll parse through it, and I've also emailed you so you can send me anything pertinent.