Fifesoft

Open Source Stuff

Skip to content

Scala enhancements

Post a reply

Smilies
:D :) ;) :( :o :shock: :? 8-) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :!: :?: :idea: :arrow: :| :mrgreen: :geek: :ugeek:
BBCode is ON
[img] is ON
[flash] is OFF
[url] is ON
Smilies are ON
Topic review
   

Expand view Topic review: Scala enhancements

Re: Scala enhancements

Post by Lalit Pant » Mon Nov 26, 2012 2:53 pm

FYI...

Another round of optimizations, and this thing is in production now:
http://www.kogics.net/webkojo

The TokenMaker source:
http://code.google.com/p/kojolite/source/browse/src/main/scala/net/kogics/kojo/lexer/ScalariformTokenMaker.scala

Re: Scala enhancements

Post by Lalit Pant » Fri Nov 23, 2012 6:57 pm

Next steps: test this a lot more, and then think about ways to optimize this approach (currently the whole document till the current line is scanned to find the tokens for a line).


Quick Update:
I have a much more performant version now: http://code.google.com/p/kojolite/sourc ... aker.scala

There are many ways to optimize this further, but performance seems to be at a point where it is good enough (with no noticeable issues). I'm seriously thinking about taking this into production!

Hacks to RSTA to get this working:
Code: Select all
Index: src/org/fife/ui/rsyntaxtextarea/TokenMakerBase.java
===================================================================
--- src/org/fife/ui/rsyntaxtextarea/TokenMakerBase.java   (revision 720)
+++ src/org/fife/ui/rsyntaxtextarea/TokenMakerBase.java   (working copy)
@@ -9,6 +9,7 @@
 package org.fife.ui.rsyntaxtextarea;
 
 import javax.swing.Action;
+import javax.swing.event.DocumentEvent;
 import javax.swing.text.Segment;
 
 
@@ -132,7 +133,10 @@
 
    }
 
-
+    // Doc change hooks added by Lalit
+    public void onInsert(DocumentEvent e) {}   
+    public void onRemove(DocumentEvent e) {} 
+   
    /**
     * Returns whether this programming language uses curly braces
     * ('<tt>{</tt>' and '<tt>}</tt>') to denote code blocks.<p>
Index: src/org/fife/ui/rsyntaxtextarea/TokenMaker.java
===================================================================
--- src/org/fife/ui/rsyntaxtextarea/TokenMaker.java   (revision 720)
+++ src/org/fife/ui/rsyntaxtextarea/TokenMaker.java   (working copy)
@@ -10,6 +10,7 @@
 package org.fife.ui.rsyntaxtextarea;
 
 import javax.swing.Action;
+import javax.swing.event.DocumentEvent;
 import javax.swing.text.Segment;
 
 
@@ -135,7 +136,11 @@
    public Token getTokenList(Segment text, int initialTokenType,
                                  int startOffset);
 
-
+   
+   // Doc change hooks added by Lalit
+    public void onInsert(DocumentEvent e);   
+    public void onRemove(DocumentEvent e);   
+   
    /**
     * Returns whether this language is a markup language.
     *
Index: src/org/fife/ui/rsyntaxtextarea/RSyntaxDocument.java
===================================================================
--- src/org/fife/ui/rsyntaxtextarea/RSyntaxDocument.java   (revision 720)
+++ src/org/fife/ui/rsyntaxtextarea/RSyntaxDocument.java   (working copy)
@@ -114,7 +114,8 @@
     * @param e The change.
     */
    protected void fireInsertUpdate(DocumentEvent e) {
-
+      
+       tokenMaker.onInsert(e);
       /*
        * Now that the text is actually inserted into the content and
        * element structure, we can update our token elements and "last
@@ -187,6 +188,8 @@
     */
    protected void fireRemoveUpdate(DocumentEvent chng) {
 
+       tokenMaker.onRemove(chng);
+      
       Element lineMap = getDefaultRootElement();
       int numLines = lineMap.getElementCount();
 


Feedback welcome...

Re: Scala enhancements

Post by Lalit Pant » Wed Nov 21, 2012 5:20 pm

No worries.

I have been playing with a few different ideas for trying to get XML support to work, and have made interesting progress by hooking up a Scala Lexer with (a slightly hacked version of) RSTA:
https://gist.github.com/4126069

This seems to work correctly (with multilple levels of XML inside Scala inside XML inside Scala)! The downside is that it is pretty inefficient (although it seems to work fine for sub-1000 line scripts).

Next steps: test this a lot more, and then think about ways to optimize this approach (currently the whole document till the current line is scanned to find the tokens for a line).

Re: Scala enhancements

Post by robert » Tue Nov 20, 2012 2:59 am

That one might have to wait until after the upcoming 2.0.5 release this week, unfortunately.

Re: Scala enhancements

Post by Lalit Pant » Sat Nov 17, 2012 6:25 pm

I can see how this is pretty complicated.

Can we do one level of Scala within XML (with no further XML levels within the nested Scala)? This would go a long way.
If you have (almost) working code for this, and are not inclined to try to include this within the next release, would you mind sharing this code - to help get me jump started trying to get some basic XML literal support in there.

Thanks,
- Lalit

Re: Scala enhancements

Post by robert » Sat Nov 17, 2012 4:01 pm

I thought more about this, and I believe it's simply too difficult for RSTA to parse in its current form. Only because you can nest Scala snippets in XML literals and vice versa, to unlimited depths, and each piece of XML can also be of unlimited "depth." RSTA can only remember 32 bits of information per line. What we need to remember is

  • Whether we're in Scala or XML
    • If in Scala, remember if we're in a comment or multi-line string
    • If in XML, remember if we're in an attribute, a tag, or just in a text node
    • If in XML, remember how many still-open XML tags we've come across in the current level.
  • How many levels deep in Scala we are
  • How many levels deep in XML literals we are

I don't see how we can code this information into 32 bits without a tiny (unusable) limit on how nested we can go. For this kind of parsing we really need a stack of states.

Ideas (and patches!) welcome of course, but for the next release (hopefully this week) I don't see XML literals happening in Scala, unfortunately.

Re: Scala enhancements

Post by Guest » Wed Nov 14, 2012 5:34 am

Does Scala accept "incomplete" tags?


No it does not. Without the closing tag, Scala keeps interpreting tokens as xml content, and eventually runs into a compile time error.

Thanks,
- Lalit

Re: Scala enhancements

Post by robert » Wed Nov 14, 2012 3:48 am

Next question: Does Scala accept "incomplete" tags?

scala code:

val anotherOne = <opened>
                    <child1/>
                    <child2/>
                    <child3>Parent is not closed!</child3>


Just want to make sure; I'm not sure if the above scenario is common, but if it's possible and the syntax highlighting doesn't recognize it, it would keep highlighting future (non-markup) tokens as XML tags, which would be bad. If it does not accept this, then we're golden.

Re: Scala enhancements

Post by Guest » Sun Nov 11, 2012 5:19 pm

Code folding and multi-line comment highlighting were added to SVN in commit 699. They'll be in the 2.0.5 release, which will hopefully be in a week.

Great. I'll try them out.

Can you confirm that the first scenario is all that would need to be handled?


I have never used anything other than the first scenario, but it turns out that Scala supports the second scenario too!
For all practical purposes, though, if the first scenario is supported, that should be good enough.

Thanks,
- Lalit

Re: Scala enhancements

Post by robert » Sun Nov 11, 2012 4:25 am

Code folding and multi-line comment highlighting were added to SVN in commit 699. They'll be in the 2.0.5 release, which will hopefully be in a week.

As for XML literals, this will be tougher. RSTA already supports this for JavaScript (e4x), but we have an easy way to know when the XML literal is terminated there - stop on the first ';' character. In the case of Scala, this doesn't seem to be something we can rely on. The only way I can think of implementing this is to require the XML literal to be a single element with child content, i.e. this would be valid:

scala code:

val foo = <html>
             <body>
                Hello world!
             </body>
          </html>


But this would not be:

scala code:

val2 = <script>
          doSomething();
       </script>
       <script>
          doSomethingElse();
       </script>


For all I know, this is also enforced by the actual Scala compiler. If it actually does handle the second scenario, then I congratulate them on writing such an intelligent parser. :P But it's probably more than RSTA's parsing capabilities can currently handle.

Can you confirm that the first scenario is all that would need to be handled? If so, the parser will probably count the number of opening tags and closing tags, and when the net amount == 0, assume the XML literal has ended.

Top

Powered by phpBB® Forum Software © phpBB Group
cron