I'm not sure if I have an answer for you. I created a 50MB, single line text file and tried some stuff out. Loading it in a simple Java app, with just a line-wrapped JTextArea, even the JTextArea cannot handle it. It seems to burn CPU indefinitely (at least, > 5 minutes on a 3GHz CPU). I imagine it's having trouble calculating all of the line break positions. Since RSyntaxTextArea is built on top of JTextArea, if JTextArea can't do it, then RSTA can't either. We naturally do more work than JTextArea does (parsing the text, trying to break at token boundaries and not just word boundaries).
Interestingly, even Windows Notepad can't handle a file like that.
With word wrap enabled, Notepad becomes unresponsive. Never even paints the canvas. I stopped waiting after 2 minutes.
Even with word wrap disabled (should be an easier case), it appears that Notepad inserts line breaks for you, even though there aren't any in the file. These may be "logical" line breaks that aren't saved when you modify the file, but a workaround for a limitation in standard Windows text widgets and line length. But that's just a guess.
Notepad++ seems to have better luck. With word wrap enabled, it can load the file and is generally responsive. Editing is slow, but if you're just viewing the file it should be fine. Strangely enough, if the last logical line is in the viewport, the editor becomes considerably slower, even when just moving around with the arrow keys. Must be some implementation detail.
So in short, I think you're going to be out of luck finding a Java text component that can handle such large files, unless you use one that's not built on top of the standard Swing text package. Most editors in general (not just Java) aren't capable of handling very long lines, since this is a tricky scenario to handle; when looking into using one, you should check whether this is a handled case.