You must be doing something to enter "composed text mode," which is one of those magical things still private/package-private in Swing. Perhaps your keyboard is a non-English one? I'm not sure under what scenarios this occurs.
In any case, I believe the only way around this, and still allow for proper input from all input devices, is to remove the hard requirement on ConfigurableCaret. Can you try downloading and using this jar:http://fifesoft.com/temp/rsyntaxtextarea.jar
and see if that fixes the problem? The only strange thing that may happen is you see the caret change appearance slightly, from its two pixel-wide bar (by default at least) to the standard Swing 1-pixel caret, *if* and *when* you're going into "composed text mode" for whatever reason. Insert/overwrite modes and all other functionality should still work as expected though, I believe.
I haven't committed this change yet; please let me know whether it fixes the problem for you and I'll push it to the SVN repository.