The best way to do this is to extend TextEditorPane
instead of RSyntaxTextArea. TextEditorPane provides the extra functionality on top of RSTA for a text editor in an application (loading and saving, encoding handling, etc.). It keeps a "dirty" flag that is set when the editor's contents change, and is cleared whenever the file is saved. This is probably the best way to know if the contents have been modified at any given time.
If you'd really rather just use an RSyntaxTextArea, you can either use a custom javax.swing.event.DocumentListener or javax.swing.event.UndoableEditListener. Attach either to the Document, and any event notification means the Document has been modified.