Aargh! You found a terrible bug, thank you! It seems you don't have Perl installed, correct? The Options dialog will never come up if you don't have Perl installed, because it checks your PATH for a Perl installation, and basically forgets to do a null-check if one isn't found. Thus you're getting an uncaught NullPointerException.
I just made a fix; please go download RText 1.3.1 from SourceForge
and see if the Options dialog comes up with it. It should also fix your issue where the tab size change isn't remembered from one RText session to the next.
Again, let me know if you see any more issues with the 1.3.1 release!