The gutter API isn't very clean at the moment. There's no easy way for you to intercept mouse events yet. This is one thing I'd like to add (besides adding custom gutter components). You'll have to hack into the source to add this functionality. In particular, take a look at:org.fife.ui.rtextarea.Gutterorg.fife.ui.rtextarea.IconRowHeader
IconRowHeader is likely what you'll need to edit. It already has code to intercept mouse events, you'll just have to add your own custom logic there.
There's already a Feature Request
to add MouseLIstener support to the Gutter. Feel free to comment with suggestions there, or open a new Feature Request if you think it isn't accurate enough for what you want.