Refactoring them to inner classes might be a good idea (though I am not as sensitive to small anonymous inner classes). Nevertheless
- it's unlikely that given some huge legacy code, it's often unrealistic to do this in one swoop, and
- if you don't hold a reference to the instance, named inner classe will provide you with the same kind of problem.
I also don't think that not holding direct references to objects you create isn't always iffy at all. To the contrary: if the object you listen to has a shorter lifetime than the object creating the listener, it's the most elegant solution to make sure that the listener is destroyed as soon as it isn't needed any longer.