If you normally keep it in your pocket, I suggest checking the jack for lint buildup. I have an iPhone 3G, myself, and once I got the lint out (straightened paperclip and compressed air), the problems went away! It's so simple, I had kicked myself for never thinking of it as it had been bugging me for months!
It's really amazing to think that there was so much lint in there that it was pushing the plug out of alignment...but there it was.
If this, by chance, doesn't solve the problem, it may be a short or fault in the jack or it's wiring. However, that's rather unlikely. The i-series mobile devices, regardless of generation, are pretty solidly built.
I hope this helps!