Sounds to me as though the headphone jack has come loose due to impact damage from its previous owner. My typical response would be to send it to Apple and they would most likely fix it for you free under warranty, however since you didn't buy it yourself then there isn't much choice other than taking a look at it yourself or send it to a phone repair store and let them do it for you.
Here is a link to a repair guide for the headphone jack.
iPhone 4 Headphone Jack & Volume Control Cable Replacement
Please note that it is a rather difficult repair and if you are not confident with iPhones internally then let someone else that knows what they are doing to do it for you :)