What's wrong with my N64 controller?

Ok so I've had this controller for a while now and I decided to replace the loose joystick from it from another controller with a tight one. So I took it out, greased the joystick so it doesn't wear down, and placed into the other controller. I then popped in Super Mario 64 and it worked beautifully. But after a couple of minutes the game thought the joystick was pointing up, so I turned it off and on again thinking it was a glitch. But It kept happening. Anyone got an idea of what happened and how or if I can even fix it?