The buttons on the d-pad were broken. It started off being only a minor issue when I had to press the buttons a little harder. Then a buttom got stuck and that input wouldn't stop registering.
I discovered the buttons were ripped off from the rest of the rubber between and that was causing them to get misplaced inside. The d-pad cover was all I needed for it and now it plays just like before.
Careful dismantling the controller it can be easy to lose parts and some of them can break so look up a video.