Technically you are limited to what the system came with in this case:
- ATI Radeon HD 4670 w 256 MB - Apple P/N 661-5314
- ATI Radeon HD 4850 w 512 MB - Apple P/N 661-5315
The rub is these MXM GPU cards have custom firmware on them so you can't drop just any MXM card into the system, besides the heatsink need to fit the card.
While I commend you are your doggedness here to keep it running. Apple has stated the next release of MacOS won't support these older systems. Believe me I'm not happy either with their direction as of late. So consider this on how much effort and moneys you put into your system.
For now you could try cleaning the old thermal paste and apply a fresh coat (you'll also need to thermal pads for the VRAM). Often refreshing the paste on these older systems does wonders! To prove it for your self get this app: TG Pro I would get the full version for less than $20 its well worth it! Run it for a good week so you have idea on how hot you are running then do a good dust clean and thermal paste redo on both the CPU & GPU's and marvel at the lower running temp!
Even though this will help it still might not fix things. The rub here is the lead free solders Apple and the others were using just didn't handle high heat well.
Some solders grew Tin Whiskers shorting out the pins within the chip to board interconnection. In some the solder degrades (crystalizes) becoming a semiconductor (what we call a cold solder joint).
To fix this correctly you need to completely remove the old solder as the old solder can decompose the new solder! Acting as a seed for recrystallization to start again more quickly. This is what re-balling is all about. Your board may need it. I think this is pushing things and you need the correct equipment to do it. It's not something I would do my self either and it might be too expensive to do here.