Yes, you can replace your current defective graphics card with a 2 GB unit. Here's the info for it:
- AMD Radeon HD 6970 2 GB - Apple P/N 661-5969
Be aware this part is expensive! Around $750 US for the 2 GB, and the 1 GB board will be a bit cheaper by about $100 US:
- AMD Radeon HD 6970 1 GB - Apple P/N 661-5968
As to putting something more powerful sadly, thats not an option as Apple never offered anything that uses the MXM II interface your system uses. The newer 'Thin Series' systems don't use the MXM card but instead solder the GPU chip and support logic to the main logic board.
We don't have a guide for replacing the GPU board, but this guide will get you better than 80% there: Installing iMac Intel 27" EMC 2429 Dual Hard Drive Kit. Once you get to Step 33 you'll need to finish disconnecting the cables to free up the logic board, then once the board is out place it on a ESD mat and make sure you are using a wrist strip while working on the logic board (both properly grounded). Remember, winter time is static time! (ESD damage is possible)
Carefully remove the remaining screws which hold the card & heat sink on. If you do need to take it off the heatsink from the GPU you'll need to clean off the thermal paste and apply a fresh coat as well as replace the gapping thermal putty on the memory chips with thermal pads. Do be careful with the black shim as it is brittle! As you'll need to reuse it.
If you haven't worked on an iMac before you might want to find someone with the needed skills and tools as this is a big job!