Perhaps this is your problem, an I quote from ifixit's page for that device:
The iMac Intel 27" EMC 2429 Is largely unchanged from it's predecessor with the exception of some internal lay out and an inconvenient change to the hard drive; the internalizing of the thermal sensor. Moving the thermal sensor into the hard drive has done one major thing for this computer, it has made changing hard drive changes expensive and options limited. The thermal sensor wiring is now part of the SATA cable and is handled by proprietary firmware in the hard drive. In other words, even if you manage to attach an after market drive, you will be stuck with fans running at full blast unless you can find a fan controller.
Maybe if you found fan control software, that would fix it. I don't think replacement will help. If you do decide to replace it, use one of ifixit's guides for that Mac.
I hope this helps.