Sounds like you have either a bad mic or a cold solder joint on the IC. The iPhone 4 has two mics one is used for hand held calls the second to filter the ambient noise in speaker phone mode. In your case I suspect the second mic is working and your primary mic is bad. Or, the A/D chip has a bad joint on the primary mic leg.
First try replacing the mic board and if that doesn't work you'll need to look into getting a new logic bd or finding someone with the needed tools to repair the solder joint. The contacts you need to get to on the IC are not exposed. Which is why its best just to replace the board.