Your speaker has nothing to do with how people hear you, it is your microphone, and specifically the microphone at the bottom of your phone near your headphone port, because when you put your phone in loudspeaker mode, it uses the microphone by your front camera.
The first step I would do is check the small hole next to your headphone port, is it blocked with dirt? If so, clean it, toothbrush should work well.
If this step doesn't work, you'd have to open the phone, and possibly replace the mesh, or the microphone gasket, as the adhesive that holds the gasket or mesh has moved and has blocked the hole.
If that then doesn't work, you have to then replace the part, it is apart of the charging port assembly so you'd have to purchase one of them. :-)