Occasionally there's no audio coming from my speakers on cellular calls

Occasionally I make or receive calls that only have audio going to the other person. I hear nothing from them. It happens with incoming and outgoing calls, and also happens in various locations. It generally occurs on approximately one in 20 calls but on some days it will happen noticably more frequently. Testing with other networks isn't convenient as my home has only one good network (three mobile in the UK). Testing other phones is possible but again not ideal. If anyone has any ideas what may be happening I'd be very grateful to them.