Xbox one controller freezing from headset plugged in

So my Xbox One controller works fine. when I plug in my headset after about 30 seconds my controller freezes up and will not let me control anything I have to remove my batteries put them back in or resync my controller. If I don't plug my headset in the controller works perfect as if I just bought it. But as soon as I plug a headset in and I use Turtle Beach headset the controller doesn't shut off and nothing like that it just freezes if it's locked up is there any answers to my problem.