By drivers I assume you mean the headphone speakers. It is most unlikely that this is were the problem lies. You mention drivers (plural) in your question therefor I take it that both distort. For both of them to be faulty will be rare. I don't know if you have listened to each 'driver' individually and found that both distort or o ly one.
The common factor here then is what 'drives' the drivers. In other words, what are you using as the input to the headphones. Is it an amplifier, a stereo, a Bluetooth receiver, a cellphone, a tablet, a laptop, etc.
Depending on the latter's output impedance (ohms) and power (watts), and your phone's equivalent specs, there may be a mismatch. Too much power into speakers rated at a lower power will cause distortion. Some speakers have high impedance (in the kilo-ohm range) input needs while others have quite low (4-50 ohm) input needs. A mismatch here will cause distortion. For example a high input impedance load (speaker) fed by a low impedance device will require more power to produce an acceptable output with resulting distortion.
Best way to test is by plugging your headphones into different sources and hear if both of them distort.
What is also not clear: Are your XM5s Sony's noise cancelling phones, in which case their active impedance is 48 ohms while non-active impedance is 16 ohms.
Maybe you can clarify some of the unknowns mentioned above.
3 