I'm not 100% sure why you are experiencing what you are, but I can tell you that the Note 5 charging port is notoriously bad, I have changed 2 this week and about 10 this year...by far, more than any one model just short of the galaxy s5. You can check your cable, make sure it is not wore, tore or damaged in any way. Clean your port. Use 90% or stronger isopropyl alcohol and soft brush like tooth brush, blow out with can air. You could try a wireless charger, but Note 5 don't like them...usually charging for 10 to 20 minutes and stopping. Lastly, reboot your phone...I have read that it can improve charging.
If none of those work for you, you may consider a new charging port. It's a pain in the rear to do, basically taking the entire phone apart including your lcd.