According to this link,( http://www.amazon.com/gp/product/B00Q50X... ) it seems that the Fire Stick requires >900mA to work effectively. You may wish to check what is printed on the supplied power adapter that came with the Fire Stick as regards the output current to verify this. It should be greater than 900mA (it probably is 1A).
Your TV's USB port seems to be supplying at least this amount of current as you say that there have been no issues.
An issue may (I stress may only) arise if for example you have the Fire Stick being powered by the TV's USB port and you connect another USB device (if you have that many USB ports on your TV) and find that it doesn't work or that the Fire Stick then also doesn't work because the total power that is being supplied to the USB ports is not enough for both devices. This is only conjecture on my part but the Fire Stick is drawing more current than the average USB device.
To be sure you would have to check the TV's specifications and see if you can find out what each USB port's maximum current output is.
As to what damage may occur to the Fire Stick by powering this way. In a way it may be a safer option than using the supplied power adapter. As long as there is enough current being supplied by the USB port (see above proviso) it should be OK. If there is a situation where the current is less it could lead to possible 'data corruption' in the firmware of the Fire Stick if the power was fluctuating.
By using the TV's USB port, in effect the TV is acting as a power surge protector for the Fire Stick. In the event of a wall outlet power surge hopefully the TV will bear the brunt of it and the Fire Stick will be spared. Whereas, unless you have the power adapter connected to the wall outlet via a surge protector there is no protection whatsoever for the Fire Stick if a power surge occurs.
My only suggestion to be really sure of not causing any problems is that if you wish to remove the Fire Stick from the TV is that you turn the power to the TV off before you remove the Fire Stick, rather than removing it from a powered TV.