The unit is designed to be left on so one would assume that if it were not in use that it would be in a low power state and the adapter would not have to supply maximum power, therefore less heat.
When you say really warm, do you mean so warm that you can't keep your hand on it (use back of hand to try) i.e hot, very uncomfortable, can keep my hand there?
Most adapters give off some heat but not that you can't touch it comfortably.
If it is too hot to keep your hand there (at least for a little bit -and please be careful when you try), there is probably something wrong with the adapter and it needs to be replaced.
Be sure that you get the correct adapter either from the manufacturer or if you wish to buy generic, that it has the same specs and plug type (both ends) as the original. The specifications are on the adapter.
This link has info regarding Roku 3 power and other issues.
These two quotes taken from the link are of interest in that it shows that the Roku 3 could be still operating, (thus drawing most power and emitting most heat from the adapter), if you inadvertently did not disconnect the session when you finished watching. (The first quote was not using the Roku 3 remote control)
At the cottage, I use a Roku 3 for Netflix. Out of habit on a few occasions, I simply did the same thing, hit the off button on the Harmony without stopping the Netflix show (Sons of Anarchy) and happily went to bed. I have a 30GB internet plan and half way through the month, I got a warning message from my ISP saying I had used over 90% of my quota. Considering I was only using it on weekends and 1 week's vacation, I was quite surprised.
You're probably correct as you were watching a series at the time. It probably just continued to play. It's always best to exit any channel to end the session when you're finished watching. I'm suspecting that the Roku happily continued showing episodes, not knowing that nothing was receiving it.
If you feel uncomfortable about it, switch it off. In fact if you turn it off by unplugging the adapter all you'll lose is some of its functionality as in automatic updates of the channels and software.