Maxed out memory upgrade, but still experiencing memory drain

You are describing a memory leak within the application (or some other app also running here). You'll need to run Activity Monitor to see whats leaking. A dual monitor setup should not be an issue here unless the OS driver has a memory leak (doubtful).

But before we go down that path lets make sure your system is correctly running here. Do you have at least 1/4 to 1/3 of the systems primary drive free? If not you might just be running into a drive constraint as the app needs work space to run that it can't get (HD space Vs RAM).

OK, you have lots of room then we need to narrow down whats leaking:

Shut down your system (cold reboot here) restart it and with nothing running startup Activity Monitor. Let it run as long as you can (a good hr) watch what is using CPU cycles and take a screenshot of the heavily active processes. This is your baseline. With Activity Monitor still running! Startup your Think or Swim app. Again let it run without any input for at least an hour take another screenshot. This is the Apps baseline. Now leave Activity Monitor running put it to the side on your desktop so you can still see it and can monitor your Apps processes. When you get the message again stating the App ran out of resources snap a screenshot of what the Activity Monitor shows (don't exit the App!). Now exit the App and take one more screenshot to see if some resources are not releasing. Don't touch anything! wait a good 1/2hr and do another screenshot. Make sure you document which screenshot was taken when.

Collect your screenshots and send them off to your Apps support group at TD Ameritrade as they can then see your issue. If the resource hog is not the App then you'll need to send the outputs to Apple.