To start with you have four memory slots which are broken into Banks (0 & 1). Ideally, you want to use a matched pair within a given bank.
So when you bought your iMac it came with two 4 GB modules (8 GB total RAM). You can see this by going to the About This Mac and tab over to the Memory tab.
You could add in a second bank of RAM of two 4 GB modules which would get you to 16 total RAM which for most people is more than enough. But, you could also go all the way to 32 GB if you needed to!
Here's the Apple T/N on what is possible and needed: iMac: Memory specifications and upgrades.
Hopefully that gives you a framework on what is possible. Don't forget its not just the RAM you need to review when you get to these larger sizes often you need to be running the newest OS and make sure the given App or Apps can support it.
Lastly, don't forget your drive storage having lots of RAM in a system won't help much if the drive is slow or does not have much free space!