This system has a PCIe ATI Radeon HD 2400 XT graphics processor using PCI Express with 128 MB of GDDR3 video memory.
Video Card options:
- P/N 661-4672 - ATI Radeon HD 2600XT (256 MB of GDDR3)
- P/N 661-4673 - ATI Radeon HD 2400XT (128 MB of GDDR3)
Here's the IFIXIT guide to replace the graphics card: iMac Intel 20" EMC 2133 and 2210 Video Card Replacement. If you jump down to Step 41 you'll see the dedicated RAM chips which can't be replaced.
As to your question of upping the RAM beyond what these cards offer. I don't think thats going to improve things for you in photo editing enough. This systems memory limit (6 GB) and its SATA II (3.0 Gb/s) HD interface are the bigger limiters.
You should try to get a system with SATA III (6.0 Gb/s) or faster I would look at 12 GB or 16GB depending on the image sizes you are working with either a i5 or i7 CPU or newer.
Depending on the size of the image or images if you have more than one open at the same time you can run out of physical RAM.
And yes the system will leverage the drive as virtual RAM. So depending on what the Application (photo or photos) needs, the amount of RAM the system can support and the speed of the drives storage as well as the type of storage (SSD, SSHD or HD) all come into the mix here.
If you are plaining on doing a lot of photo editing you need to first look at the amount of RAM the system can support. This one is limited 6 MB which I think is too little working with a lot of 12 MP or larger hi-res images. Which is why I think you need to look at updating your system Vs upgrading it if you can.
As to upgrading the OS to El Capitain: At this point I don't think that would help you here. Often newer OS end up using more of the systems RAM for its self. So its likely to starve your app needed RAM space which in this system is limited.