Depends on what speed 320GB hard drive you have now. A western Digital scorpio black model, one of my favorite model line laptop drives, may be a nice upgrade in performance, as well as seagates newer model hybrid drives.
SSD's are great for application specific needs (programs that use your hard drive a lot) however some people have complained about reliability and data recovery can be extremely difficult and sometimes not possible...... everything is lightening fast, personally i can't justify the cost per GB of storage yet and can wait a few seconds more to open an app.
As far as memory, an 8GB set from GSkill or another good brand would not set you back that much....... esp. if you hang back for a sale....... ( memory not branded as "mac" memory can save you a significant amount of money, it has for me in the past). Leaving you the option of offsetting the cost of an upgrade selling the 4GB of memory currently installed.
I don't know what you use the computer for so I can't say if it would be a waste of time or not.