it's possible that you have solder jumpered between a couple of the battery pads on the motherboard..
Looking at the battery connector, the right pad is ground.. the other two should have voltage. And, this voltage should be equal to what the battery is new out of the box (unless discharged).. This battery is 3.8 volts normally, but I have seen batteries with voltages of up to 4 volts.
Once soldered in place and with the ipod OFF *make sure you haven't touched the on/off switch whilst doing the repair*, you should be able to get your volt meter and measure the voltages on the solder points. They should be identical to the readings you got before you installed the battery.
Two, it almost sounds like you have a bad battery. These are cheap, I would almost recommend trying another battery if you have one handy and are good at soldering.. be careful unsoldering the battery, because the more times you do this, the more likely it becomes that you could lift a pad from the motherboard, and then have to do some fancy soldering to fix it.. (and no, gluing the pad back won't fix it either) patience is a virtue, take your time and you will be fine..
any of the batteries that physically fit the 3gs will work fine.. I have always just ordered the generic battery replacements for my iphones and ipods, with no problems encountered.. fortunately, this item is NOT soldered in, however you gotta remove the screens and pull the mb.. make sure the contacts on the mobo are clean, you can use alcohol and the little pink eraser on a number 2 pencil to clean.. do NOT use anything abrasive.. Heck, even using plain paper rubbing on the parts will work in a pinch..