It could be caused several things.
The power button has a small metal bracket that moves from side to side. Since the noise goes away once you actually press and hold the button, my guess is that the bracket is loose or another component close to the power button is loose.
There are screws in that par of the phone which are used to secure the power button to the frame. One of those could be loose.
There is a small piece of black plastic that covers the back of the proximity sensor and ambient light sensor. It's right above the main earpiece speaker. When I took the phone apart, that part came off in the process. I could see that as the culprit although I'm not sure how pressing or holding the power button would have anything to do with that.
My iPhone 4's power button rattles very slightly when it is not in it's silicone case. t could just be the way the phone is designed.
Getting to the inside of the iPhone 4 to do some investigating is pretty easy. Just follow this guide to get to the components I'm talking about.
If you just bought the phone, you can also take it back to Apple and see if they will do anything about it. It's still under warranty and they should help you.