-A- Rule out the simple things first:
1/ Charge your new battery beyond 50%;
2/ Gently clean the Home Button flex cable connector (on the cable and the board) using a cotton swab and 90%+ isopropyl alcohol;
3-a/ Power cycle your phone = disconnect the battery, press on Power Button for 15 seconds, reconnect your battery.
AND/OR (since you already replaced your battery = power cycled your device):
3-b/ Reboot your iPhone by pressing & holding Home + Power Button for ~10 seconds. If the iPhone has not restarted then, press once on Power Button to restart it;
4/ Adjust your time settings by connecting to Wi-Fi or iTunes and make sure the time and date are correct;
5/ Try your home button again.
-B- If the above do not fix it, you may have damaged something around the battery connector or along the board on the battery side during the initial repair. Think back: did you lever anything AGAINST the board to lift the battery? Did you pry between the battery and the board?
6/ Use a magnifying glass to examine the logic board, specially around (and east) of the battery connector. Look for anything broken, missing, or part removed from the board. Let us know what you see. Note that you may need 20x or more magnification to be able to see some of the potential damage. Also examine the home button flex cable itself.