Replaced display touchscreen only works after 2nd wake up

After I replaced the display of my SE 1 I experience this strange behavior:

After having the phone sit silent for a while, I hit either the home or power button to wake up the display. It should now receive touch events, e.g. slide left to enable the camera. This touch event does not work. No tapping, no sliding, no input at all. Hitting the home button again lets the code input field appear. But again no touch events.

So I can either wait for the display to turn off again or force by hitting the power button, then start over right away by either hitting power or home button. This time touch works for the camera gesture and after hitting home again it also takes the unlock code.

The worst part about it is that I am also not able to answer calls, because I would need to slide right which doesn't work.

Why is this happening? I know that replacing the display disables touch ID but not this. Is this due to a setting I have missed? Has anyone else experienced this? What can I do to fix this?