Real time control of a Voron 2.4 using a gamepad

For what it is worth, what you’ve described is the intended behavior of Klipper. Klipper is designed to read batches of movement commands (as one would find in a typical “3d print g-code file”) and generate the appropriate movement and acceleration to produce the desired “printed object”. Klipper is not designed to support dynamic low latency user control of motors.

There is some information on how Klipper processes multiple move requests at Kinematics - Klipper documentation

I’m sure it’s possible to alter the Klipper code to do something else, however that’s not something I’ve looked at.

Cheers,
-Kevin