It sounds like you are looking for low-latency motion control using the Klipper motion planner. I suspect that will be a very challenging task because the Klipper code is not designed that way.
Klipper is designed so that the host software plans motions with precise timing, that movement is then buffered well in advance on the micro-controllers, and then the micro-controllers execute the requested movements using the requested timing.
I’d guess that if you want low-latency arbitrary movement you could accept some amount of latency and try to rework the host code to do less buffering, or alternatively rework the mcu code to directly read the joystick inputs, perform the motion planning, and directly execute the resulting movements. In either case, I expect it would be a bit of work.
Cheers,
-Kevin