Are you using any usb hubs? or multiple usb ports on your pi?
This sounds quite similar to an issue i was working though.
Im running a multi mcu setup (6 mcu’s all via usb) and as part of this, i have found that some usb hubs (known as STT or single transaction translator) can exhibit blocking behaviors with the strict timing requirements of klipper.
Ensuring that the USB hubs are MTT (Multiple transaction translator) resolved my issues.
What ive found is that some RPI designs include an internal usb Hub, of which on some of them (and it seems hard to find out exactly which ones) the internal hub is the STT type. You can work around this by only using a single USB port on the RPI connected to a known good MTT type usb hub
Unfortunately, finding MTT type usb hubs seems to be a black art as they dont advertise them as such.