Today, I just decided to run a usb-c cable along with the can cable to the toolhead. Everything is working perfectly now, a shame I had to give up on can tho.
You can update to 6.6 kernel by using the rpi-update command. For me that successfully resolved any bytes_invalid issues.