CR10V3 OCTOPUS conversion - UART errors during print

[laminaytrap]

Printer Model: CR10v3
MCU / Printerboard: BTT Octopus Pro V1.0.1 (F429) Pi 4 Model B Rev 1.2
klippy.log

Newly configured unit. RPi running Octopi V1.0.0 Octoprint Version 1.9.1
Pi is powered by a dedicated Meanwell 5V 3A PSU with the Octopus USB power jumper removed
Two additional 120mm Case fans are powered by SW PWM linked to the RPi core temp.

Mainboard is powered by an LRS350-24 (brand new replacement) with independent leads to the Main Power, Motor Power, and Bed Power of the Octopus. I would have included pics, but apparently I’m new.

There are 8 BTT TMC2209 V1.3 Drivers installed on the board with included heatsinks (only 4 in use at the moment) All jumpers are configured in UART.

The filament runout sensor is not connected, not configured, for testing.

The only non-standard part of the wiring is that there are two 3mm LEDs with resistors in parallel with the hotend fan to provide illumination for my RPiCam V2, which is run via CSI to the Pi in the main enclosure. There is also a USB webcam installed on the system.

This setup worked very well (1000+hrs) on the stock v2.5.2 board until I had a wire wear through and short on the X gantry. This resulted in blowing the traces off the x axis breakout board for the fan wiring, which were subsequently repaired and the printer put back into service.

The printer at this point experienced intermittent shutdowns of the main board, I figured this was due to mainboard damage and I ordered complete replacement hardware (Octopus/2209s/PSU)

After getting Klipper all working properly I still have intermittent failures which I am now trying to troubleshoot. I have reviewed the major topics on this relating to wiring and ESD, but I just want to make sure I am not missing something obvious before I waste many more hours chasing this problem. It seems that in most cases these issues have been quite difficult to resolve.

The printer operates properly for an amount of time, 0.5-3h or so, then produces an error, causing the print to fail.

Klipper is all new to me, so I want to make sure I have not missed something obvious. Any suggestions would be appreciated

I have tried turning off StealthChop and went over the board and pushed each pin of the extruder and end stop connectors down to ensure they were all seated, visual inspection of the leads shows no abnormal bends or damage.

I have tried raising the current for the drivers, removed the hold current parameter, the motor temperatures appear to be reasonable (only extruder gets warm)

The controller enclosure is actively cooled, but not as much as I would like, as I have been unable to complete my new duct prints.

At various points I have seen the following errors on failure of the print: I have tried printing with and without filament, errors regardless.

Transition to shutdown state: Unable to read tmc uart ‘stepper_y’ register GSTAT
Dumping gcode input 50 blocks

Transition to shutdown state: Unable to read tmc uart ‘stepper_z’ register GSTAT
Dumping gcode input 50 blocks

Transition to shutdown state: Unable to read tmc uart ‘stepper_x’ register DRV_STATUS
Dumping gcode input 50 blocks

klippy.log (1.3 MB)

[laminaytrap]

[EddyMI3D]

That switching power supply and the power cord are quite close to the printer board. In particular the power cord is too close.

At first you get quite a lot of EMI.
And on the other hand, you violate some electric regulations with the power cord inside the printer board case.

[laminaytrap]

You are correct on both counts, I didn’t really consider that emi would be that big of a problem between these components, with the exception of the 5v PSU in the case, this is essentially the same arrangement as I was running on the cr10 before, as creality routes the line voltage through a receptacle and switch on the rear panel, right beside the 24v wires.

I could try pulling the 5v and pi outside of the enclosure and test that. I just thought it would be nice to have everything in the same box, dividers and baffles have yet to be printed as the printer is not reliable yet. I tried 3 times to print the fan shrouds and got 10-20% of the way there.

I’ll try a bit of re routing and see how that goes.

…

Removed from mainboard enclosure: RPi, LRS350 and 5V3A PSUs.
Removed from system: USB Cam, RPi Cam, EXP1/EXP2 connections to screen.

Funny sidebar, I forgot that this configuration would leave the Pi without active cooling - temps on Pi are still maintaining 45C.

Thanks.

Dry motion testing underway.

…

7h of motion without errors - I will add back in the cameras and stealthchop and screen and make another attempt.

[laminaytrap]

Did another 9h run overnight with the PiCam, USB Cam and StealthChop Enabled, without errors. I will mock up and attempt printing a new baffle duct and divider design to appropriately segregate the AC lines and PSUs.

[laminaytrap]

Still no resolution, I have tried to run the printer partially assembled to print the new fan shrouds - similar errors again.

Tried running the printer without extrusion again, partially assembled, and failed about the same.

I’ve included a truncated log of the printer starting up, printing, reset and TMC dump.

Will be removing the Meanwell 5V PSU and replacing with the RPi brick PSU to test again.

Is there some way to enable any more detailed logging of the UART failures in order to track my progress in a more granular way?

klippy_no filsment.log (1.2 MB)

[hcet14]

@laminaytrap

I figured this was due to mainboard damage and I ordered complete replacement hardware (Octopus/2209s/PSU)

What PSUs are you using?

Good luck, hcet14

[laminaytrap]

LRS350-24 for the printer and RS15-5 for the Pi, both new. The RS 15-5 voltage was set unloaded to 5.1V, I have not had any indications of any other errors besides failure to communicate with UART.

Both are new. Same errors occured using another LRS350-24.

I should be able to test the Pi on a different supply this afternoon.

Is there any issues with proximity between the Pi and control boards?

That’s the last variable before I give up and mount the PSUs to the table below.

Thanks

[laminaytrap]

Running out of patience on this setup.
Errors in various Configs:
{
Transition to shutdown state: Unable to read tmc uart ‘stepper_z’ register GSTAT

Unable to read tmc uart ‘stepper_z’ register DRV_STATUS

Transition to shutdown state: Unable to read tmc uart ‘stepper_y’ register GSTAT

Transition to shutdown state: Unable to read tmc uart ‘stepper_x’ register DRV_STATUS
}

Enabled this logging and restarted system - I’m lost

Intermittent TMC UART weirdness

If you’re comfortable making code changes, you could try something like this in klippy/extras/tmc_uart.py: --- a/klippy/extras/tmc_uart.py +++ b/klippy/extras/tmc_uart.py @@ -220,10 +220,13 @@ class MCU_TMC_uart: reg = self.name_to_reg[reg_name] if self.printer.get_start_args().get('debugoutput') is not None: return 0 - for retry in range(5): + for retry in range(25): val = self.mcu_uart.reg_read(self.instance_id, self.addr, reg) …

klippy.log (835.4 KB)

[Sineos]

Generally speaking, this often is some wiring / connection issue, but could also be some influence that is messing up the entire communication.
Made a similar experience with TMC2209 on UART and a MAX31856 module in a driver slot on SPI. The MAX module was faulty and dragged all TMC drivers into not working.

@laminaytrap

There are 8 BTT TMC2209 V1.3 Drivers installed on the board with included heatsinks (only 4 in use at the moment) All jumpers are configured in UART.

Not sure if this is a good idea.

[laminaytrap]

Thanks,

I have read that thread, and a few others with similar errors. I intentionally did not install the filament sensor or PT100 for testing in case they added additional complication. I had no idea I would be 20h+ into with no progress. If anything this situation seems to have degraded substantially with the fans off the case, I will put more direct cooling on the boards and remove the extraneous drivers and retest.

Just so we are clear though, if I have an overcurrent or thermal shut down that should definitely be indicated in the error, correct?

Laminaytrap

[Sineos]

@laminaytrap

Just so we are clear though, if I have an overcurrent or thermal shut down that should definitely be indicated in the error, correct?

Typically, yes. The GSTAT errors either are:

• A wiring issue
• A hardware defect
• Incorrect board jumper settings
• Some strange bus error, often caused by the above points

[laminaytrap]

Ok, I removed the extraneous drivers and set the target temp for the RPi fan to 35C, so it’s running at 100% fan speed for the control box. It was doing fine with motion for an hour or so, then I moved the fan off to the side and it has since stopped. The Pi core temp was ~45C when I pulled the fan, it peaked at 57C prior to failure, and maintained 57C after shutdown. I can’t seem to get the mcu core and the Pi logging temps simultaneously.

Transition to shutdown state: Unable to read tmc uart ‘stepper_z’ register DRV_STATUS

I’ll put the fan back on and let it run all afternoon while I’m out. Hopefully this is a thermal issue and the real error is that we aren’t seeing overtemp in the log for whatever reason.

Trying not to get my hopes up, troubleshooting intermittent failures on unusual hardware configurations is hard, just have to keep at it and keep notes/logic in the test design.

I have another question about hardware config if anyone has an opinion. Should I have the RPi always on and switch the controller on and off? That’s the way my old setup was running, but it seems with klipper I have to reboot the firmware after turning on the printer, which seems strange. Even if I power cycle the RPi and printer simultaneously I have to do a FW reset in order to get ready. Lmk if anyone has thoughts on this.

Laminaytrap

[Sineos]

@laminaytrap

on unusual hardware configurations is hard

I would not call an Octopus board with TMC2209 unusual. On the contrary, this is likely a well proven combination.
Klipper does not care if you have installed your motors, hotend, etc. on a CR10, an Ender 123 or whatever else.

@laminaytrap

I have to reboot the firmware after turning on the printer, which seems strange

This is by design: Klipper relies on a working connection between the host process on the SBC and the firmware on the MCU. If one of them stops responding for whatever reason, the other partner will go into an error state, which must be cleared with a RESTART / FIRMWARE_RESTART.

If it bothers you, you could automize this via an UDEV rule that contains a call similar to:

RUN+="/usr/bin/sudo -u pi /bin/sh -c '/bin/echo FIRMWARE_RESTART > home/pi/printer_data/comms/klippy.serial'"

[laminaytrap]

I appreciate your insight and input. I haven’t even gotten to the fun part of klipper yet lol! Makes sense that is the default behaviour, I’ll have to think about how to handle that, it’s definitely a pretty safe mode to operate in, but it means I must load the web interface prior to sending prints out from the slicer.

With respect to my ‘unusual hardware’ comment I more meant that if emi was such an issue then cable routing, module orientation, PSU locations all come into play. I’ve swapped car engines from different generations, different vehicles and if you don’t have a guide then getting all the electronics talking to each other can be very overwhelming, even on older systems. Clearly there’s something about my setup that is atypical if others are making this work easily.

At this point by moving the fans from off the printer to on the printer at 100% power I’m 4 hours in on my current test. Prints without the fans were lasting 5-10 minutes. Might have a light at the end of the tunnel.

Funny that everything I read seemed to indicate that active cooling was not necessarily required for octopus and 2209s, yet here we are. Fingers crossed again.

Btw I couldn’t find a config for the cr10v3 screen as it is unique from the other cr10 and reprap models, is there somewhere useful that I should post it for others to find? I’m not sure that the beep is working yet, and with the module upside down I haven’t been able to give it a thorough test, but the connectors do need to be inverted and use exp1 and exp2 in case anyone finds this thread, the code is in the klipper.log above.

Laminaytrap

[laminaytrap]

Ok, seems like an unpopular opinion, but the failure is thermal exclusively.

I did a 9h print without filament - removed the fans and the print failed within 10 minutes, slightly faster than my last test at 14 minutes.

I will redesign the control box and PSU mounting to enable better airflow on the mainboard. Hopefully, this exercise saves someone else some headaches in the future. I will update this thread once the printer is fully assembled and tested with the new hardware.

---

Fan shroud print ~11h. Clearly I have some tuning to do on temps/retraction

[Sineos]

Interesting. Thanks for sharing your solution.