MCU shutdown with BTT CB1

[Nem0815]

Basic Information:

HyperCube Custom LR Build
BTT Manta M8P / CB1
klippy.log

Describe your issue:

Printing with 16 Microsteps on X / Y / E (interpolation: True) and Z = 32 microsteps (Interpolation: False) works fine without any errors (including webcam stream).

Once I set X / Y to 64, E to 32 (all with Interpolate: False), the printer starts to hang sporadically with the famous MCU shutdown - timers too close issue.
Mostly it prints without issue, but sometimes it stops in the middle of a long print like in the log attached (before that i already printed a 9h print without issues with the same settings)

Could that really be a load issue? Cannot see much in the klippy.log…

What does the host buffer increase mean when it stops?

klippy (1).zip (1.2 MB)

[EddyMI3D]

Hello @Nem0815

@Nem0815

Once I set X / Y to 64, E to 32 (all with Interpolate: False), the printer starts to hang sporadically with the famous MCU shutdown - timers too close issue.

This is 4 times more data for each of the X and Y axes, and double the E axes.

Quite a huge amount of more data to be transferred and to be processed by the MCU.

You may look here:

Timer too close

Background This error typically occurs when the host sends a message to the MCU, scheduling an event at a time that is in the past. Reasons High system load of the host High disk activity of the host Swapping due to low free memory Disk errors / dying SD card Unstable voltage Other hardware hogging the USB bus or other system resources Running in a Virtual Machine USB, UART or CANBUS wiring faults leading to extremely delayed messages ElectroMagnetic Interference (EMI) affecting proper signa…

and also consider whether the increasing of microsteps really matters.

[Nem0815]

So it means basically it is a load issue, as I can rule out all the other issues mentioned in that quote since it works fine with 16 microsteps all the time.

Means a CB1 (which is faster than a Raspi 3B) is too weak for klipper running higher microsteps than 16?

/Edit:

I just reduced Z and E back to 16 (X/Y at 64) and reduced camera framerate to 8fps, but only after some minutes this time the MCU Shutdown happened again…

Again the same it seems:

Right before the shutdown there also seem to be several overtemp warnings - could that be related?

Stats 49596.2: gcodein=0  mcu: mcu_awake=0.019 mcu_task_avg=0.000026 mcu_task_stddev=0.000030 bytes_write=1479837 bytes_read=935272 bytes_retransmit=9 bytes_invalid=0 send_seq=41396 receive_seq=41396 retransmit_seq=2 srtt=0.000 rttvar=0.000 rto=0.025 ready_bytes=0 upcoming_bytes=0 freq=64001144 sd_pos=521547 cb1: temp=44.3 mcu_temp: temp=45.4 heater_bed: target=60 temp=60.0 pwm=0.664 sysload=1.72 cputime=1892.434 memavail=640632 print_time=2379.446 buffer_time=2.074 print_stall=1 extruder: target=230 temp=229.9 pwm=0.434
TMC 'stepper_x' reports DRV_STATUS: 00130101 otpw=1(OvertempWarning!) t120=1 cs_actual=19
Stats 49597.2: gcodein=0  mcu: mcu_awake=0.019 mcu_task_avg=0.000026 mcu_task_stddev=0.000030 bytes_write=1481592 bytes_read=935807 bytes_retransmit=9 bytes_invalid=0 send_seq=41435 receive_seq=41435 retransmit_seq=2 srtt=0.000 rttvar=0.000 rto=0.025 ready_bytes=0 upcoming_bytes=0 freq=64001147 sd_pos=521810 cb1: temp=44.3 mcu_temp: temp=46.1 heater_bed: target=60 temp=60.1 pwm=0.080 sysload=1.72 cputime=1892.527 memavail=640392 print_time=2380.538 buffer_time=2.164 print_stall=1 extruder: target=230 temp=230.1 pwm=0.356
Stats 49598.2: gcodein=0  mcu: mcu_awake=0.019 mcu_task_avg=0.000026 mcu_task_stddev=0.000030 bytes_write=1484240 bytes_read=936441 bytes_retransmit=9 bytes_invalid=0 send_seq=41488 receive_seq=41488 retransmit_seq=2 srtt=0.001 rttvar=0.000 rto=0.025 ready_bytes=0 upcoming_bytes=0 freq=64001150 sd_pos=522643 cb1: temp=44.4 mcu_temp: temp=46.1 heater_bed: target=60 temp=60.1 pwm=0.155 sysload=1.72 cputime=1892.635 memavail=640348 print_time=2381.662 buffer_time=2.287 print_stall=1 extruder: target=230 temp=230.0 pwm=0.375
Stats 49599.2: gcodein=0  mcu: mcu_awake=0.019 mcu_task_avg=0.000026 mcu_task_stddev=0.000030 bytes_write=1487882 bytes_read=937150 bytes_retransmit=9 bytes_invalid=0 send_seq=41559 receive_seq=41559 retransmit_seq=2 srtt=0.001 rttvar=0.000 rto=0.025 ready_bytes=40 upcoming_bytes=0 freq=64001151 sd_pos=523797 cb1: temp=44.1 mcu_temp: temp=45.9 heater_bed: target=60 temp=60.1 pwm=0.245 sysload=1.72 cputime=1892.741 memavail=640416 print_time=2382.574 buffer_time=2.199 print_stall=1 extruder: target=230 temp=230.2 pwm=0.347
Stats 49600.2: gcodein=0  mcu: mcu_awake=0.031 mcu_task_avg=0.000034 mcu_task_stddev=0.000039 bytes_write=1490953 bytes_read=937811 bytes_retransmit=9 bytes_invalid=0 send_seq=41620 receive_seq=41620 retransmit_seq=2 srtt=0.001 rttvar=0.000 rto=0.025 ready_bytes=0 upcoming_bytes=0 freq=64001150 sd_pos=524199 cb1: temp=44.2 mcu_temp: temp=46.7 heater_bed: target=60 temp=60.1 pwm=0.416 sysload=1.82 cputime=1892.838 memavail=640680 print_time=2383.639 buffer_time=2.263 print_stall=1 extruder: target=230 temp=230.0 pwm=0.445
Stats 49603.0: gcodein=0  mcu: mcu_awake=0.031 mcu_task_avg=0.000034 mcu_task_stddev=0.000039 bytes_write=1492108 bytes_read=938625 bytes_retransmit=673 bytes_invalid=0 send_seq=41659 receive_seq=41647 retransmit_seq=41658 srtt=0.000 rttvar=0.000 rto=0.025 ready_bytes=305 upcoming_bytes=0 freq=64001149 sd_pos=524284 cb1: temp=43.1 mcu_temp: temp=46.4 heater_bed: target=60 temp=60.2 pwm=0.000 sysload=1.82 cputime=1892.911 memavail=640456 print_time=2384.001 buffer_time=0.000 print_stall=1 extruder: target=230 temp=230.1 pwm=0.356
MCU 'mcu' shutdown: Timer too close
clocksync state: mcu_freq=64000000 last_clock=152586138768 clock_est=(49571.219 150550813444 64001149.146) min_half_rtt=0.000087 min_rtt_time=49078.081 time_avg=49571.219(908.131) clock_avg=150550813444.303(58121457740.700) pred_variance=2475756.160
Dumping serial stats: bytes_write=1492108 bytes_read=938635 bytes_retransmit=673 bytes_invalid=0 send_seq=41659 receive_seq=41647 retransmit_seq=41658 srtt=0.000 rttvar=0.000 rto=0.025 ready_bytes=305 upcoming_bytes=0
Dumping send queue 100 messages

[Sineos]

Try with your high microstepping but disconnect the camera.

[Nem0815]

Looks better indeed, no issue so far. Bur my webcam is only set to 8 fps at 640x480 - can that really be too much? At least the load readings in htop don’t show anything unusual when it is plugged in…

[Sineos]

I cannot provide any technical details, but experience has shown that webcams apparently can “mess” with the USB port.

[Nem0815]

Next try:

I’ve upped the microsteps for X,Y,Z,E all to 256, disabled interpolation for all and set camera FPS to 25 (before it was 8).

AND: I disabled the Obico service.

Result: Until now no issue, same low CPU usage in htop, smoother camera stream in fluidd (with Obico running the camera FPS dropped even when only set to 10fps below that or even stuttered completely!)

[Blake100]

Hello @Nem0815,

This exact issue happened to me, with the exact same issues you have, with the EXACT SAME HARDWARE. The solution (for me) is to update your m8p’s mainboard chip (MCU). It is separate from your cb1’s processor, and BTT ships it with WAY out of date fw. This is extremely annoying to me, but there is nothing we can do about it. This tutorial walks you through every step, and it solved my issues immediately: Tutorial: BTT Manta M8P + CB1 Klipper guide – 3DP and Me Scroll down to where it says “Flashing the MCU Firmware”, and continue from there. I run insanely high microsteps, AND I connect klipper to the interwebs so I can look at it from anywhere, and I don’t have issues. Try and do this, and reenable everything that you had and loved before. Hopefully it works!

ps: Before you do this, check for updates in the “Machine” section. Update them if needed, and then proceed with the tutorial. EDIT: I don’t have a webcam, but am planning to install one and will let you know if I have any issues.

Hope This Helps,
Blake

[Nem0815]

Think I did that already initially when I got the board, but I will give it a try again, Thanks!

Edit:

After FW reflash the first 2h print with all @ 256 microsteps was fine incl. Obico Service running + Camera set at 25fps and htop running in parallel.

Somehow 256 microsteps for all seem to run alot smoother then having different settings for X / Y / Z / E as I did before… also print quality seems alot better (currently printing big single wall parts for an RC plane which show every tiniest bit of layer shifting or other inconsistencies).

This is how my load looks with these settings:

[Nem0815]

Think I nailed it now down to Obico service.

Ran another print now, everything was running fine and all I did was opening the Obico app on my Smartphone - exactly at the same time the MCU shutdown happened on my printer.

Edit:
While printing @ 256 microsteps and camera streaming in fluidd, I now uninstalled Obico through kiauh, installed OctoEverywhere, configured it and am streaming now also through OctoEverywhere - no issues so far.

[Blake100]

Hello,

Ok! Seems like Obico is a super intensive program, I’ll run octo everywhere as well.

Thanks,
Blake

[EddyMI3D]

@Blake100

Seems like Obico is a super intensive program, I’ll run octo everywhere as well.

You run them side by side?

[Sineos]

Ping @obico-neil
Cannot comment, I’m using neither one of these tools, but it seems strange.

[Blake100]

Hello,

Oh heck no. I meant instead of using Obico, I will use “Octo Everywhere”. I just set it up, and it is working fine. Octo Everywhere is so painless and easy to setup, compared to Tailscale. . . I just need to uninstall Tailscale (what I was using previously) and I should be good.

Thanks,
Blake

[kenneth]

I’m the developer behind Obico. I’m wondering if its a bug in our code that costs a lot of CPU load. I’ll be really concerned if it’s the case.

Can one of you run Obico service again (while you are not printing), and take a screenshot of the “htop” command to see how much CPU moonraker-obico is taking? Thanks a lot!

[Nem0815]

Just reinstalled to give it a try. When the printer is idle and does nothing it is ~1% CPU load.

However, when I open the Obico webpage and login or open the mobile app there is a short spike of up to 20% CPU load!

Same happens when refresh is pressed in the app.

When it is printing the CPU load of Obico toggles between 1 and 10% (seems pretty high to me), even when I don’t have the Obico page or app opened.

[kenneth]

Thank you for the details. 1% is ok but 10% or higher is definitely high.

Can you set the logging level to debug and send the moonraker-obico.log file to support@obico.io? Thank you so much!

[Sineos]

@kenneth

Thank you for the details. 1% is ok but 10% or higher is definitely high.

The H616 CPU on the CB1 is more powerful than a RPi 3B one. 10% - 20% load from the service should leave more than enough room for Klipper.
I have a hard time to believe that this is the root cause for a Klipper shutdown, TBH.

[Nem0815]

As i said - only opening the app on my mobile was sufficient to get a mcu shutdown.

This was a single event so far, but very obvious.

Afterwards i tried this again but besides the spikes in CPU load no occurrence anymore.

So it seems more like a sporadic issue not immediately leading to a shutdown or related to CPU load (but bad enough If it happens after 8h printing… ). More like a blocking service due to bad Internet connection or similar.

[kenneth]

True. 10%-20% shouldn’t usually cause too much of a problem. It’s just the python part of moonraker-obico shouldn’t usually take more than 1% of the CPU. On a Raspberry Pi, ffmpeg will kick in to do the h.264 encoding and it may take more than 20% or even 30%, but ffmpeg shouldn’t even start on BTT CB1. What I’m a little worried about is there is a bug that somehow makes moonraker-obico try to start ffmpeg on a BTT CB1.

@Nem0815 can you still send moonraker-obico.log file to support@obico.io so that I can take a look to see if we are trying to start ffmpeg? None of the Obico team members have a BTT CB1 so we can’t reproduce it here.