MCU shutdown: Missed scheduling of next digital out event

[Automobilie]

Basic Information:

Printer Model: SecKit Tank CoreXY on Fluidd
MCU / Printerboard: BTT Manta M8P V2.0, BTT EBB36 v1.2
Host / SBC: BTT CB2 with v1.0.5 BTT’s Debian 11 image here
klippy.log A: With MCU: EBBcan error
klippy(1).log (428.9 KB)
klippy.log B: With MCU: MCU error
klippy.log (4.8 MB)

Hello,

Been working on getting a SecKit Tank going with CANBus on the above boards, but keep getting randomly-timed shutdowns with the error “Missed scheduling of next timed event”. It alternates between the error on the EBB36 (EBBcan) and the MCU (Manta M8P) and will fail in anywhere from a few minutes to a couple hours, though sometimes it will complete prints without errors (Was able to run a few calibration cubes and smaller parts).

*Another user with the error here

Error in klippy.log:

1. CANBus wires to the EBB36 are 18g power directly off the 24v PSU and a twisted pair of 20g for data to the CANBus port on the Manta M8P
2. Stepper-motor wires are all shielded and grounded to the printer frame
3. Both ends of the cable have the jumpers installed for the 120ohm terminations and probing both sides will show 60ohms (parallel 120ohm) and 120ohms if removed.

Printer has a heated enclosure, heatsinks on important chips (Temps below)

I can see that bytes_invalid doesn’t ever increase, but bytes_retransmit will jump just before the shutdowns.

Within SSH on the CB2 I’ve tried:

1. Reflashing BTT’s OS image a couple times as well as firmware for the MCU and EBB36
2. Different CANBus speeds from 250k to 1M (flashed across all three boards and updated in the interfaces config)
3. txqueuelen’s of 128 and 1024

Interfaces config settings:

I also ran the two klippy.logs through the Advanced Trouble-Shooting graph tool:

EBBCan Error:

MCU Error:

I’ve read through I don’t know how many forum posts about CANBus and at this point I’m stuck and not sure what to check/try next?

[mykepredko]

Hiya,

The big thing to check is your CAN bus wiring - probably not what you want to hear but an intermittent connection in the cable can cause your problems.

Do you have the cable (not relying on just the connector) anchored at both ends with the most important end obviously being at the toolhead.

I make sure that the cable is securely fashioned. Here is a view showing how I cable tie the CAN cable to the toolhead so there is no movement at the connector.

Can you supply images of your cable at the toolhead?

---

When I looked at your klippy.log files and read over your description, two things jumped out at me.

The first one is when you said " 2. Stepper-motor wires are all shielded and grounded to the printer frame". What exactly does that mean? I’m asking because you’re using TMC5160s - are they standard 18 pin modules or are they physically large drivers?

BIGTREETECH TMC5160T/TMC5160T Plus Stepper Motor Driver SPI Mode

Secondly, when I looked at your klippy.log files, I see two spaces in the START_PRINT macro displayed in the printer configuration section:

First is close to the start:

[gcode_macro START_PRINT]
gcode = 
	{% set BED_TEMP = params.BED_TEMP|default(110)|float %}
	{% set EXTRUDER_TEMP = params.EXTRUDER_TEMP|default(270)|float %}
	{% set CHAMBER_TEMP = params.CHAMBER_TEMP|default(60)|float %}
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	BED_MESH_PROFILE LOAD=default
	M118 Warming Extruder...
	M104 S{EXTRUDER_TEMP}
	M118 Warming Bed...
	M140 S{BED_TEMP}
	M118 Warming Chamber...
	M141 S{CHAMBER_TEMP}

The second is at the end of the macro:

	M118 Homing Z axis...
	G28 Z
	M118 Adjusting Z steppers...
	Z_TILT_ADJUST
	
	
	
	M118 Priming nozzle
	G92 E0
	G1 X10 Y20 Z0.3 F1500.0
	M118 Drawing 1st line
	G1 X10 Y60.0 Z0.3 F1500.0 E6
	G1 X10.3 Y60 Z0.3 F1500.0
	M118 Drawing second line
	G1 X10.3 Y20 Z0.3 F1500.0 E6
	G1 X10.6 Y20 Z0.3 F1500.0
	M118 Drawing third line
	G1 X10.6 Y60.0 Z0.3 F1500.0 E6
	G1 X10.9 Y60 Z0.3 F1500.0
	M118 Drawing Fourth line
	G1 X10.9 Y20.0 Z0.3 F1500.0 E6
	G1 X10 Y20 Z0.3 F1500.0
	G1 X10 Y22 Z0.3 F1500.0
	G1 X10.9 Y22 Z0.3 F1500.0
	G92 E0
	M118 Centering
	G1 Z5.0 F1500
	G1 X200 Y200 F1500
	
	M118 Beginning print!

[gcode_macro END_PRINT]

These spaces indicate that there are problematic characters in the file that contains the macro code. I recommend you go through and figure out what the problem is as any statements that are not shown in the blank spaces won’t execute.

The issue regarding the blank space in your configuration files are almost certainly not related to the CAN issues.

As I indicated above, the most likely cause of your CAN issues is connection related.

[Automobilie]

Got the cable pretty well secured I think, jiggling, wiggling, tapping, and even smacking haven’t been able to trigger the problem. Is there a utility or a logging script I can run to see if it’s the cable disconnecting?

The thing that makes me think the cable may be ok is that the CB2 is directly socketed to the Manta even though most of the errors I’ve gotten are MCU-related (MCU ‘MCU’ shutdown, with only a few MCU ‘EBBCan’ Shutdown)?

The 5160T Pro steppers are profiled like common ones. The wires are 4-conducter + shield, with all the shields tied together from the 5 (Extruder is on the EBB36), steppers to a wire connected to the frame/AC-Ground.

The spaces in the macros looked to be commented out old code, could that cause a problem or just a bit messy?

I had considered picking up a length of dedicated CANBus cable, but have been hesitant to spend the $30-$50 on cable/ends/etc. if it’s something wrong with the boards

[mykepredko]

Nice looking printer.

Could you get a better image of the middle one (showing the CAN bus going into the printer’s frame)? Your toolhead connection looks fine but I’d like to get a better view of the other end, especially since it looks like it moves.

I should have noted in my previous reply that you have to be just as concerned about the power wires going to the toolhead as the data lines - your drawing current continuously because of the nozzle heater and there’s very little/no reserve in the input capacitor(s) if power is lost even for a fraction of a second.

@Automobilie

The 5160T Pro steppers are profiled like common ones. The wires are 4-conducter + shield, with all the shields tied together from the 5 (Extruder is on the EBB36), steppers to a wire connected to the frame/AC-Ground.

Are you saying that you’ve tied your digital ground (“V-” on the power supply) to the Frame/AC Earth?

That could actually be a problem.

As people here will attest, I’m a bit of a zealot when it comes to grounding.

First off, the Frame should be connected to what I think you’re calling “AC ground” which is the third pin on your AC plug (the other two being Live and Neutral) and NOTHING ELSE! The AC Earth (correct term) is a low impedance path to, well, the Earth in case a live circuit or wire comes loose and comes in contact with the Frame. The AC Earth will dissipate the energy keeping you safe.

Things get a bit confusing with the term “Ground” as the Ground on the controller board and any other devices should be connected to the “V-” barrier terminal blocks on the DC power supply. There should only be one ground path to each device (no doubling or “looping”).

Depending on how you’ve wired things, this could be the source of your problems. I suggest that you disconnect any connections to the Frame/AC Ground and make sure V- is floating relative to the Frame/AC Ground and there is only one ground path to each device.

As for the spaces, it could be due to the commented out code - it’s just that each time I’ve seen that in my own klippy.log files, there’s been a problem with an unseen but invalid character. If you have commented out code and it seems to take up the same amount of space as the whitespace and things are working okay then leave it.

Let’s concentrate on your grounding followed by the wire connection going into the chassis.

[Sineos]

See Missed scheduling of next digital out event

[Automobilie]

Hey Myke!

Got my meter out and verified V- is floating and isolated from AC-Ground and the frame. Also verified the stepper-wire shields are connected only to the frame/AC-Ground. I can try running it with those disconnected and floating?

Hello Sineos,

Ran my Klippy.logs through the web utility for the three heaters. Can’t see anything noticeable like temperature drops or PWM failings, but I also don’t really know how to interpret everything:

There doesn’t seem to be any obvious signs of system overload, but again I’m not entirely sure how to interpret :

Undervoltage does pop up at what looks like the start, but only very early and the crashes happen well into prints. Any way to get more information out of the log file?

htop doesn’t seem to show any excessive resource usage:

Kinda half-tempted to replace the CB2 with a CM4 and try out a 32-bit image as it seems BTT’s CB2 image is 64-bit and had read a couple posts of folks having trouble with that.

And I really appreciate your help folks!

[Sineos]

@Automobilie

Undervoltage does pop up at what looks like the start

The messages indicate that the drivers recover from the UV condition but this is at least very fishy and should not happen at all.

@Automobilie

There doesn’t seem to be any obvious signs of system overload

With the graphs you only see very obvious topics. It is just a cursory indication and also depends very much on the resolution and time horizon that is being looked at.

Generally speaking, often such events are spurious and intermittent and very hard to catch with tools like htop or similar. This cannot be considered a reliable statement.

The bottom line is that something is interrupting / blocking the communication between host and board and / or the host is not able to process information from the board in a timely manner.

[mykepredko]

@Automobilie

Hey Myke!

Got my meter out and verified V- is floating and isolated from AC-Ground and the frame. Also verified the stepper-wire shields are connected only to the frame/AC-Ground. I can try running it with those disconnected and floating?

That’s good - I’d say leave it as is.

While I understand why you have the connectors just inside the cabinet, I don’t love the ability of the wires at the connectors to move with the toolhead. Is there any way in which you can rigidly (but not permanently) attach the cable, just past the connectors so the connectors don’t get any flexing during operation?

@Automobilie

Undervoltage does pop up at what looks like the start, but only very early and the crashes happen well into prints. Any way to get more information out of the log file?

Like @Sineos I think seeing an undervoltage condition is worrying.

What is your power supply and how did you calculate your power budget for the printer?

I’m wondering about your “three heaters”, could you explain them in more detail and how they are powered? I’m guessing their bed, extruder and chamber heaters. Could you also explain what kind of fans you’re using?

@Automobilie

Kinda half-tempted to replace the CB2 with a CM4 and try out a 32-bit image as it seems BTT’s CB2 image is 64-bit and had read a couple posts of folks having trouble with that.

I’d hold off on that for now. Let’s only change one thing at a time and observe it’s effect on printer operation.

Right now, I’m interested in understanding your power usage followed by your heaters and fans, followed by seeing what happens if you can stabilize the connectors leading into the chamber.

[Automobilie]

Hey Sineos!

Hmmm, well I guess I can try setting my oscilloscope up to the power rails and set it to trigger if it drops too much. Any idea what the threshhold is to trigger that warning? Everything should be running 24v, steppers included.

Hey Myke!

So the main power supply is a Meanwell LRS-200-24 (200w 24v Supply) that powers the hotend extruder, MSU/CB2, and the axis steppers. It was the PSU that came with the kit. The bed (600w) and Chamber (500w) heaters are both AC powered, each with a solid-state AC relay, but capped at 90% power limit.

That does seem a bit underpowered though; each z-stepper is set for 1A run current, both X and Y at 2A’s of run current, Extruder 1A, the hotend would be around 1.5-2A of power, CB2 wants 2A at 5v (10W), and then everything else would need current as well.

That’d total out to 9-10A on just the steppers and CB2 alone, or 216W-246W…

Oh my…

Soooo, looks like I need to try a different PSU, or at least set my stepper currents down and see if that makes difference. My old Ender 3 has a 350W, 24v Supply, might try swapping that in a and see what happens.

*Dropped the total stepper currents from 8A (192W) to 5.8A (140W), we’ll see if that helps the crashing

[mykepredko]

@Automobilie

Soooo, looks like I need to try a different PSU, or at least set my stepper currents down and see if that makes difference. My old Ender 3 has a 350W, 24v Supply, might try swapping that in a and see what happens.

*Dropped the total stepper currents from 8A (192W) to 5.8A (140W), we’ll see if that helps the crashing

That sounds like a good plan, having X/Y at 2A is somewhat high. You’re still at the wall for the power supply but dropping it by a couple of amps should help.

[Automobilie]

Was able to complete a part that it’s been crashing on last night, so fingers crossed that a new PSU works out!

Looking at a 48v secondary supply as the Manta lets me run the steppers on a different supply and should be able to fit it in place or even get two high -efficiency PSU’s that can fit in place of the old one!

*Nope, crashed on the second print, same error, but the undervolt warning is gone at least. I ordered in a new PSU and I guess I’ll look at a new CANBus cable, but I kinda thought the point of CANBus was that is was robust and resistant to EMI interference?

klippy(2)trim.log (823.0 KB)

**Something else that may be relevant, if I hit ‘Firmware Restart’ in Fluidd after a crash it throws the error “mcu ‘mcu’: Unable to connect” and have to do a power-cycle to get the printer going again.

klippy(3)trim.log (89.2 KB)

***Did a power cycle and once the printer was restarted I unplugged the CANBus connector from the Manta. After a few second it threw a 'Lost connection with ‘mcu’" error, I reconnected the CANBus plug and hit ‘Firmware Restart’, but this time it was able to restart the machine correctly.

Is there any way to tell if the MCU/EBB36 is locked up?

****Ordered in new PSU’s (a 24v and a 48v) as well as new CANBus cable with a shield. Planning on removing that mid-connector and getting the shield grounded to the chassis.

[koconnor]

I took a look through your log and it seems to indicate to me that the host was no longer able to send messages to either micro-controller. Oddly, it was still able to receive messages from both micro-controllers. Very strange.

Best guess is that something went wrong at the canbus layer. If you have a separate usb-to-can adapter you could try using that to see if there is something odd going on with the canbus hardware on your main mcu. Just a guess though.

-Kevin

[Automobilie]

Hello Kevin,

My new PSU’s (24v and 48v 350W Meanwells) came in, so I’ll need to install them and then replace the CANBus cable with a fully shielded one form Digikey.

As for a USB-to-CAN adapter, I don’t have anything like that on hand, but is the above changes don’t fix it I’ll likely try re-installing everything as USB as the EBB36 takes USB-C.

[Automobilie]

Alright, I have two new supplies installed (24v and 48v, 350W each) with the steppers powered from the 48v and the boards/hotend powered on the 24v. Brand is Mean Well and they’re the nicer 90% efficiency models.

*Ran a test print and it’s still crashing, so the CANBus cable is next…

**Also, when it crashed like this I have to power it off and back on again to get it to reconnect (“mcu ‘mcu’: Unable to connect”), but I can use Firmware and Klipper Restart if it hasn’t crashed out.

[mykepredko]

Could you share your latest klippy.log? There might some some clues in there and it would be a good idea to confirm that you’re still having the same problem.

Could you send pictures of your CAN cable? If you’re suspicious of it, maybe we can see something in it (and you can avoid the same problems later).

[Automobilie]

Hey Myke!

klippy.log (4.7 MB)
That should be the latest, looks like the same issues and I still see ‘undervolt’ errors, but I’m also not sure how often the Klippy.log resets itself.

This is the cable I am planning on using and will have it wired directly from the EBB36 to the M8P with no connector in the middle and the power lines then branching off to the 48v supply. The shield would be connected to a wire which would then contact the frame of the printer and subsequently AC-Ground. That’d be the way to do it?

*While I was working on the CANBus cable and removing the current one, I saw the terminal block on the EBB36 was bent upwards and, with not much movement, it came off completely.

So looks like I’ll be ordering in a new terminal block to replace this one, although if this was the issue wouldn’t it be throwing a different error? Then again, if it was intermittently losing contact I’d imagine that’d cause a bit of noise. Sineos’ page for the Scheduling Error mentions a heater on/off confirmation, but it sounds like that’d just be between the two MCU’s

Next steps will be replacing that terminal block, then the CANBus cable, though I do want to check that the terminal block itself isn’t/wasn’t the issue before recabling. Once I pull the old cable I can certainly grab some more photos of it with the cable-cover removed!

[mykepredko]

@Automobilie

This is the cable I am planning on using and will have it wired directly from the EBB36 to the M8P with no connector in the middle and the power lines then branching off to the 48v supply. The shield would be connected to a wire which would then contact the frame of the printer and subsequently AC-Ground. That’d be the way to do it?

A few things here:

1. I use Tensility 30-2482 (22AWG for power and 28 AWG for signal, 100Ω) so it should be fine. Just a bit stiffer than mine.
2. How much current are you expecting the EBB36 to draw? I generally run my extruder steppers at 0.5 or 0.6A and the heater should not draw more than 2.1A (assuming a 50W heater and a 24V supply). Honestly, you shouldn’t be drawing more than 2.2A at 24V. I’m commenting on this because you seem to set the TMC2209s for a lot more power than I would be comfortable with.
3. Why are you not using the 4 Pin CAN connector built on the EBB36? The CAN connector has the same signals and should be more rugged than the terminal block.
4. Are you passing 48V to the EBB26? That’s 'way out of spec. The LN5016 5V Buck supply is rated to a maximum of 40V and the TMC2209 is good to 28V. I’m not sure what you would use for the heater.
5. The shield should be connected to digital ground. Don’t connect it to frame/AC ground. I’m not sure if this is due to your moniker.

[Automobilie]

Hey Myke!

1. Great! Hoping the extra shielding may make the difference!

2. Had the extruder set for 0.7A, but I can try bumping it down to 0.5-0.6A. The other drivers for the XYZ’s are all TMC5160T’s, so should be rated for well over what the steppers are good for! I’d be expecting about the same current. The part cooler is a remote fan with a CPAP line and it’s water-cooled. Only power draw would be a BLTouch and any onboard sensors like an ADXL345 and Temperature probes.

3. The broken block is for the heater cartridge, but the CANBus cable does go to the 4-PIn connector.

4. Oh no, that’s a typo, I meant connecting the EBB to 24v. o.O

5. Glad I mentioned it then, I’ll do that! Should I add a capacitor or anything like that or just straight to DC-Negative? Actually, should I do the same with the stepper-wire shields? I’m using this cable for the XY and Z’s and have their shields tied to the chassis/AC-Ground, not DC-Negative?

I really appreciate the help! Looks like for the time I’ll be waiting for the replacement terminal block to come in, then running a new CANBus cable, and potentially redoing the shields on the stepper wires.