Strange behavior on XY-movement after update

[limefrog]

Basic Information:

Printer Model: Homebrew coreXY, heavily based on HevOrt
MCU / Printerboard: BBT Octopus 1.1
klippy.log

Hi,

i’m new here, this is my first topic, english is not my mother language, so I’d like to ask for your patience.

My issue started with an update of MCU Firmware, coming from v0.10.x-xx and went to v0.12.0-98. I’m on MainsailOS and performed the updating, compiling and flashing with KIAUH. It took some runs but after some fiddling it carried out and i have the same version on RPi 3 (host and mcu) and octopus board (mcu stm32f446xx).

After that the homing wasn’t possible any more. The bed (z-axis) is lowered for homing, but movement in X or Y doesn’t work out (tried to home X and Y seperately also). Console looks like:

21:23 Homing failed due to printer shutdown
21:23 Klipper state: Shutdown
21:23 G28

Error Message is:

Klipper meldet: SHUTDOWN

MCU 'mcu' shutdown: Timer too close
This often indicates the host computer is overloaded. Check
for other processes consuming excessive CPU time, high swap
usage, disk errors, overheating, unstable voltage, or
similar system problems on the host computer.
Once the underlying issue is corrected, use the
"FIRMWARE_RESTART" command to reset the firmware, reload the
config, and restart the host software.
Printer is shutdown

But the CPU on RPi and Octopus are nearly idling and everything worked fine for months with the old Klipper firmware.

Strangely most functions work: Endstops, heaters, fans are working, so is the z-axis (3-motors) and the extruder, but the Neopixels keep to be completely dark also.

So my concern is: have i missed something? Do i have to update the printer.cfg in any way to get the new Klipper version running? Is it feasable to make a downgrade, if - how?

I’m looking forward to your recommendations, since i have no Idea what happened!

Thank you in advance
Limefrog

240125_klippy.zip (1.3 MB)

[limefrog]

Hi,

in the meantime i tried to reload older configurations - without success. I attached the grown klippy.log below.

i realized a ‘dirty’ Klipper installation

**Host** (armv7l, 32bit)
Version: v0.12.0-98-g5e433fff-dirty
OS: Raspbian GNU/Linux 11 (bullseye)
Distribution: MainsailOS 1.0.0 (bullseye)

but i wasn’t able to clean it up.

Repo has untracked source files: ['klippy/extras/gcode_shell_command.py']

Renaming the file to .bak broke the start of Klipper.

240125a_klippy.zip (1.6 MB)

[limefrog]

According to Graphing Klipper i have created loadgraphs, but i can’t see how this can help.

The system load up to 16:00 is from prints runnig with the old klipper version and in my eyes it doesn’t show critical events.

[limefrog]

[limefrog]

The MCU frequencies chart looks strange to me, Has anybody a clue how this should be interpreted?

[Sineos]

This is indeed strange, as the error basically pops up immediately. Never seen this before.
I know of no reports that updating Klipper might be causing this.

Generally, see Timer too close

• Your graphs would not indicate something amiss, although the frequency graph looks a bit strange indeed
• I’d try to simplify the system as much as possible and see if the error continues
  • Remove additional USB hardware
  • Remove non printing essential stuff like neo pixel
  • Remove displays
  • etc

[limefrog]

Hi Sineos,

i was afraid to hear exactly that answer … to go where boldly no one has gone before!

In fact the printer is really spartanic, no display (it had one but that was removed nearly immediately, since i was to lazy to walk to the printer and it ate my voltage ), no cameras, nothing. The only pieces of luxury are in fact the neo pixels, a sensor to pick up the air temperature in the compartment and the ADXL345 which connects directly to the RPi and shouldn’t affect the communication. Only 6 steppers (four of them working), 2 heaters, 2 fans (I don’t have a part cooling fan yet). Beside the heat-break fan i only have a 120mm fan cooling the Octopus and the RPi. In full action i have never seen more than 28°C on the Octopus and 35°C on the RPi.

i have already seen your Timer too close advices and my design was from scratch made to be electrically stable. I use shielded cables wherever possible, if not the cabling is braided. I’m using SD Card from sandisk.

Last but not least: everything worked before the update.

Things i plan to do:

• Remove the KlipperScreen package from old days, since the display is already gone.
• Remove the neo pixels (do you mean just to pull the cable or remove from config also?)
• If nothing helps until here: Completely wipe the RPi and set up a fresh System.

Any other recommendations or recommendations for the last step? I would go with MainsailOS again or maybe Armbian with KIAUH and installing from scratch.

[limefrog]

@limefrog

Remove the KlipperScreen package

… this does nothing …

[limefrog]

@limefrog

I’m using SD Card from sandisk

… just pulled the card - in deed it’s a Samsung …

[LifeOfBrian]

Could you maybe test a different/brand new one?

[limefrog]

Hi Brian,

nice to meet you again!

…yes i’m just up to make up a SanDisk …

[LifeOfBrian]

The world is small!
Already thought, man the nick name looks so familiar…

[limefrog]

… with the Help of KIAUH I’ve managed to do some rollbacks (20, 50, 100, 500 commits) down to v0.10.0-493, but i wasn’t able to get it running again, and flashing is a pain, since it takes several runs to work out.

I’m a bit helpless now. It feels like something is broken but I’m unable to find anything nor having an explanation what happened!

[limefrog]

… removed neo pixel and brought everything back to v0.12.0-98 - no change…

…made up a SanDisk SD card with armbian and set everything up with KIAUH just to find out that it wont connect at all. Changed back to old MainsailOS SD card and connected immediately. I’ll revert to MainsailOS on the SanDisk card an see what happens.

Positive aspect: I’ve learned that KIAUH is able to install more than one Klipper-instance. So my next aim (after getting everything working again) is to use two printers with one RPi. It should have enough CPU-power.

Question: Can a defective TMC-Driver cause the Error? Since i have a coreXY it is equal if i home X or Y, both motors have to move that for. Z is still moving 10mm downwards by every homing attempt…

[LifeOfBrian]

@limefrog

Question: Can a defective TMC-Driver cause the Error?

As it seems that nobody else has some good ideas, check it.
Can you, without homing, dump the TMC stats of each X and Y stepper driver?
Do they show any errors?

Can you either test spare stepper drivers or maybe swap with the known good Z ones?

[limefrog]

XY and Z are different sizes an steps per revolution - so no, swapping impossible. but i have an additional set of XY Steppers here (want to build two printers if the first is ‘perfect’).

…but changing the TMC-Drivers is quiet easier - less mechanic. Any concerns to try that first?

I remember i’ve dumped the TMC stats already, but that is long, long ago. let me see…

…here we go:

17:26 SG_RESULT: 00000000
17:26 PWM_AUTO: 000e0024 pwm_ofs_auto=36 pwm_grad_auto=14
17:26 PWM_SCALE: 0000001d pwm_scale_sum=29
17:26 PWMCONF: c80d0e24 pwm_ofs=36 pwm_grad=14 pwm_freq=1 pwm_autoscale=1 pwm_autograd=1 pwm_reg=8 pwm_lim=12
17:26 DRV_STATUS: c0190000 cs_actual=25 stealth=1 stst=1
17:26 CHOPCONF: 30010053 toff=3 hstrt=5 tbl=2 mres=0(256usteps) intpol=1 dedge=1
17:26 MSCURACT: 00f70000 cur_b=247
17:26 MSCNT: 00000000
17:26 TSTEP: 000fffff tstep=1048575
17:26 FACTORY_CONF: 0000000f fclktrim=15
17:26 IOIN: 21000041 enn=1 pdn_uart=1 version=0x21
17:26 OTP_READ: 0000000f otp_fclktrim=15
17:26 IFCNT: 00000010 ifcnt=16
17:26 GSTAT: 00000001 reset=1(Reset)
17:26 GCONF: 000001c0 pdn_disable=1 mstep_reg_select=1 multistep_filt=1
17:26 ========== Queried registers ==========
17:26 SGTHRS: 00000000
17:26 TPOWERDOWN: 00000014 tpowerdown=20
17:26 TPWMTHRS: 000fffff tpwmthrs=1048575
17:26 IHOLD_IRUN: 00081919 ihold=25 irun=25 iholddelay=8
17:26 SLAVECONF: 00000200 senddelay=2
17:26 ========== Write-only registers ==========
17:26 DUMP_TMC STEPPER=stepper_z

17:26 SG_RESULT: 00000000
17:26 PWM_AUTO: 000e0024 pwm_ofs_auto=36 pwm_grad_auto=14
17:26 PWM_SCALE: 0000001c pwm_scale_sum=28
17:26 PWMCONF: c80d0e24 pwm_ofs=36 pwm_grad=14 pwm_freq=1 pwm_autoscale=1 pwm_autograd=1 pwm_reg=8 pwm_lim=12
17:26 DRV_STATUS: c0180000 cs_actual=24 stealth=1 stst=1
17:26 CHOPCONF: 30010053 toff=3 hstrt=5 tbl=2 mres=0(256usteps) intpol=1 dedge=1
17:26 MSCURACT: 00f20030 cur_a=48 cur_b=242
17:26 MSCNT: 00000020 mscnt=32
17:26 TSTEP: 000fffff tstep=1048575
17:26 FACTORY_CONF: 0000000e fclktrim=14
17:26 IOIN: 21000041 enn=1 pdn_uart=1 version=0x21
17:26 OTP_READ: 0000000e otp_fclktrim=14
17:26 IFCNT: 00000010 ifcnt=16
17:26 GSTAT: 00000001 reset=1(Reset)
17:26 GCONF: 000001c0 pdn_disable=1 mstep_reg_select=1 multistep_filt=1
17:26 ========== Queried registers ==========
17:26 SGTHRS: 00000000
17:26 TPOWERDOWN: 00000014 tpowerdown=20
17:26 TPWMTHRS: 000fffff tpwmthrs=1048575
17:26 IHOLD_IRUN: 00081818 ihold=24 irun=24 iholddelay=8
17:26 SLAVECONF: 00000200 senddelay=2
17:26 ========== Write-only registers ==========
17:26 DUMP_TMC STEPPER=stepper_y

17:25 SG_RESULT: 00000000
17:25 PWM_AUTO: 000e0024 pwm_ofs_auto=36 pwm_grad_auto=14
17:25 PWM_SCALE: 0000001c pwm_scale_sum=28
17:25 PWMCONF: c80d0e24 pwm_ofs=36 pwm_grad=14 pwm_freq=1 pwm_autoscale=1 pwm_autograd=1 pwm_reg=8 pwm_lim=12
17:25 DRV_STATUS: c0180000 cs_actual=24 stealth=1 stst=1
17:25 CHOPCONF: 30010053 toff=3 hstrt=5 tbl=2 mres=0(256usteps) intpol=1 dedge=1
17:25 MSCURACT: 00f70000 cur_b=247
17:25 MSCNT: 00000000
17:25 TSTEP: 000fffff tstep=1048575
17:25 FACTORY_CONF: 0000000e fclktrim=14
17:25 IOIN: 21000041 enn=1 pdn_uart=1 version=0x21
17:25 OTP_READ: 0000000e otp_fclktrim=14
17:25 IFCNT: 00000010 ifcnt=16
17:25 GSTAT: 00000001 reset=1(Reset)
17:25 GCONF: 000001c0 pdn_disable=1 mstep_reg_select=1 multistep_filt=1
17:25 ========== Queried registers ==========
17:25 SGTHRS: 00000000
17:25 TPOWERDOWN: 00000014 tpowerdown=20
17:25 TPWMTHRS: 000fffff tpwmthrs=1048575
17:25 IHOLD_IRUN: 00081818 ihold=24 irun=24 iholddelay=8
17:25 SLAVECONF: 00000200 senddelay=2
17:25 ========== Write-only registers ==========
17:25 DUMP_TMC STEPPER=stepper_x

I did a dump of one Z-motor for reference. Do you see something suspicious?

[LifeOfBrian]

Changing the stepper drivers or just swapping them won’t alter the steps per resolution and so on as this is defined in the config.
Don’t confuse this with the stepper motors!

Why do you use 256 msteps and then interpolation as well?
Might overload the system.
Check with 32 msteps for X and Y.

Dumps look fine in my opinion.

[limefrog]

Damn, your’e good! The microstepping was set to 256 to reduce noise (had 16 before, if i remember correctly) when I switched off the interpolation for quite some time, because of “lost steps”. But it turned out that my Belts had been worn out very early because of to much tension when i began building the printer. So the belts were jumping and i believed in “lost steps”. When i found out about the belts and changed them, I turned on the interpolation for sake of noise again but forgot about the microsteps!

I’ll change that but it doesn’t made any problem for three months. As you can see in the loadgraphs above my hardware has enough power to cope with that load without reaching a limit (MCU load 25 of 100 / Host load 140 of 400).

[limefrog]

… set to 32 microsteps - but issue persists …

01:11 SG_RESULT: 00000000
01:11 PWM_AUTO: 000e0024 pwm_ofs_auto=36 pwm_grad_auto=14
01:11 PWM_SCALE: 0000001b pwm_scale_sum=27
01:11 PWMCONF: c80d0e24 pwm_ofs=36 pwm_grad=14 pwm_freq=1 pwm_autoscale=1 pwm_autograd=1 pwm_reg=8 pwm_lim=12
01:11 DRV_STATUS: c0170000 cs_actual=23 stealth=1 stst=1
01:11 CHOPCONF: 33010053 toff=3 hstrt=5 tbl=2 mres=3(32usteps) intpol=1 dedge=1
01:11 MSCURACT: 00f70006 cur_a=6 cur_b=247
01:11 MSCNT: 00000004 mscnt=4
01:11 TSTEP: 000fffff tstep=1048575
01:11 FACTORY_CONF: 0000000e fclktrim=14
01:11 IOIN: 21000041 enn=1 pdn_uart=1 version=0x21
01:11 OTP_READ: 0000000e otp_fclktrim=14
01:11 IFCNT: 00000008 ifcnt=8
01:11 GSTAT: 00000001 reset=1(Reset)
01:11 GCONF: 000001c0 pdn_disable=1 mstep_reg_select=1 multistep_filt=1
01:11 ========== Queried registers ==========
01:11 SGTHRS: 00000000
01:11 TPOWERDOWN: 00000014 tpowerdown=20
01:11 TPWMTHRS: 00000000
01:11 IHOLD_IRUN: 00081717 ihold=23 irun=23 iholddelay=8
01:11 SLAVECONF: 00000200 senddelay=2
01:11 ========== Write-only registers ==========
01:11 DUMP_TMC STEPPER=stepper_x

[limefrog]

@LifeOfBrian

Dumps look fine in my opinion.

So there is no indication to change the drivers in your opinion?
(just asking because i forgot where I put the spares )

I think, I don’t want to hear that - but may i have lost my Octopus in any way? I have two unpopulated sockets - should I move the XY drivers there?

But why have the neo pixels died also? What might be going on there?