SOLVED: Pi crashed 3 times mid-print, almost same place

Basic Information:

Printer Model: TronXY X5SA-400
MCU / Printerboard: SKR3EZ, TMC2209, BTT Pi v1.2
klippy.log
klippy (7).zip (1.1 MB)
Dmesg.txt (63.4 KB)

Describe your issue:

My BTT pi v1.2 has crashed 3 times recently mid print. I have several other succesful prints on this machine, no problem. I also run the BTT pi v1.2 on 5 other printers here, and they all run great.
Latest issue was printing an action figure and it crashed in almost the same place 2 times. Once it happens, I can’t connect to the pi at all, screen is blank with no power, etc. and I have to power cycle the power supply to get it to turn on (Pi is powered by printer 24v power supply). I don’t think it’s a slicer issue (Orca Slicer) or else it would have happened more consistently on other prints.

The Klippy logs looks like it was just humming long, and then BOOM… “Starting klippy…” but I’m not great at reading these logs yet.
I also added my dmesg since it might be a pi related. Several errors in there also, but not sure if they are related.

Hoping someone can help me out here and see what I’m missing. Hopefully the culprit is in these log files and there’s some guidance. Thanks!

Any advice?

This seems strange.
The klippy.log does not contain any errors. It somehow looks like as if your RPi just rebooted.
I’d start with a new SD card and do a fresh setup of the Pi. In your dmesg there are some strange USB errors. If you run other USB equipment on the RPi try removing it.

Have you any kind of cooling for the BBT pi?

Thanks for reviewing it! Yeah I saw the USB errors too but I didn’t know what to make of them. The only USB items I have attached are the mcu, a webcam, and the HDMI touch screen.

Also just to be clear, the pi does not reboot on its own. I have to power cycle everything and then it will start up normally with no issues. This is also my second SD card from a name brand. I use them in all of my pi’s with no issues like I used to have with the crappy SD cards.

I don’t know what could be causing this, but when I try to run that particular sliced file, it crashes about the same place each time. I sliced the file a little differently with different supports and a little bigger but still the same basic print time and it printed fine. I also printed several other objects with no issues as well.

I don’t know what Orca slicer could be doing and I don’t see anything in the g code especially since it’s not happening at the exact same place every time but within two to three layers of the same place.

I will try to unplug the camera and touch screen and see if that suspect file will print.

I do not have cooling on the pi directly. I thought about printing a case and adding its own 5 volt temperature fan but this would be my only pi that would have cooling because my other four don’t have it and have never had a problem.

What do you mean with crashes? What are the symptoms?
The log shows no error on Klipper side.

Crashing in terms of the the screen goes blank (no signal), pi is unresponsive through the browser or SSH, and the case and hot end fan go full 100%. Green status LED is still lit on the pi, but unresponsive otherwise.

I’ve never heard of Klipper killing the entire Pi before.
I think it’s more likely that the RPi has a hardware defect. At best a dead SD card, at worst the entire RPi.

Or the PSU of the Pi is dying.

1 Like

Did you test the Pi without anything connected?
Just SSH into it and regularly check for response.

I had maybe similar issues in the past with my RPi3.
Maybe check reliability of the LAN/WLAN connection and measure the voltage being supplied to the Pi.
It often helps to drive 5.1 V instead of 5…

Did you update the installed OS to the latest patches?

I don’t know if it’s Klipper killing the pi, but something just triggering a shutdown.
I tried another SD card, and so far no issues. But the strange part is that it fires right back up after the crash will print a different file with no issues, just not the 2 files that caused problems.

The PSU is the PSU of the printer. BTT pi’s have a 12/24v input to power the pi, which is what I use. PSU should be well over the wattage required to run my set up.

So the pi is fine normally, and I’m printing on that printer right now. Just during 2 sliced files, it crashed and is unresponsive mid-print. When it crashed, I could not SSH into it, as it was no longer connected to my network.

Voltage to the pi is 24v from the printer PSU. BTT pi’s have a 12/24v input and that’s what I use (less cords and wall plugs).

Yes, everything is up to date in Kiauh and in Mainsail.

Just to be clear: Klipper or the MCU crashing is one thing. Linux rebooting unexpectedly is a totally different story and I know only 3 reasons:

  • Severe undervoltage / unstable power supply / brown or black-outs
  • Hardware defect
  • Kernel panic, e.g. due to some misbehaving driver or due to defect hardware

That the host goes into such behavior because of Klipper or any gcode seems very unlikely. A user-space application needs to do really extreme things to cause a Kernel panic.

You should analyze

  • /var/log/syslog
  • /var/log/kern.log

around the timestamps of the reboots and check if there are any telling entries.

Forgive my incompetence, but I don’t think I have those files, or at least can’t get to them. I tried to WinSCP into the pi, but nothing matches those file names in the /var/log folder.

Also, the SSH pulls nothing as well:

biqu@Paladin:~$ /var/log/syslog
-bash: /var/log/syslog: No such file or directory
biqu@Paladin:~$ /var/log/kern.log
-bash: /var/log/kern.log: No such file or directory
biqu@Paladin:~$

Perhaps it is a different place or name for the BTT pi?
Here’s what I got in the /var/log folder:
image

sigh Debian 12, right?
Debian 12 “improved” (personally, I think it is horrible) the traditional logging mechanisms and is only using systemd’s journald.

Relevant commands are:

# Limit the time scope:
journalctl --since=2015-11-15 --until="2015-11-16 20:59:59"

# Only errors:
journalctl -b -p err

# Only kernel messages
journalctl -t kernel

# Messages from last boot
journalctl -b-1

# Show how many boot have been logged:
journalctl --list-boots

This is only a small selection of useful journalctl commands, but should give you a start.
But to make things even worse, many distributions chose to do only volatile logging, so nothing is saved and logs will not survive a reboot. This could look like:

$ journalctl -b-1
Specifying boot ID or boot offset has no effect, no persistent journal was found.

If this is the case try:

  • Edit /etc/systemd/journald.conf and make sure it looks like:
    [Journal]
    Storage=persistent
    
  • Run sudo killall -USR1 systemd-journald
  • Reboot and try again and above commands should work
  • All edits and commands should be executed via sudo or as root

IMO systemd is so big and so complex that it still has various quirks and bugs. I’m not a particular fan of it, but YMMV.

Now, of course you need to provoke another crash before you get any logs.

Awesome feedback, you are much better at this stuff than I am, and thanks for the commands to try. I am traveling for work but I will be home at the end of the week to try these.

I have not had a crash since, but I do have the original file that caused a crash so I can run it again and give the logs.

Thanks again!

Ok, I was able to make the journal storage persistant (storage=auto before) and now I can run those journal commands. However, I don’t really know what I’m looking at. Most of the errors are general network errors that seem to happen every few seconds.

Also, what’s the best way to export what I find for you to review? The journal commands you gave produces several hundred lines in PuTTY.

And won’t you know it, I ran the old file that crashed twice, and No failure! Once I get another failure, I’ll post back up.

You can redirect the output into a file and upload here, e.g.

journalctl -t kernel > kernel_log.txt

But do not have your hopes high: If it is a kernel panic then it is definitively beyond my pay grade and usually

  1. Something is badly amiss in the Linux OS (could be solved by setting up the OS from scratch)
  2. There is a hardware defect causing the issue

The best way to move files to/from the Linux Pi is to share the directory/folder over the network. Yes there are other ways but they involve doing someing every time you want to move a file. The shared directoy is passive, the files are allways on both computers.

This logging thing is a good reason to install a more traditional distribution like “Ubuntu”. I have Klipper on Ubuntu 22.04.3 LTS Server.

I do suspect a kernal panic. But it might be a hardware problem that causes it. One thing to look at is temperature. Not just the CPU chip but other chips might fail as they pass above 50C.

Also do look at power. Yes you have a 24V power supply but this means the on-board voltage regulor is very stressed and I’d not be surprized if it goes into thermal shutdown. Try powering it with lower voltage. The heat generated is propportional to the voltage drop across the regulater. Ging from 24V to 3.3V is worse then going from 5V to 3.3V.

If it is a thermal shutdown the log file will not show it. You would need to monitor a 5V or 3.3V pin.

The simplest solution might be to simply replace the BTT Pi clone with a “real” pi and see if the problem goes away.

Thanks for the heads up. I’ll look into a shared directory, but I don’t talk to my pi’s very much unless there’s a problem.

And speaking of which, this is the ONLY pi I have issues with. I have 4 other printers with the same BTT pi without issues, all running off of the printer power supply. One of them is a clone of the printer with issues, but it doesn’t have issues.

I’ve enabled the logging like Sineos suggested, so I’m just waiting for another crash, which I will post up about. If we suspect thermal shutdown, then I’ll rig up a fan and try it again. If still, then I will reflash a fresh image. If still, then I’ll look into a more traditional distribution like Ubuntu.

I like to confirm the problem rather than get it fixed and move on, and you guys gave me a lot to think about!