If you are running a macro you cant run another GCode command at the same time. I think everyone already understands this. One of the consequences of this is that during a macro you cant make changes to z offset with SET_GCODE_OFFSET Z_ADJUST=-0.005 MOVE=1. The command would get queued until after the current macro exits. If your macro is a very long macro, like the Live-Z test pattern print, you cant make any changes for the entire print. Same goes for CANCEL_PRINT while your “PRINT_START” macro is running. It wont execute until the PRINT_START macro exists which could be several minutes later.
All this to say that: if you make a macro to tune the baby stepping value you cant use baby stepping while the macro runs. My dreams of a Live-Adjust-Z demo for my Voron incomplete dashed
So I’d like to hear ideas on how we might solve this. I have a few I’ll list here:
Hand Roll the GCode - If I was shipping a commercial printer and I had to make this work today, I would generate the GCode with Python and then write it to a file. I could even use macro programming to look up variables and whatnot in the print, but all the Gcode commands would be unrolled. I don’t think this works well for our community. Macros help us customize gcode output for each user’s printer, its very annoying to have to generate static gcode and even more annoying to have it be in another language. This also woudn’t do anything for CANCEL_PRINT, id have to expand PRINT_START contents into every GCode file.
Capture macro output to a file. You have some commands that start and stop a recorder function that writes all gcode to a file rather than executing it. Then you can run the resulting lines of GCode as a print job. This is hairy because macros both read and alter the state of the printer. E.g. If your macro is waiting for a heater to reach temp by querying the printer’s state it would go into an infinite loop because the heater never turned on. Accessing the printer’s state may not even be possible. I just see too many sharp edges and potential footguns with this idea.
Add a high priority queue to klippy’s command processor. Commands submitted to this queue all execute immediately and suspend execution on the main queue. Front ends would have to know which commands need to be submitted to the high priority queue. Or maybe klippy can tell if the command starts with a bang: !CANCEL_PRINT. This could have sharp edges as well. Changing printer state while a macro is executing could lead to unexpected results.
Yes, it has those controls on the LCD. But the commands don’t take immediate effect if a macro is running. So if you write Prusa’s Live-Z calibration print as a macro, which is a totally reasonable and easy thing to do, it won’t work. You can hit those buttons on the LCD all you want, all of those commands will be queued and execute after the print macro ends.
It works during a print because it runs the command after the current line of Goode ends. But macros are treated as a single line of Gcode, even if they expand to hundreds or real Goode
commands and span many minutes if wall clock time.
That’s what I posted this to developers. It’s not intuitive to users that this just won’t work. They think it works already, which is its own sharp edge/footgun.
For what it is worth, Klipper enforces the “only one command runs at a time” to make the system easier to understand. The Klipper code is not itself fundamentally limited in that way. If one wanted to they could create a “webhooks” interface to apply changes immediately (at least for requests via the api interface).
Separately, I would recommend against using “baby stepping”. In my experience, it is better to spend time designing a series of reproducible steps to self calibrate the printer (ie, write a robust homing macro). In my experience doing that provides dramatically better results than a “human Z probe”. In my opinion, “baby stepping” is a “crutch” that leads to tons of frustration and lost time. YMMV.
Thanks for the reply Kevin. When I get “don’t do that / its not a problem” type answers to a question I assume that I have done a poor job of explaining the problem… so please allow me to try asking in a different way:
When a print runs from the SD card we can inject commands via the front end and get them to run before the print ends. E.g. I can change the print speed or heater temp. The printer feels “interactive”. This is because the VirtualSDCard module drip feeds GCode commands to the GCode module at intervals. We get commands into the printer between those drips.
When we run a macro, the macro expands into many GCode commands all at once. All of those commands are instantly added to the command buffer in the GCode module and have to execute before the printer becomes “interactive” again. There is no drip feed for macros.
When a macro is run directly from the front end this makes total sense. We don’t want the users next command to preempt the last one, that would be weird! But during a print we expect to be able to issue commands to the printer for tuning/canceling/pausing etc. The fact that there are parts of the print where this doesn’t work is also equally weird.
What I’m trying to get to is: can the printer remain interactive while a macro is executing if that macro is part of an SD Card Print? I.e. we buffer the template expansion results and drip feed them just like it was regular GCode?
Maybe I could pre-process the GCode file by expanding the macros back into the GCode file. Or modify the way GCode runs to “push” macro expansion results back into VirtualSD’s line buffer and let it do the drip feeding. Either way the printer would remain interactive during the entire print.
(and I understand that there are long running tasks like QGL or TEMPERATURE_WAIT that this wont interrupt. That’s a different problem, not trying to solve that one)
Oh, I understand and agree with your analysis. A g-code command in Klipper can not be interrupted by another g-code command.
This was done intentionally, because the alternate would be really confusing for users. As an example, just imagine how confusing it would be if one’s “print start” macro occasionally moved the toolhead to some strange location (because something else decided to run commands at that time). Fundamentally, macros aren’t designed to be interrupted, and it would be really painful to write every macro in such a way that some unknown events could occur between each command.
Just to be clear, Klipper is not constrained to g-code commands. If you use Mainsail or Fluidd (or similar), this should be readily apparent - one can continue to get temperatures and take some actions even while the printer is running commands. Similarly, one can navigate the lcd menus in Klipper even while a command is running (only submitting a new command from the menu is serialized).
It is possible to issue requests to Klipper via the API Server, and those requests are not blocked by g-code commands. This was an intentional design decision when we built the api server, for all the reasons you cite. API server - Klipper documentation
As for changing Klipper so that some g-code can run at the same time as other g-code. Seems a difficult task. G-code is terrible. It’s really hard to use and really hard to understand. In my opinion, it is best to use alternative systems that don’t have the fundamental limitations of g-code.
After re-reading your message, perhaps you are suggesting that some macros could be made non-atomic. I suppose it is possible. Seems like it would be difficult to document for typical users. Lots of weird corner cases (eg, what happens if an atomic macro calls a non-atomic macro). Certainly, I’d expect most users would want their macros to be atomic.
Thinking about it further. If non-atomic macros are what you are looking for, then it is possible to obtain something like that using the existing delayed_gcode feature. Each delayed_gcode macro will itself run atomically, but one could activate a series of delayed_gcode (or, a delayed_gcode could reactive itself). Execution of other g-code commands can then be interleaved between delayed_gcode instances.
I guess what I want is macros run from an SD Card print to be non-atomic. That would probably solve the largest number of use cases.
I agree that I don’t want to visit further pain on macro authors. I don’t need 2 commands to run truly simultaneously and I don’t want to have to write my own custom front end either.
Seems like I would have to implement and demo something. Reading the source, one of the major features of the atomic macros is recursion prevention. That would be a challenge to maintain if execution can be interrupted.
I don’t think the “bad stuff might happen” argument would stand up to real use. Bad stuff can already happen because prints are not atomic. If a user runs a command during a print it is most likely both safe and urgent. The front ends go to some lengths to protect users as well. But anyone can G91 in the console and cause chaos at any time.
I’ve actually done this for my thermal equilibrium macro. Its complicated and I cant imagine building a GCode drip feeder that way.
Macros that use Jinja2; in particular if they access {printer.something}. It would be unexpected for the queried state to change by external actions within those macros. Anything with a Jinja2 conditional (eg, if/else) would likely be surprised.
Macros that use G91, G1, G90 sequences.
If you only disable atomicity for virtual_sdcard, it’ll seem strange to octoprint users using the gcode pseudo-tty. If you do it for the gcode psuedo-tty and virtual_sdcard then it risks interleaving the pseudo-tty with virtual_sdcard.
Macros are expanded and then drip fed by virtual_sdcard. Its not 100% or anything, just enough to get a feel for how the printer would behave. I can pause the print in the middle of a long running macro that’s just moving the print head around. Or I can just change the print fan speed. It mostly works as expected.
But commands are buffered by the GCode engine. Commands like PAUSE sent from the console still have to get to the top of the buffer. This makes the result feel less interactive than other printers. For my use-case of applying adjustments during a test print this wouldn’t work as those prints tend to have very few long slow moves. I don’t think this is what people really want for the PAUSE/CANCEL use-case either.
On observing this result I’d have to change my mind to wanting to run 2 commands in parallel.
I agree, that would be great. Have macros not be atomic, but have each line of GCode within them be atomic. This is exactly the behavior I observed when using startup GCode from my slicer. I moved all my Start and End GCode out of my slicer into macros, but now I might go back. I can’t tell you how many times I start a print and realize I forgot a slicer setting and need to cancel and start over. Now I have to wait 15 min for my bed to fully warm up and my bed leveling to run before the job cancels.
This is only one example, but I agree with @garethky. This change in behavior would be nice and more intuitive. I was very surprised when I realized that Macros are atomic and not each line within them.
I could see this feature being easy to implement if only GCode is allowed in a macro, but it’s not. Junja2 code muddies the waters here. Maybe this feature would be more viable if there were two kinds of macros. GCode exclusive macros that behave like we desire and Junja2 macros that behave as they do now.
In the end, I think the current behavior is really annoying, but also not enough of a problem to warrant the effort to change it.
Two points there… 15 mins??? Are you using a baby heating pad to heat up your bed? I’m assuming your exaggerating, or maybe I’m under estimating how long it takes for my bed to head up?
Second… What I usually do in those scenarios if I’m impatient is just “Emergency Stop”, Klipper restarts fast enough that I’m not at the risk of setting anything on fire or melting something.
I could see non-atomic macros being useful in limited situations, but to Kevin’s point I can also see it causing a lot MORE unexpected behaviors, debugging time, forum posts with questions and borderline indecipherable macros that happen half in real time and half delayed.
I am not sure what kind of printer abnoba12 is sporting. But if you have a “flagship” Creality printer like the K1 Max… It comes with a 1000W heater… and the advertisement claims are true in this case. It will heat up in about a minute… However… It appears that the printer’s bed is made out of cold rolled aluminum plates that harbor tension in them because of the way they are manufactured. So when you heat up said bed, the tension in it will twist the plate before it finds a new equilibrium. A friend attached a dial gauge and watched the plate flap in the heat for about 10 minutes before it settled. After 9 minutes the center still moved by as much as 0.5mm. You’d need live adaptive (proper) LIDAR based probing during the print to follow that bed curve.
So the bed may heat up to temp in 1 minute… but now you have to wait 10-15 minutes for the bed to settle. Some people claim they need to wait for 30 minutes - perhaps the tension varies. I was very much surprised about all this myself. But there you have it.
The K1 Max is a great printer… I hear. In my experience all it needs is a new probe (prtouch has a StdDev of 0.02 and the loadcell quality varies quite a bit)… And now open source engineers like garethky are fixing the prtouch for Creality (btw thank you garethky). Well then a new bed with a new heating pad would be nice too… Shave those 15 minutes of the prints… Oh and a new board with a little more CPU power would even let you use a cartographer 3D probe with the camera on and do proper temperature calibration. Ah, the drag chain is a bit of a drag as well and Creality support recommends you print a raiser for the lid… And then all you have to do is swap out the puny belts. Possibly the idlers too. Keep in mind the x-axis may not be rigid enough for the load cells, and my z-axis leadscrew reminds me of a belly dancer. I may need to swap those out as well. With a BIQU Microprobe installed I can now see how bowed-out my Y-axis rods are. Other than that - it’s great. As long as you strictly adhere to the unwritten filament change procedure or it may encase the extruder gears in molten plastic. Oh… I should mention the door… Well you have to be careful with the door. Uncle Jessy… Yeah… He wasn’t careful with the door. You are probably ok with having the filament spool at the back of the printer, because the K1 Max deserves to be the centerpiece of a room (plus it gives you lots of space for maintenance and mods you will need to do from day 1).
Side note… I saw the first layer that my CR 10 mini made today and it made me cry a little on the inside when I compared it to the one my K1 Max makes. Creality sales assures us though that the K1 Max comes with all the latest tech that guarantees a perfect first layer. All I need to do is calibrate my filament.
That’s an interesting point. I am in the EU and my home has 16A breakers for pretty much everything - not sure if that’s standard. But at 240V that doubles the wattage. I didn’t have any issues with the printer in that regard. The only thing that occasionally trips breakers here is the car charger and the network rack’s rush current on power recovery. So in North America only the Kitchen and the Bathroom get 30A breakers and big-boy cables? But I digress.
I feel there might be two separate issues on these long running macros:
Ability to interrupt them
Ability to interleave commands
I’d find it somehow useful to be able to interrupt certain macros… And here the state may not be so much of an issue. Neither the printer state or expected motion hardware state matter much if a macro is aborted. It’s a mini emeryency stop - just a little more smooth of an experience. Perhaps macros could be marked as interruptible. What TGC suggests makes a lot of sense to me in terms of interruptibility… One could even skip the atomic attribute and wrap interruptible sections in ALLOW_INTERRUPT … DISALLOW_INTERRUPT statements. But if I understand koconnor correctly it is difficult to implement. I found this thread because I tried to hack my START_PRINT to start a delayed_gcode macro to wait for my bed to thermally settle and then proceed with the START_PRINT_PHASE2. Unfortunately that didn’t work since Klipper detected that the call to START_PRINT ended and attempted to start pumping the print g-code. I didn’t find a way to let Klipper know that the printer isn’t actually ready to print yet.
Interleaving commands is an even more complex issue. There is the state to consider as koconnor says.
And then all you have to do is swap out the puny belts. Possibly the idlers too. Keep in mind the x-axis may not be rigid enough for the load cells, and my z-axis leadscrew reminds me of a belly dancer. I may need to swap those out as well. With a BIQU Microprobe installed I can now see how bowed-out my Y-axis rods are. Other than that - it’s great. As long as you strictly adhere to the unwritten filament change procedure or it may encase the extruder gears in molten plastic. Oh… I should mention the door… Well you have to be careful with the door. Uncle Jessy… Yeah… He wasn’t careful with the door. You are probably ok with having the filament spool at the back of the printer, because the K1 Max deserves to be the centerpiece of a room (plus it gives you lots of space for maintenance and mods you will need to do from day 1).
since Klipper detected that the call to