Brainstorming: Long Running Macros and Urgent Commands (Live Adjust Z / CANCEL_PRINT)

Not sure what to title this one…

If you are running a macro you cant run another GCode command at the same time. I think everyone already understands this. One of the consequences of this is that during a macro you cant make changes to z offset with SET_GCODE_OFFSET Z_ADJUST=-0.005 MOVE=1. The command would get queued until after the current macro exits. If your macro is a very long macro, like the Live-Z test pattern print, you cant make any changes for the entire print. Same goes for CANCEL_PRINT while your “PRINT_START” macro is running. It wont execute until the PRINT_START macro exists which could be several minutes later.

All this to say that: if you make a macro to tune the baby stepping value you cant use baby stepping while the macro runs. My dreams of a Live-Adjust-Z demo for my Voron incomplete dashed :weary:

So I’d like to hear ideas on how we might solve this. I have a few I’ll list here:

  1. Hand Roll the GCode - If I was shipping a commercial printer and I had to make this work today, I would generate the GCode with Python and then write it to a file. I could even use macro programming to look up variables and whatnot in the print, but all the Gcode commands would be unrolled. I don’t think this works well for our community. Macros help us customize gcode output for each user’s printer, its very annoying to have to generate static gcode and even more annoying to have it be in another language. This also woudn’t do anything for CANCEL_PRINT, id have to expand PRINT_START contents into every GCode file.

  2. Capture macro output to a file. You have some commands that start and stop a recorder function that writes all gcode to a file rather than executing it. Then you can run the resulting lines of GCode as a print job. This is hairy because macros both read and alter the state of the printer. E.g. If your macro is waiting for a heater to reach temp by querying the printer’s state it would go into an infinite loop because the heater never turned on. Accessing the printer’s state may not even be possible. I just see too many sharp edges and potential footguns with this idea.

  3. Add a high priority queue to klippy’s command processor. Commands submitted to this queue all execute immediately and suspend execution on the main queue. Front ends would have to know which commands need to be submitted to the high priority queue. Or maybe klippy can tell if the command starts with a bang: !CANCEL_PRINT. This could have sharp edges as well. Changing printer state while a macro is executing could lead to unexpected results.

Anyone have any other ideas?

Does your Voron came with a LCD?
Usually the Z-adjust is made there.

Yes, it has those controls on the LCD. But the commands don’t take immediate effect if a macro is running. So if you write Prusa’s Live-Z calibration print as a macro, which is a totally reasonable and easy thing to do, it won’t work. You can hit those buttons on the LCD all you want, all of those commands will be queued and execute after the print macro ends.

It works during a print because it runs the command after the current line of Goode ends. But macros are treated as a single line of Gcode, even if they expand to hundreds or real Goode
commands and span many minutes if wall clock time.

That’s what I posted this to developers. It’s not intuitive to users that this just won’t work. They think it works already, which is its own sharp edge/footgun.

That is a good reason not to put everything into a macro.

Why not just print is as a usual gcode file?
Then you also have the possibility to abort the print.

I stick with @Sineos to keep the printer.cfg a slick as possible.

This is the same reason you can’t abort a print in PRINT_START. We don’t tell people to write all that in their slicer.

For what it is worth, Klipper enforces the “only one command runs at a time” to make the system easier to understand. The Klipper code is not itself fundamentally limited in that way. If one wanted to they could create a “webhooks” interface to apply changes immediately (at least for requests via the api interface).

Separately, I would recommend against using “baby stepping”. In my experience, it is better to spend time designing a series of reproducible steps to self calibrate the printer (ie, write a robust homing macro). In my experience doing that provides dramatically better results than a “human Z probe”. In my opinion, “baby stepping” is a “crutch” that leads to tons of frustration and lost time. YMMV.

Cheers,
-Kevin

1 Like

I would be very interested in learning more about this.

Thanks for the reply Kevin. When I get “don’t do that / its not a problem” type answers to a question I assume that I have done a poor job of explaining the problem… so please allow me to try asking in a different way:

When a print runs from the SD card we can inject commands via the front end and get them to run before the print ends. E.g. I can change the print speed or heater temp. The printer feels “interactive”. This is because the VirtualSDCard module drip feeds GCode commands to the GCode module at intervals. We get commands into the printer between those drips.

When we run a macro, the macro expands into many GCode commands all at once. All of those commands are instantly added to the command buffer in the GCode module and have to execute before the printer becomes “interactive” again. There is no drip feed for macros.

When a macro is run directly from the front end this makes total sense. We don’t want the users next command to preempt the last one, that would be weird! But during a print we expect to be able to issue commands to the printer for tuning/canceling/pausing etc. The fact that there are parts of the print where this doesn’t work is also equally weird.

What I’m trying to get to is: can the printer remain interactive while a macro is executing if that macro is part of an SD Card Print? I.e. we buffer the template expansion results and drip feed them just like it was regular GCode?

Maybe I could pre-process the GCode file by expanding the macros back into the GCode file. Or modify the way GCode runs to “push” macro expansion results back into VirtualSD’s line buffer and let it do the drip feeding. Either way the printer would remain interactive during the entire print.

(and I understand that there are long running tasks like QGL or TEMPERATURE_WAIT that this wont interrupt. That’s a different problem, not trying to solve that one)

Oh, I understand and agree with your analysis. A g-code command in Klipper can not be interrupted by another g-code command.

This was done intentionally, because the alternate would be really confusing for users. As an example, just imagine how confusing it would be if one’s “print start” macro occasionally moved the toolhead to some strange location (because something else decided to run commands at that time). Fundamentally, macros aren’t designed to be interrupted, and it would be really painful to write every macro in such a way that some unknown events could occur between each command.

Just to be clear, Klipper is not constrained to g-code commands. If you use Mainsail or Fluidd (or similar), this should be readily apparent - one can continue to get temperatures and take some actions even while the printer is running commands. Similarly, one can navigate the lcd menus in Klipper even while a command is running (only submitting a new command from the menu is serialized).

It is possible to issue requests to Klipper via the API Server, and those requests are not blocked by g-code commands. This was an intentional design decision when we built the api server, for all the reasons you cite. API server - Klipper documentation

As for changing Klipper so that some g-code can run at the same time as other g-code. Seems a difficult task. G-code is terrible. It’s really hard to use and really hard to understand. In my opinion, it is best to use alternative systems that don’t have the fundamental limitations of g-code.

See also New gcode to interrupt the pending command queue

Cheers,
-Kevin

1 Like

After re-reading your message, perhaps you are suggesting that some macros could be made non-atomic. I suppose it is possible. Seems like it would be difficult to document for typical users. Lots of weird corner cases (eg, what happens if an atomic macro calls a non-atomic macro). Certainly, I’d expect most users would want their macros to be atomic.

Cheers,
-Kevin

Thinking about it further. If non-atomic macros are what you are looking for, then it is possible to obtain something like that using the existing delayed_gcode feature. Each delayed_gcode macro will itself run atomically, but one could activate a series of delayed_gcode (or, a delayed_gcode could reactive itself). Execution of other g-code commands can then be interleaved between delayed_gcode instances.

-Kevin

I guess what I want is macros run from an SD Card print to be non-atomic. That would probably solve the largest number of use cases.

I agree that I don’t want to visit further pain on macro authors. I don’t need 2 commands to run truly simultaneously and I don’t want to have to write my own custom front end either.

Seems like I would have to implement and demo something. Reading the source, one of the major features of the atomic macros is recursion prevention. That would be a challenge to maintain if execution can be interrupted.

I don’t think the “bad stuff might happen” argument would stand up to real use. Bad stuff can already happen because prints are not atomic. If a user runs a command during a print it is most likely both safe and urgent. The front ends go to some lengths to protect users as well. But anyone can G91 in the console and cause chaos at any time.

I’ve actually done this for my thermal equilibrium macro. Its complicated and I cant imagine building a GCode drip feeder that way.

Cheers :beers:

I suppose one could try.

Some possible challenges:

  1. Macros that use Jinja2; in particular if they access {printer.something}. It would be unexpected for the queried state to change by external actions within those macros. Anything with a Jinja2 conditional (eg, if/else) would likely be surprised.
  2. Macros that use G91, G1, G90 sequences.
  3. If you only disable atomicity for virtual_sdcard, it’ll seem strange to octoprint users using the gcode pseudo-tty. If you do it for the gcode psuedo-tty and virtual_sdcard then it risks interleaving the pseudo-tty with virtual_sdcard.

Cheers,
-Kevin

Would it be a bad idea to explicitly state that a macro is not atomic and explicitly mark the atomic regions within it?
Something like that:

[gcode_macro example]
atomic:false
gcode:
...
DISALLOW_INTERRUPT
G91
G1 X1 Y1
G90
ALLOW_INTERRUPT
...

I made an experiment: Comparing Klipper3d:master...garethky:incremental-macros · Klipper3d/klipper · GitHub

Macros are expanded and then drip fed by virtual_sdcard. Its not 100% or anything, just enough to get a feel for how the printer would behave. I can pause the print in the middle of a long running macro that’s just moving the print head around. Or I can just change the print fan speed. It mostly works as expected.

But commands are buffered by the GCode engine. Commands like PAUSE sent from the console still have to get to the top of the buffer. This makes the result feel less interactive than other printers. For my use-case of applying adjustments during a test print this wouldn’t work as those prints tend to have very few long slow moves. I don’t think this is what people really want for the PAUSE/CANCEL use-case either.

On observing this result I’d have to change my mind to wanting to run 2 commands in parallel.

:man_shrugging: :beers: