The current Klipper macro system has shown impressive results. Impressive, both in a “terrific” way and in a “terrible” way. Terrific in that I’ve seen macros used to implement very impressive functionality that can directly improve user printing experience, and terrible in that some macros have grown to a level of complexity that makes them very hard to troubleshoot.
One of the weaknesses of Klipper macros is that they rely on Jinja2 and G-Code - both of which are very “quirky”. They both also lack good error handling capabilities - thus making it very difficult for macro authors to check for and meaningfully react to abnormal situations. This lack of error handling makes troubleshooting even more difficult.
I’d like to start a discussion on the possibility of enhancing Klipper to support an alternative mechanism for adding new commands to Klipper. I’m tentatively calling this proposal a “Klipper plugin”, though I admit that may not be a good name for it. This is just for discussion at this point - it may not be a good idea, and even if it is a good idea it would still need to be implemented.
To be clear, this would be in addition to the current Klipper macros. I have not plans or intentions of removing support for macros in Klipper.
The core idea of this “plugin system” would be a mechanism for users and developers to be able to add new commands to Klipper. I envision it would have the following capabilities:
-
“Plugins” would not need to reside in the Klipper3d github repository, and would not need to go through the Klipper3d review process. Regular users could define their own plugins or obtain plugins directly from other developers.
-
The new code could be implemented in Python or some other high-level language. New functionality would not need to be implemented in Jinja2.
-
Code could “register a command” with Klipper, and if that command is run the plugin would be notified. It could then generate a series of G-Code that Klipper should run (as macros do today). It could also “pause” the g-code command stream (as tools like TEMPERATURE_WAIT do today). As an extra capability, if any g-code commands that a plugin issues raises an error, then it could “catch” and handle that error (potentially without the error causing Klipper to stop a print).
-
It could export “status information” that other macros and API users could access (eg,
{printer.myplugin.x}
). -
Configuration parameters for the plugin could live in the standard printer.cfg file.
-
As a capability not currently available to macros, I envision a plugin could execute a limited set of “asynchronous actions” that could occur even when the G-Code stream is nominally busy (for example, change the color of an LED).
-
As another capability not currently available to macros, I envision there would be limited support for a plugin to be able to use SAVE_CONFIG to alter its own configuration sections in the printer.cfg file.
A “plugin” would not be a Klipper “extras module” - there are some things that I would envision are “out of scope” and thus explicitly would not be possible from a “plugin”:
-
It can not implement new types of kinematics, nor introduce custom “g-code coordinate transformations”. Plugins would not be intended to alter the Klipper movement “fast path code”.
-
It would not have low-level access to the internal state of Klipper objects, not be able to directly change that internal state, and would not be run within the main Klipper OS process.
-
It would not be able to directly send commands to MCUs. It can not add support for MCU commands that the main Klipper code does not know about.
-
It can not reliably respond to hardware events with low-latency. It would not have direct access to Klipper’s low-latency background thread.
In summary, the goal would be to make it possible to implement new g-code commands using high-level languages and to support meaningful error handling. At the same time, the goal would be to support a meaningful logging and troubleshooting system so that if a plugin has a defect its users can be rapidly directed to the developers that can solve the underlying issue. (That is, we want to try to limit the burden that a buggy plugin could place on the mainline Klipper contributors.)
I have some ideas for a possible implementation - mainly by extending the existing Klipper API server unix domain socket:
-
Klipper could be extended to check a directory that “plugins” live in. If an unknown config section is encountered and the config section matches a file in the plugin directory, Klipper could run the plugin (in a sub-process) and pass Klipper’s unix domain socket to that plugin on its command line. When the plugin starts it could use the unix domain socket to obtain the Klipper config and handle any configuration items in its associated config sections.
-
A new “register_gcode_command” API endpoint could be added to Klipper. A plugin could use this capability to register a command. If Klipper sees this new command in a command stream it could forward a notification via the API server. The plugin could then acknowledge the command, or issue new “run_gcode_from_command” API endpoint requests back to Klipper. Thus the API server could be extended to support the addition of synchronous G-Code commands.
-
Similarly, a new “set_status_state” API endpoint could be added to Klipper so that a plugin can update
{printer.myplugin.xxx}
information. -
The plugin would have all the existing capabilities of the API server. It could thus “subscribe” to status updates, invoke asynchronous “endpoints”, submit G-Code commands, and similar.
-
A Python reference library could be implemented with the low-level details of opening the unix domain socket, handling the registration of commands, and similar. Thus, I envision most plugin authors could just write basic Python code for their new G-Code commands, and not have to worry about all the gory details of the underlying unix domain socket.
Again, this is just a starting point for discussions. I haven’t made any decision one way or another if this is a “good idea” or not.
Even if this is a good idea and even if it is successful, I’m sure the results will still be both “terrific” and “terrible”. Even with plugins I’m sure we’d still have issues with over-complicated plugins, bugs, and troubleshooting. Perhaps, at least though, it will be possible to build complex plugins that are maintainable, which I suspect may not be achievable with today’s macros.
Thoughts?
-Kevin