Possible Klipper "plugins" instead of macros?

Developers
21

Thinking something like a github repo with a json file:

{
 extensions: [
  { name: "led_control",
    repo: "https://github.com....",
    author: "You can trust me",   
    description: "Totally benine led controller"
    install_files: ["led_control.py"], //some sanity control
    deps: ["klipper >=1.0"],
    os_deps: ["numpy 1.26.2 >=1.0"],    
  }, 
  ...
 ]
}

In the printer.cfg

[load_extensions]
 load_led_control=latest
 load_toolchanger=1.0
 load_multifan=latest

That throws error on startup if extensions are missing.

And a horrible Gcode:

INSTALL_MISSING_EXTENSIONS

Or a shell command, but gcode is more convienient :fire:

Should be no more than 100 Python LOC and we are in business.
Probably 100 more to integrate nicely with the update manager.

1 Like
22

This is quite a strange mindset and in my view the idea of this thread is outright the opposite of what you are proposing, since installing stuff from somewhere is already possible today.
Messing up users’ experience and creating issues that they cannot troubleshoot on their own seem not like a challenge. I do not even need your 100 LOC for it.

The discussion is about designing a “plugin” system that

  • is “officially” supported
  • offers the flexibility to extend Klipper’s functionality
  • upholds at least some sort of quality control, documentation and support.
23

Klipper is experiencing an important shift in the 3D printing community, particularly regarding Klipper’s transition from a technology-savvy niche product to a mainstream tool. This transformation is evident in its integration into commercial products and the development of dedicated slicers’ support, among other advancements.

However, this growth brings to light a crucial dilemma. The majority of Klipper’s new user base lacks experience in tasks like flashing printer boards or managing Linux systems. This stands in contrast to the more accessible Marlin firmware, which offers prebuilt binaries readily available for download. Consequently, Klipper’s complexity can appear daunting to these users.

This situation creates a dual challenge. On one hand, there is a need to maintain Klipper’s flexibility, its capacity for introducing new features, and its advanced functionalities that appeal to power users. These users are adept at modifying and troubleshooting their setups, and they greatly value these aspects of Klipper. On the other hand, there is a pressing need to enhance the ease of use, ease of installation, and overall user experience for general users. These users often find themselves overwhelmed with standard installations and struggle with even the simplest of challenges.

Addressing this divide is essential for Klipper’s continued growth and user satisfaction. It’s about striking a balance where advanced users can still innovate and customize, while new users are not left behind, feeling unsupported and frustrated.

In a nutshell: I would not want to walk in Kevin’s shoes, since every decision taken, will be the wrong one. Either for the power user side or for the standard users. There is an abundance of threads and posts here clearly highlighting this conflict of objectives.

3 Likes
24

Sorry that was sarcasm, aimed to move away from abstract ideas and into okay, what can we build today that is better than nothing.

What I described is exactly what’s happening today, sans the tiny bit of user convenience.

Edit: yes there is no right decision. IMO the decision to do nothing inevitably leads to a fork. At least in the python frontend level. Not sure what the situation is down in the c++ libs.

25

A couple of comments back on the proceeding paragraph. I would challenge the phrase “more accessible Marlin firmware”. Yes, there are builds available for specific situations but as soon as you want to deviate from them, even in simple ways (ie adding a filament out sensor, changing to a BL Touch), you’re going to have to do your own builds in VSC.

Secondly, “Klipper’s complexity can appear daunting to these users” isn’t necessarily true if Klipper is properly introduced. I find that if just the basics are introduced including showing the wide range of sample printer.cfg files only give the option of a Raspberry Pi as a host, things don’t appear that difficult or complex - when you do this Klipper comes across and logical and well thought out. It’s when people start talking about the ability to write their own Macros or modify the base Klipper code to get the best performance is when Marlin users shy away.

Agreed.

1 Like
26

FWIW, I think there are some lighter-weight changes that can be made to the system that may reduce the need for a plugin system. It seems to me (both in my own macros, and in the suite of various super-popular ones out there) that a big chunk of their complexity is to work around the fact that there are a lot of settings (most, in fact) in the configuration files that can’t be updated via GCODE/macros. (Unlike, say, RepRapFirmware where everything can be.)

GCODE-based configuration is foundational to RRF, and I’m certainly not suggesting its needed or appropriate with Klipper. That said, the ability to make transient changes to the configuration in-memory and persist those to disk via macro would make a lot of the things people are hacking around go away.

Tool changers could be implemented by changing includes, bed and filament management can be vastly simpler by just changing the respective configuration values, mixing hotends could be supported by implementing M165 and just changing the settings on the extruder.

These are all things that are really easy to do on RRF, really hard to do on Marlin, and Klipper is ending up somewhere in-between, where things are possible with sufficiently ugly hacked-up macros, but they shouldn’t be. “Tweak the configuration in Linux and restart the server” really should never be the answer.

1 Like
27

I’m not a developer of klipper (yet) but have read quite some of the klippy code and found it quite surprising and a little difficult to develop own modules. My problem was the missing internal API documentation and definition against which to program. I also find that many macros would be way better implemented in python as opposed to G-Code and Jinja2. So here are my 2 ct. I would welcome a “plugin” mechanism!

1 Like
28

mainly by extending the existing Klipper API server unix domain socket:

This sounds like a horrible way to write plugin code. There would be a lot of extra work involved to even get started:

  • understand how to write to a unix socket
  • some might try to use the built-in sockets library, which would introduce many more headaches, because it is very low-level
  • the api would be prone to typing mistakes which could result in bugs that are hard to find (there would be no traceback like there would be for python code that calls a nonexistent function)
    → To address those issues, someone would have to develop a separate python library that abstracts away the underlying socket api? Is this really desirable? Who would write and maintain that boilerplate code, and why would they?

Minimizing Fragmentation

As identified here, the Klipper ecosystem is increasingly dispersing. It is vital for any upcoming system to actively discourage this trend.

That is not going to happen with a limited plugin system. The proposed system would introduce too much extra work. Many of the klipper forks have features that deeply integrate into klipper and could not be implemented through a limited API.

As can be seen in the many pull requests on GitHub that are either stale or closed, it is very difficult to get anything merged into mainline klipper. A limited API would inevitably result in people requesting many more extensions to the api. My concern is that the same would happen with the limited API. Nothing will change or only very slowly, which would render it useless.

In my opinion, the best approach would be to trust the plugin developers and let them do whatever they want. The extras API is really well written, and I think with good documentation, it could be used for all the motivations mentioned in this thread.

Thinking something like a github repo with a json file:

 extensions: [
  { name: "led_control",
    repo: "https://github.com....",
    author: "You can trust me",   
    description: "Totally benine led controller"
    install_files: ["led_control.py"], //some sanity control
    deps: ["klipper >=1.0"],
    os_deps: ["numpy 1.26.2 >=1.0"],    
  }, 
  ...
 ]
}

Moonraker is already quite well established for updating external repositories (and their API is very good). I do not think it is smart to fragment the ecosystem by introducing another mechanism for updating third party code.

1 Like
29

Warning message may be a little chaotic as I have gone through everything and I’m replying to things as I remember.

I have mixed fellings with your (@koconnor) proposal, while I agree that some macros (klicky comes to mind) add good functionality they are harder to debug, such macros could benefit of a better system.

I’m not very fond of the idea of using a unix socket to communicate with the plugins, while it’s a way to isolate the code feels cumbersome and without a well thought and simple SDK won’t get much traction in my opinion, while this allows any language, it creates fragmentation if some functionality is deemed good to simply merge into Klipper itself (if the plugin author approves of course) if done in other languages then everything needs to be “translated” this can bring new bugs in.

Regarding hosting I don’t think this should be Klipper and it’s maintainers responsibility, for multiple reasons, may create the illusion to users that who has to support the plugin is the Klipper team itself, and cuts out control from such plugin developers, I personally would not like that. Currently a lot of “plugins” (extras modules) exist and the risk of compromised quality and documentation exists as well, I don’t see the need for a central management or listing of plugins, I think it falls outside Klipper scope, Moonraker already has an update manager, and it seems to me good enough for this, let’s not reinvent the wheel, something exists, why fragment?

Whether we like it or not (I do actually) “extras” modules are the only way to add some functionality to klipper, on this territory Happy Hare, Tradrack, Led Effects, Auto Z Calibration are some that come to mind, and who wants to use them will keep using them and linking them into extras directory, this adds complexity on updates, if something breaks it does indeed not make easy to know the root cause. That being said I think we need a way to load such modules preferably each from it’s own directory whithin a parent directory and adhering to a basic file naming convention, this separation would allow for a “safe mode” in Klipper where it does not load any code from such “extras”, technically I don’t know if it is possible to know if something broke from that “extra” code or not, at least initialization should be, they can still mess up with klipper but that already happens as well, a warning can be given to users on startup as well and make it explicit that module X or Y is external to Klipper. As for changes in Klipper breaking one of these “extras” plugins this also happens already, not common but does, the only solution I see for this is clearly define what methods on a class are public and what methods are private (I believe this already happens) so surface becomes reduced, public ones can be marked as deprecated and after 1 or 2 releases removed, this gives plugin developers time to catch up.

To wrap up, I think allowing for a new “extras” directory would solve the issues of more complex macros becoming high level code, maybe provide a base class to extend from and only provide to these classes a more limited object to interact with. Current extras give them full access as it already happens today, at least a more clear boundary can be made by splitting them into a different directory.

30

These staements are contradicting themselves, aren’t they?
This is exactly the core question being discussed: How can a plugin / extra / whatever system look like that:

  • Allows the freedom to develop new features
  • Allows accessing core methods / functions in a safe and controlled way
    • Even this is probably a contradiction in itself. I guess there will be no way to expose core kinematic / timing relevant functions in a “safe” way
    • Finally boils down to the balance between “freedom to develop” and “safe”. Virtually no API of known / successful products allow you mess with the most critical things

IMO, exactly this is the surest way into fragmentation. I do not know a single successful project that implements plugins and just lets it go wild whatever happens.
Projects that have successful plugin systems typically have:

  • A central plugin repository where plugins can at a minimum register themselves
  • A way to search existing plugins
  • A way for users to rate plugins
  • Often an integrated plugin manager that allows installing / updating them
  • A minimum set of requirements such plugins need to fulfil

HappyHare, TradRack, The SMuFF, ERCF (and a bunch more of the MMU type) is the best example of such fragmentation. Only experienced users will know that they exist, what differences there might be, how mature they are etc.
E.g. if I was a new user and would search for such stuff without knowing the exact nomenclatures, then search like this or this would give me a hell of a time to find something useful.

31

Agree. OctoPrint does exactly this.
Also the possibility to update the plugins from there.

And plugins on installation that do not appear there are viewed suspiciously.

32

No, the first one is just related to using a completely separate process and talking with it in some way, the second one is about running the plugin code within the same python process.

It’s the way it happens now with extras directory, still give the current access to klipper objects but clearly define what can and what can’t be called by these extras, my Python knowledge is still limited, can we have methods that are really private and can’t be called from outside classes?

If the goal is to still allow people to extend Klipper (I believe it is) then I believe some way to allow current extras to work should exist.

As for the fragmentation I don’t really see HH, TR, ERCF etc as fragmentation, I see these as plugins you can install that’s it.
As for what a new user would search I can’t really say if you’re right or wrong on what a user might search, to me it’s hard to imagine what someone completely new would search, would they search “klipper multi colour”? or “voron multi colour” (replace voron for printer name)? something else?

As for the plugin registry and system, I believe this is fragmenting simply because Moonraker already has such feature, currently limited to updates, but if Klipper adds it’s own then we are fragmenting into two systems, I would propose collaboration with Moonraker to push such effort forward, even if Klippper has it’s own, Moonraker would still be needed for it’s own plugins or even Mainsail/Fluidd themes or macro repos, and I’m not aware of anyone running Klipper without Moonraker alongside, but I may be wrong and be more common than I know.

For having a plugin registry where they can register themselves I’m not against, but the boundaries should stop there, hosting should be left to plugin developer, a standard for release structure etc can and should exist.

Something else I thought of, there could be a “Core” registry with Klipper vetted plugins, and then others could be allowed to be added in the config, as an example for Debian and derivatives there are the PPAs, Gentoo supports overlays, Docker is not limited to a central registry.
Again this could be something in Moonraker, but that also depends on Arksine desire to integrate such feature.

33

I proposed a better way to add custom components into Klipper, similar to how this is done in Home Assistant and ESPHome: [WIP] custom_component: allow to load python components natively by ayufan · Pull Request #6492 · Klipper3d/klipper · GitHub.

This connected with externally maintained list of plugins, that could be consumed by Moonraker/Fluidd/Mainsail could be a good middle ground. It exposes what “custom components” are used explicitly in a config, and primary purpose is to prevent people from messing with extras folder.

It has own downsides, as extras is not a stable API. However, the same can be told for Linux kernel, and all DKMS style distributed modules. Hope, this provides a balanced approach and as well explicit approach.

What could be done with this:

  • Is to annotate “what” is stable/allowed and what is not in a Klipper codebase when using custom components.
  • Publish repository of components, and user voted quality of them,
  • Moonraker/Fluidd/Mainsail could read custom_components from Klipper and present this information, and disclaimers that you are running possibly unsafe Klipper. In Linux kernel is “tainted”.
  • Moonraker/Fluidd/Mainsail offer auto-update function and good versioning shown for all loaded and used custom components.