Klipper development goals for 2024

Developers
1

It’s a start of a new year! I’d like to take a moment to go over some development targets that I have for Klipper in 2024. These are things that I’d like to see merged into the main Klipper repository this year.

In brief:

  • Support for MCU flashing from the graphical frontends.

  • A Klipper “plugin” system for external modules.

  • Rework the documentation and installation scripts to recommend Moonraker and Klipper specific frontends.

  • Support for fan and output_pin “templates”.

  • Support for probing and homing with load-cells.

  • Rework of Klipper’s “homing” and “probing” code.

In addition to the above, there are a few high-level topics that I’m planning to closely watch in 2024:

  • Support for multi-material filament switching systems.

  • Filament flow rate detection hardware and systems for improved flow-rate calibration.

  • Heater modeling improvements.

All of the above are “aspirational”, and they only reflect my goals. Other contributors will have their own goals.

Some additional context:

Support for MCU flashing from the graphical frontends

I’ve discussed this a few times in the past (eg, Experiment with automated mcu flashing ). I think it would be useful if most users could install and upgrade Klipper without having to use “ssh”. That is, I believe it should be possible to extend Klipper and the various web frontends to handle basic MCU management. This would include detecting available micro-controllers (eg, by scanning in /dev/serial/), compile the micro-controller code (eg, by automating make menuconfig), and flash that code.

This isn’t an easy task due to the many different boards that are available. Klipper supports hundreds of boards and there are seemingly thousands of different ways to flash those boards! I believe the benefits are likely to outweigh the costs however.

A Klipper “plugin” system for external modules

Klipper is highly programmable, both via its “Jinja2 macro” system and via changes to its core Python code. That flexibility has enabled a vast array of impressive extensions. I think it would be useful to further encourage developers to enhance their 3d printers and make those enhancements rapidly available to others. An official Klipper “plugin” system could help spur these enhancements.

Ideally there would be two parts to a Klipper “plugin” system - support in Klipper itself for plugins, and a global repository that developers could list their plugins (and users could search to find the plugins they are interested in).

It is important to me that Klipper have a robust error reporting system that can rapidly point users and developers to any code causing problems. As such, I do not envision any plugins running within the main Klipper process - instead I envision plugins would run in a separate process. This should make it much easier to identify who to contact in case of a problem. It also means that a plugin will not be able to do everything that Klipper itself could possibly do. However, I still think there are many powerful enhancements that could be done via a plugin.

Rework the documentation and installation scripts to recommend Moonraker and a Klipper specific frontend

The official Klipper documentation still recommends installation using OctoPrint. However, most installations today use one of the Klipper specific frontends, and the documentation should be updated to reflect this. In particular, if we add new features like MCU flashing to the frontends, there will be even more incentive to recommend those frontends.

In addition to the documentation, I think we need to review the installation scripts within Klipper along with the example config files. We really want users to be able to use an example config file from the Klipper repository “as is”, and we really want the internal regression testing code to test example config files just as a user would use them. I think this means reviewing common Klipper modules like “virtual_sdcard”, “pause_resume”, “save_variables”, and similar. For example, we may want to have these modules auto-load in Klipper so that they are available for all users “out of the box”.

Support for fan and output_pin “templates”

Klipper has a lot of support for setting fans, pins, servos, and similar. However, much of this functionality is controlled via G-Code commands which can be delayed when there are other commands running. Klipper has an “led template” system that allows LEDs to be configured using a “formula” so that they update separately from g-code commands. I think it would be useful to extend this to fans, output_pin, and similar modules. It should allow more robust control of peripherals and make Klipper more flexible.

Support for probing and homing with load-cells

There has been a lot of interesting work with load-cells and there are several readily available 3d printers shipping with them today. It would be good to see this support merged into the main Klipper repository.

Rework of Klipper’s “homing” and “probing” code

At the core of Klipper’s homing and probing code is the G28 and PROBE commands. I think we may have reached the limits on these commands and that it may be time to “take a step back” and rethink how “probing movements” should be implemented in Klipper.

The current system has some known limitations (in no particular order):

  • Sensorless homing requires elaborate macros and it is difficult to detect errors in these macros. Sensorless homing would benefit notably from extra error checking.

  • Probing often requires elaborate macros to “dock” and “undock” the probe. It is difficult to implement error checking in these macros and error checking would have high value.

  • The manual_stepper module supports a limited form of “homing”, but using it can result in excessive delays.

  • It can be useful to tie filament loading to a filament sensor, but Klipper does not support this type of “extruder homing” today.

  • It can be useful to have multiple simultaneous probes on a toolhead, but Klipper supports only one active PROBE module.

  • The endstop_phases system can provide notable benefits to repeatable Z positioning, but it doesn’t currently support many printers (in particular, printers that home Z via probe hardware). The core functionality should be usable on nearly all printers if it is made more flexible.

  • Printers really want “homing moves” to occur at a constant stepper velocity, however klipper currently homes at a constant “toolhead velocity”. This is problematic for some types of printer kinematics (such as scara, polar, and rotary delta).

  • And others…

Much of these limitations are due to Klipper’s internal code abstraction for homing and probing movements. As such, I suspect Klipper could benefit from some internal work in this area.

For what it is worth, I do not envision a future “all encompassing homing configuration” system in Klipper. The current G28 command effectively means “do what ever it takes so that Klipper knows where the axes are relative to the bed/frame”. Unfortunately, this doesn’t scale - understanding the configuration system needed to describe “whatever it takes” is far more complicated than explicitly stating the “steps to take”.

Alas, I don’t know what a future homing/probing system should look like. I do think it is something worthwhile to investigate and enhance in Klipper.

Support for multi-material filament switching systems

There are several popular filament switching systems available today (eg, ERCF, MMU2S, Pallette). Support is not a core goal of mine for 2024, but it is something I hope to “keep an eye on”.

Filament flow rate detection hardware, and systems for improved flow-rate calibration

Extruder “flow rate” calibration is an important, but painful process today. Calibrating it is important to get parts with accurate dimensions and good surface quality.

Currently, one can calibrate “flow rate” via “rotation_distance”, via slicer “extrusion multiplier” configuration, via M221 “extrude factor override” commands, or via SET_EXTRUDER_ROTATION_DISTANCE commands. All these settings do effectively the same thing, but they are commonly described using different wording or with different calibration mechanisms. It is madness!

We’ve also seen some interesting developments on the hardware side - hardware tools and processes for automatically detecting an appropriate flow rate. (This may involve cameras, lidar, photo sensors, filament width sensors, or similar.)

I don’t have any “answers” in this area, but it is something I hope to “keep an eye on”.

Heater modeling improvements

Klipper has a very basic PID system for extruders and beds. This system works surprisingly well. However, we do seem to be hitting some limits, in particular with high-power extruder heaters and with very high extrusion rates.

There has been some interesting work in “feed forward” control systems recently (eg, “Marlin’s MPC” system). Something to “keep an eye on”.

Again, these are “aspirational goals” for 2024. Without doubt, there will be other features merged that are not mentioned here.

Happy New Year!!
-Kevin

13 Likes
2

This is great! I’m someone who helps out with the ERCF project and, in the past, some small stuff for it’s accompanying software.

Something that would be incredibly useful for either the future plugin system or current custom python modules would be a more robust event system. Being able to listen for essentially any Klipper/Moonraker action with accompanying metadata would make plugin development MUCH easier. The current event system is very useful and I think many plugin developers would appreciate expanding that functionality!

1 Like
3

Will love the MCU flashing feature. It was a pain point even for me as a developer.

A plugin system will really help as well.

4

Thats quite the list!

Hopefully I will have the load cell work ready to PR by the end of February.

I’d be interested in X/Y probing for CNC machines and multi-tool 3D printers. Right now there are many places where the probe direction is assumed to only be in Z.

5

I think another aspect the community and the project would benefit from is faster code review for features someone already wrote code for. Too many pull requests end up discarded.

I wanted to start a thread soon about the topic and about filament width diameter compensation, but let me add some information here for now.

Between “rotation_distance” (which is a hardware property) and “extrusion multiplier” (which I see as something the user may want to tweak for whatever reason manually) it would be useful to implement an array of compensation factors, so that an arbitrary number of factors could be handled by various plugins.
Each item of the array would be identified with a string and would carry a value.

For example, the current Hall filament width sensor uses the extrusion multiplier to compensate for variable filament width. This has the actual effect of removing the extrusion multiplier functionality from the user, which won’t be able to touch it, since it jumps around continuously, handled by the width sensor.

With an array of compensation factors, the filament width compensation plugin could add an entry, like “filament_width_factor → (value)”. The user would retain the ability to tweak the extrusion multiplier, which could be the first, always present entry in the array of factors.

Then other plugins could add additional entries, for example let’s imagine a future plugin can compensate for material type (which gets squeezed differently by the gears), or a speed-dependent compensation factor to flatten, up to a certain upper speed, the extrusion curve (ok this example is not good, since it’s not based on external sensors). They could add their own multiplication factors to the array.

So yes some reworking of the flow rate system could help move forward.

6

Maybe you forget that in the real world the filament string is not always round and a Hall effect sensor can’t properly determine the correct diameter or better cross-section.
So for me the manual calibration of the desired flow is needed as long as you aren’t using something like Lidar measurements or other sophisticated stuff.
But even then as this is not perfect either you should be able to tune the flow rate/EM.

You will only make it more complex but less reliable.

7

  1. whether the sensor works or not has no relevance for the issue I mentioned. I raised the issue that if someone wants to correct the flow for ANY reason, as of now there are no solutions besides hacks (changing the rotation distance or the flow multiplier). I specifically mentioned that the filament diameter is not the only reason for my request.

  2. you may think it’s not reliable but filament roundness is proven to be quite good actually, see just as example, not as only source, https://youtu.be/RYgdLPe_T0c and see what the actual results of a filament diameter sensor in Klipper can be: GitHub - Deutherius/Filament-Width-Comp-Experiments: A couple of results from filament width compensation using a single mechanical/hall-effect based sensor
    Summary: no damage with good filaments, potentially impressive results for poorer filaments, and almost no need to calibrate the flow when changing filament.
    Moreover… It’s just a matter of using two Hall filament width sensors, if you really want, at 90° from each others and use an opamp to average the two readings! Super simple.

So independently from the remark being off-topic, I’m not sure about the basis for your belief.

8

Despite the fact that even good filament is often not perfectly round different filaments need different flow rates due to shrinkage and other tolerances.
And with a good printer you will see deviations of the cross section of a filament string.

Back in the days I measured the filament with an outside micrometer to get the same settings and flow for the same filament type from the same vendor.
And there are some settings in the slicer where you want to have control over the flow/EM.

So it is not really off-topic.

9

I am exited about what can be done with the load-cells.
The nozzle of the CR-10 SE is mounted on a load-cell, as opposed to the K1 and Ender KE which have it under the beds.

  • Print collision detection
    Whenever the nozzle hits curled up or over extruded plastic.
  • Extrusion over pressure or blockage.
    The nozzle flexes at high extrusion rates, because the load-cell mount is very flexible. ( Actually too much, which is a fixable hardware design flaw.)
    We would be able to detect the flex and throw a warning/error.

The challenge would be to filter out noise. There is a LED on the extruder PCB which lights up at the slightest touch to the nozzle, even high acceleration/jerk or an unbalanced fan registers!

@garethky I was going to point you to the Creality implementation (prtouch_v2), but they chose to close the code…
See my comments on
https://www.reddit.com/r/Creality/comments/1bnejkc/cr10se_open_source_and_firmware_update/
(u/Former-Specialist327)

10

@garethky Never mind, I see I am a few years late to the party.

2 Likes
11

@koconnor I totally agree on the homing side of things. While I can’t admit I fully follow the code (I’ve been digging through it and piecing together a mental model) I know it does result in offsets when probing/homing with a toolhead based probe and multiple z-axis steppers.

I’ve been trying to think of a way to more closely tie the stepper movements together to do like you said, a constant stepper movement not a constant toolhead movement. Because, especially when you try to do things like Quad Gantry Level, that’s what you want. You KNOW the toolhead is in a bad position, you want to know how the steppers are.

I’ve also started poking around in the firmware code for the mcus to see if I can tighen up some of the code to decrease the srtt any to help improve Multi_mcu homing and eventually support Multi_mcu on a shared axis (z).

12

Are there individual threads for each of these issues?

It would probably be a good idea to see how other projects handle plugins, and document them somewhere. I almost wonder if a wiki style would be worth it. Issues with config and plugins aren’t new, so seeing how it’s been handled in the past and what “out of box” solutions exist could save a ton of work.

Throwing my hat in the ring for the build issues. Modern CMake could solve some of these problems, since cmake-presets is JSON! Seriously, there’s a ton of legacy cruft and a learning curve, but CMake is designed to handle the things that are currently coded by hand in a much simpler way.

13

98% of Klipper is programmed with Python, using CMake seems to me like trying to solve patching a hole in the wall by filling it in with dirt from a dump truck. Maybe that’s a harsh analogy, CMake is just extremely clunky every time I’ve used it and is only really useful on older languages that don’t have the modern tools like we have now (hence the C in CMake).

In my mind the best bet to the plugin solution will take some refactoring of Klipper itself, it’s already fragmented a bit because of the dynamic loading of modules that have been added in over time. For example, Controller fan, temperature fan, generic fan and fan are all in different python files with their own classes instead of in one single file where the different fans inherit from a fan base class.

That’s because the config parsing setup looks for the files that have the corresponding name, aka, [temperature_fan] looks for temperature_fan.py
This means that EVERY addition that changes the cfg will require ANOTHER .py file. That was smart in the beginning but Klipper has grown by leaps and bounds. I just counted 113 python files in the “extras” folder

My personal opinion is we move towards a more robust config parsing, where maybe anything with the name “fan” checks out the “fan” file which has a factory method that will dynamically create the right fan type based on the config settings. In other words, You define a fan and add a heater, the fan class sees that as a heater fan. No problem. You define a fan with a heater and a temp sensor. You get a warning “Cannot define a fan as temperature controlled and heater controlled simultaneously.”

I’ve been working on something similar to that as a “proof of concept”.

I would invision that plugins are handled in a similar fashion. That after klipper is cleaned up to consolidate the existing functions and pare down to the fundamentals of 3d printing (Heaters, Steppers, Sensors, Fans etc.) then there could be some revised documentation written that says “Okay, Here are the base classes and what they do, if you want to write a plugin to expand on these here’s how”.

Then the plugin can inherit the functions that already exist and add to them. Instead of shoving in another file and telling klipper “Here’s the 114th file for you to reference”. It could say “Here’s some configuration regarding a fan that I don’t recognize… Are there any fan plugins loaded that describe this new functionality?”

Now, The question is, How do you prevent it from getting right back to to where we are right now?

I personally thing refactoring makes it easier to add the “best of the best” plugin functionality to the core functions without a ton of additional work as long as people don’t try to go too crazy with things like “I made a plugin that controls 8 extruders in a sphere hooked to DC motors that spin them and slip rings for the connections and now I can 3d print in full 360 degrees! I think this is the future of 3D printing, please add compatibility!”

Whatever the plugin solution may be, It needs to be…

  • Simple to use (As much as possible anyways)
  • Clearly, concisely and cleanly documented
  • As modular as possible (As in, snaps right into Klipper without jumping through hoops)
  • Is failsafe - If a plugin has an error, throw an error and load Klipper without it

To that last point, I’d be hesitant to let plugins change some of the core functionality. As in, plugins that change the kinematics or low level motion control are off limits. If someone wants to tinker with them they can play with the source code at their own peril.

Last thing anyone needs is someone making a plugin that’s not well tested. Then a large number of people download it and use it without knowing better because it sounds useful. Then the first thing it does is slam the toolhead into the bed wrecking their printer.

14

I was mainly focusing on the Microcontroller C/C++ code.

CMake suffers from the same problem C++ has. A ludicrous number of tutorials and guides act like it’s the early 2000s. CMake is really it’s own programming language, and I’m not going to even try to pretend it doesn’t have some serious quirks. Once you understand the current version, you’ll find things are much simpler.

There are Python based build tools, though i am not familiar with them nearly as much. My reasoning for CMake is the code currently uses Makefiles and shell scripts, so CMake is a logical progression from that in a more standardized method. However, I am not married to it, and would welcome thoughts and insights into how things work now and other alternatives.

I suppose that’s my big thought on the build system. The more the project can move from custom bash scripts to a build system designed to handle these things, the less maintenance and potentially faster builds. Massive corporate codebases with “custom” Makefiles and scripts have shown me the dangers I would like this project to avoid.

15

The make and Kconfig approach that Klipper is using is not the issue and the menu-driven approach is easy to use.

The issue is that

  1. It is a challenge setting up the various interacting products of a complete Klipper ecosystem
  2. It is a challenge for many users to set the correct parameters for building the board’s firmware (or to rephrase: A challenge to follow the instructions for their board)
  3. A challenge to work under a Linux and even worse in a shell environment for 1. and 2.
  4. A challenge to flash the board with the right approach and knowing some unpleasant quirks that are lurking (e.g. using big SD cards can already spoil the success)
  5. Finalizing the printer.cfg for their given setup

Quite unlikely that CMake will do any good in these steps.

16

@Sineos

I think 1 and 2 stem a lot from 3, I’m aware some boards instructions are not the most detailed either, but the biggest struggle I have seen so far is navigation in a shell environment, understanding how to edit files when needed, the influx of users with less knowledge on that seems to be increasing (personal opinion)

The board quirks are another issue, better docs could help a lot on that front I believe.

As for printer.cfg the Klipper samples are usually a good starting point for a lot of users, again I think some more detailed documentation (specific for each board maybe) could help here.

Maybe it would be worth having a Wiki where board configurations, guides, or other information can be gathered wouldn’t be a bad idea, currently the board configuration docs are spread on GitHub on each board repo.

More automated config generation flashing etc, from a WebUI might be worth looking into, but I feel this should be something split from Klipper itself, I personally wouldn’t use it, but I do recognize the value it brings.
MainsailOS and RatOS are 2 I know of that bring some of that, RatOS seems to handle more config generation as well, MainsailOS seems to be more of a base OS only.

17 
 I personally wouldn’t use it

Man I would, I have a shell script on my Pi that compiles and flashes the Klipper updates to all my CANBus toolheads (I have 7 of them). Right now I have to SSH into my Pi and navigate to the directory and run the shell script. Is that insanely difficult? Not really, but how nice would it be to update and flash them all when I use the existing Mainsail functionality to update my host and do it all at once so all my mcus are in sync with the newest Klipper?

Also, You just made me think of a possible idea but it might not work without some tweaks.

Maybe possibly adding in some functionality in the UI to generate a “shell” of a config object to help people out. Even if it was straight from the Klipper website.

In other words, If I want to add a controller fan, I could use a dropdown and select controller fan it populates the generic config setup but commented out and some simple explanation for each setting and the user can edit it and uncomment.

It wouldn’t be necessary for everyone, but it might make things easier for newer people. The only enhancement I can think of on top of that is having board config plugins for popular boards to help populate pins. Something similar to the pin aliasing system but more plugin oriented.

18

I’m willing to bet you have a very different use case from most, I only need to flash 3 and it doesn’t happen that often (but I do have the configs for each saved and a small shell script as well)

Isn’t this quite close to the sample configs here?

19

I only need to flash 3 and it doesn’t happen that often

Depends on how often you want to keep your Klipper updated, there are minor fixes pretty often. It would be nice to know they’re always in sync. I used to have 3, I just recently converted all my Z axis to CANBus (which I wouldn’t recommend unless a fix I submitted in a pull request gets added).

Isn’t this quite close to the sample configs here?

Yes, but instead of having to copy and paste it from there and then reference the Klipper config website it could be done all in one place. It’s more of a user friendly suggestion.

That’s usually the difference between open source projects and a nicer commerical project. Commercial projects by design are based around user experience, the assumption being users don’t want to have to go look for ways to implement something, the system should meet them 90% of the way.

I know Klipper is open source, and I know that’s a highly debatable view for something programmed by volunteers and in this case if enough people thought it was useful I’d be willing to implement that.

As anyone on this forum or in the Discord can attest to, probably 85%+ of the users of Klipper aren’t savvy when it comes to overly technical things. They hear Klipper is good and they follow a guide they read online. Anything past that they have to dig through the documentation (which can be confusing), copy and paste other peoples configurations and hope for the best or… In a lot of cases come to the forums and say “I don’t understand this, What do I do?”

Making things as seemless and easy to use as possible saves everyone time in the long run, at the cost of programming implementation up front.

Which again, I’m not volunteering anyone else for, I’m willing to help out in that regard.

2 Likes
20

retracted as I was in error.