New submission process for feature enhancements

Developers
1

Hi,

I’m planning to change how I handle new feature requests for Klipper. The current process involves submitters opening a PR on github and then a reviewer (typically me) provides feedback. Going forward I’d like to change to a process where feature enhancements are first discussed here on Klipper Discourse, and we can collectively get a sense of those features that are considered highest priority. I’d then like to focus my time on the coding and reviewing of those topics of high priority. Ideally, with new features and new priorities selected every 8-10 weeks or so.

The reason I want to change how I review feature enhancements is that I feel there are some “big ticket” items in Klipper that have been continually delayed. I have limited time that I can work on Klipper and I fear that some high priority items aren’t getting enough attention.

Some of these “big ticket” items include:

  1. The docs need to be upgraded so they recommend Mainsail/Fluidd/etc. instead of OctoPi/OctoPrint. This may also require changes to the install scripts in the main Klipper repo.

  2. I’d like to make it possible for most users to be able to compile and flash the MCU code without using ssh. I suspect this would involve additions to printer.cfg that define the pertinent MCU settings, and changes to Klipper, Moonraker and the Klipper GUIs to support automated compilation and flashing.

  3. Some of the existing PRs really need to get merged (for example, laser support)

  4. I think we need to figure out a way to empower developers to add powerful new commands without using goofy jinja/gcode hacks and without having to add a Klipper “extra” module. That is, some kind of Klipper “plugin system”.

  5. The current “trapq flushing” code is fragile and needs some refactoring (so that we can support multiple toolheads, synchronize toolheads, synchronize manual_steppers to toolheads, etc.). Similarly, the G28/homing system in Klipper is not flexible - improvements are needed to better support advanced kinematics, filament change systems, and similar.

  6. The next Klipper release needs to be tagged and released.

This change in process will, in practice, mean that I’m unlikely to review any new Klipper feature enhancements for the next 6-8 months. If a feature enhancement PR was submitted after July 1st, I’m unlikely to review it. As part of this change in process I’ll look to provide feedback on the PRs opened prior to that (the review feedback, however, may be that the PR wont be merged in 2023).

This process change wont impact PRs that are reviewed by other listed reviewers ( Contributing to Klipper - Klipper documentation ). Nor will it impact simple bug fixes, nor regular code updates from the “established owner” of existing Klipper modules.

As part of this process change I’ll look to update the Klipper documentation to point new developers at Klipper Discourse (instead of github PRs).

Let me know if you have any comments.

-Kevin

5 Likes
2

I believe that this is a commendable strategy. However, I’d like to offer some additional thoughts from my support perspective:

Handling Macro Utilization

My interpretation is that a primary objective in the design is to ensure that the intricacy of features and settings remain manageable. Regrettably, the current frequent usage of macros by users to circumvent absent features or enhance flexibility seems to contradict this goal. To illustrate, many of the klippy.logs we encounter (even from novices) exceed 1000 lines of configuration, whereas a standard printer.cfg has about 130 lines. This circumstance creates various issues:

  • Configurations that are excessively lengthy and complicated
  • Mostly undocumented macros
  • Sourced from a spectrum of quality, from reliable to questionable
  • Most users struggle to comprehend the mechanisms behind these macros.

Derivative Works

There’s a rising trend of derivative works that provide extra modules or reintroduce rejected features. I feel this could potentially pose a risk to the continuity of the Klipper mainline:

  • These features are unavailable in the mainline
  • They are neither officially documented nor supported
  • Updates or modifications to the mainline can be inconsistent for users and heavily depend on the feature creator’s motivation and ability to sync with the latest Klipper code.
  • Combining these extras or features can be challenging for average users and may lead to unintended consequences, particularly in the context of the macro situation.

Features and Flexibility

While your suggested “focus items” are impressive, I perceive them as somewhat technical features that may not directly impact everyday use. Despite not requiring most of them myself, I suggest considering the following additions:

  • Adjustable fan controls, like heating, cooling, nevermore filters, heated chambers, varying trigger points, PID, etc.
  • More comprehensive heated chamber support
  • Gradient color mixing capability
  • CNC support
  • Enhanced laser support (partially included) with proportional support (yet to be addressed)
  • Additional axes

Miscellaneous Thoughts

  • Emphasis should be placed on implementing features that reduce macro dependence, both in terms of integration and new development.
  • Could we consider a feature voting system to capture user preferences?
  • Perhaps it’s just my observation, but many PRs seem to focus narrowly on specific aspects of a feature set. For instance, linear scaling for fans but not a broader scope like flexible fan controls. Would it be beneficial to publish a catalog of requested features, accompanied by a brief specification detailing expectations and potential pitfalls for that specific feature?
3 Likes
3

Thanks for the feedback. I agree with all of your points and concerns. I also agree that slowing down the merging of new feature enhancements could make things worse. I’m open to alternative solutions.

Note though, irrespective of this discussion, I will be spending less time on Klipper in the future than I have been in the past. So, I think solutions that involve more contributors will be important.

I think it would be great to get more feedback from users on what they’d like to see in Klipper.

Just to be clear though, we can’t “vote” to have developers implement code they don’t want to develop nor “vote” to have maintainers maintain code they don’t want to maintain. So it would be great to have feedback, but ultimately it’s the developers and maintainers that make the choices on what they’re willing to implement and support.

That’s an interesting idea that I had not thought of. I’m certainly open to it.

Cheers,
-Kevin

4

The biggest issue to me is manpower…

Companies using klipper should be asked repeatedly to provide enough funding to pay one or half person salary to speed up (high quality) review and merge of PR.
As of now Kevin is the only one doing the final review, it seems.

It cannot go on this way or the project will slow down.

Qidi Biqu and Crealitg should have enough budget for that.
Bambu firmware should be decompiled to find out where they got the code, which they likely haven’t written from scratch themselves.
And so on.

Many things will become easier or possible once another person can work full time or almost on it.

1 Like
5

Hi Kevin, Sineos and Klipperites :stuck_out_tongue:

…feature enhancements are first discussed here on Klipper Discourse, and we can collectively get a sense of those features that are considered highest priority.

This is definitely a great approach. The tricky bit is participation I would guess. Most users are not very vocal…This can however be addressed as discussed later.

I’d like to make it possible for most users to be able to compile and flash the MCU code without using ssh

Very nice if that was possible…

I think we need to figure out a way to empower developers to add powerful new commands without using goofy jinja/gcode hacks and without having to add a Klipper “extra” module. That is, some kind of Klipper “plugin system”

Some functionality will require changes and/or additions in the main code. With the “plugin system”, developments that are basically stand-alone could be made available to users at their own risk (just like OctoPrint plugins). So far, so good. For functionality which interferes with the main code base, I suggest to clearly outline development guidelines, as in, what developments are clearly NoGos such as error recognition and prevention rather than reporting it to the user.

If a feature enhancement PR was submitted after July 1st, I’m unlikely to review it.

:frowning: I just resubmitted the basic zhop PR…anyways, different topic.

Just to be clear though, we can’t “vote” to have developers implement code they don’t want to develop nor “vote” to have maintainers maintain code they don’t want to maintain. So it would be great to have feedback, but ultimately it’s the developers and maintainers that make the choices on what they’re willing to implement and support.

IMHO, features should be voted on by the users to get a clear sense of what has priority to the user and what doesn’t. However, features and/or improvements to be voted on should only be submitted by developers that are willing to put in the work if their feature receives good feedback. Regarding users usually being quite silent, OctoPrint solved this by pushing messages and vote topics to users on a regular basis. Same was the case for plug in developers, who could update their user base this way. Is this something that could be implemented by our friends over at Mainsail and Fluidd?

Would it be beneficial to publish a catalog of requested features, accompanied by a brief specification detailing expectations and potential pitfalls for that specific feature?

This would be a great pro-active way to get user feedback and ideas for developers. I second that.

It cannot go on this way or the project will slow down.

Totally agree, however, given Klipper’s license, companies have no obligation to fund any development. There is a way to fix this though. Change the license to non-commercial use…This is however entirely Kevin’s call.

In any case, thank you all for putting in the work and making Klipper the best firmware out there.

Cheers
Florian

6

This is a tricky one, and I’m torn.

Pro “open license”

  • Recognition by commercial entities is a clear sign of success
  • Some actually do sponsorship
  • Increases Klipper’s reach

Con "open license

  • Some just take the profit, e.g. my most hated example, the Creality Sonic Pad and its marketing blather about making the printer faster. It is not the lousy pad that makes it faster, it is Klipper. (Positive aspect: Even this fosters Klipper’s recognition and spread)

I am a strong believer in OSS and would always vote for being as open as possible.

Perhaps something like this could be effectively combined with the sponsorship scheme. Existing sponsors could volunteer to offer a “bounty” (hardware or monetary) as an incentive if developers successfully implement features from such a list.

1 Like
7

I am a strong believer in OSS and would always vote for being as open as possible.

So am I. However, on all my own designs, I always exclude commercial use. If a company wants to make money off my work, they should pay for it just like they make their customers pay for it. There is no glory in developing somebody else’s product IMHO. Ultimately, by prohibiting commercial use, you basically force sponsorships rather than hoping for the goodwill of companies. For personal use, just like Fusion360, Onshape, etc. free-use is the way to go as it activates a wider user base and makes product development much more agile and diverse.

Perhaps something like this could be effectively combined with the sponsorship scheme. Existing sponsors could volunteer to offer a “bounty” (hardware or monetary) as an incentive if developers successfully implement features from such a list.

That is a very cool idea! Sounds a little bit like a feature kick-starter setup :slight_smile: Nice.

Another idea to accelerate feature submissions would be to (partially) replace reviewing by in-depth testing. Instead of “only” having a tried and tested main line of klipper, there could be an “alpha”-line, used mainly by developers who can effectively figure out where the problems are in a new code. Every quarter, alpha-features, that have not been reported to have issues for a period of 3 months, are moved to a “beta”-line which is used by more advanced users without developer aspirations. Any feature, where the initial developer does not address bugs in a timely manner, are simply not moved ahead and eventually removed from the alpha or beta line. If the features work without issues in the beta line, they could be merged to main line without extensive reviewing given that the debugging should be mostly accomplished already by moving the code through the different alpha and beta lines. This would require proper error handling to be able to pin-point error sources, which can be mandated by a design-philosophy and guidelines section on the Klipper documentation. This approach is similar to the “Experimental” features in Cura. Just a thought…Comments?

Cheers
Florian

8

Sounds interesting.
One challenge I see: Let’s say you have 20 new features in the alpha pipeline, and suddenly non-specific errors like “timer too close” start popping up more frequently, but still intermittently. Now you have to debug all 10 of them to find the culprit.

In my opinion, a review process is mandatory, at least a light one:

  • Ensure certain quality standards, e.g. code, documentation, etc.
  • Ensure a consistent code style and maintenance aspects, e.g. avoid patching the patch of the patch instead of doing a refactoring
  • Ensure that at least no critical code areas are affected (there is a difference between making an M204 command say “Hello World” or working on step compression etc).
  • Maintain a certain vision/philosophy for the overall product
9

One challenge I see: Let’s say you have 20 new features in the alpha pipeline, and suddenly non-specific errors like “timer too close” start popping up more frequently, but still intermittently. Now you have to debug all 10 of them to find the culprit.

Good point. Github version control would certainly help to figure out the culprit. If all developers are using the Alpha branch, which they can also roll back once some issue pops up, figuring out the issue fast would be in everybody’s interest. Hence, every developer becomes a reviewer and trouble shooter :slight_smile: Anyways, this might also be a bit too optimistic…

In my opinion, a review process is mandatory, at least a light one

Absolutely agree. To enable light review, an honor code could help. Meaning, developers who want to merge features into Alpha not only provide a copyright sign-off but also confirm that they adhered to best-practices and standards as defined by Kevin. This would ensure klipper’s philosophy, quality standards, code style, clear identification if core components are touched, etc. Before merge to Alpha, the Alpha-Maintainer (yes, there will have to be someone) would do a light screening before merging a new feature into Alpha. This light screening would focus on checking that the honor code has been respected.
For my personal taste, I would insist on extensive commenting of the code to understand the developer’s intentions. This makes collaboration and troubleshooting much easier than “just” having plain code to look at.

I personally would sign-up to Alpha to get access to the latest features and test out as much new stuff as possible. Currently, to test new developments, you need to switch repo’s and branches, loosing functionality that may lead to shutdowns due to required config changes…It is just not very straight-forward. An “official” Alpha could help here…

10

I think there are two key aspects that would really help:
Proper code style and documentation(from people who propose code and the current code base), cause right now it feels as if every python class has it’s own codestyle which makes it even harder to debug and write new features (I know this takes time and resources but I think it would be better to stop development on new features for let’s say a year and just focus on code cleanup than go the riot way and have an unmaintainable piece of spaghetti in the future[I don’t want to attack anyone but I have seen too many problems resulting from that and it’s way more work to fix in the future than right now])
The other aspect ties into th first one: get more head-devs/maintainers/whatever you like to call it cause I think Klipper as it is right now is way too big to be handled by one person alone, especially if that one person wants to have a life outside of klipper.
The new devs could be people who have proven themselves with good code already, or kevin does applications or I don’t know.
Right now I also think a lot of people just don’t want to open a PR cause they think it will just be ignored(again, I don’t want to attack anyone, I know that Kevin has a life as well and can’t solely focus on klipper).

11

Thanks for all the feedback.

Your z hop stuff was submitted prior to July 1st - I will try to review it regardless of the discussion here.

I’m happy with the GNU GPL v3 license for Klipper. I don’t have any plans to change the license. (There are also many copyright holders now, so it’s not clear I could change the license even if I did want to.)

FWIW, in my experience, “testing branches” do not provide much value in practice. What I’ve seen happen in other projects is that they tend to consume precious contributor time and rarely catch bugs. The main issue I’ve seen is that merging, remerging, tracking conflicts and coordination require active work. However, when users (and other developers) run into an issue on the “testing branch” they tend to think, “I guess that’s what happens on a testing branch” and the issue rarely gets reported or fixed. So, in effect, the coordination effort increases but the benefits don’t.

That said, I think it’s fine if someone wants to run a development or testing repo. I think it is great when more people to run tests, provide feedback, and try new experiments.

Cheers,
-Kevin

2 Likes
12

As an update, it did not seem there was consensus on using Discourse over Github for development discussions. I still plan to try to focus more of my time on new features instead of reviewing of features ( Klipper development goals for 2024 ). However, I don’t think we need to encourage Discourse over Github for discussions.

Cheers,
-Kevin

13

I see many contributions which are lost due to lack of reviewers so I could review documentation changes, if it can help reducing load on other reviewers.

Is there a procedure to follow? for now I’ll do it and ping once done? or just do it and see if an official reviewer pops up within few days?

14

There are some guidelines at Contributing to Klipper - Klipper documentation . It helps a lot if reviewers are frequent contributors as well (as hands on knowledge of the whole system helps when reviewing changes to individual sub-systems).

-Kevin

15

I still plan to try to focus more of my time on new features instead of reviewing of features

Forgive me if this is too harsh but, this is what I see on every company I have been
(more features, no time to review/cleanup) and given enough time makes for code harder to maintain, sometimes we need to stop and clean things up a little, improve from our experience in order to more easily move forward.

16

I think you misunderstood Kevin’s statement.
He wants to spend time on developing Klipper’s core functionality and exactly doing such chore work like refactoring etc. See Klipper development goals for 2024

He does not want to spend his time reviewing PRs with various “spot-light features” that really finally add to code complexity, maintenance burden etc.

17

Maybe I misunderstood, but to me the sentence seems pretty clear, more time on new features, not reviewing existing features (cleaning up and refactoring), but I think Kevin is the best person to answer that.

I did see that topic as well, besides the 2 quoted below it all seems like new features to me.

18

Check his commits and you’ll see @Sineos got it right.

19

On the other hands, if he doesn’t do review, who does? there are quite a number of useful additions (not everything lost is) which get dropped due to lack of reviewers.

It’s a pity.

It would be good to try to ask and ask and ask the companies using Klipper to contribute enough to pay a couple of salaries or more, so that not everything is free work.

Or maybe it happens already, I have no idea whether Kevin or others are working full time on Klipper.
For example, the author of SuperSlicer is now paid by a company to work 100% on SuperSlicer (80% for features related to the company, 20% for his own interests, if I remember correctly).

The development is now moving quite fast since he started working full time on it.
I can’t pay Kevin a salary, but given that Creality, QIDI, and at least another three use Klipper, it would be nice to see something similar

20

Maybe we need to start a campaign drive for people to donate via Ko-Fi/Patreon. “If Klipper users donated $1 a month we could reach our goal of financially supporting Kevin so he can devote his time solely to Klipper”

I’m joking of course, that’s so full of ethical conundrums and implicit expectations it would be a nightmare.

I know there are a few “official/unofficial” reviewers right now, mostly related to people reviewing changes to portions of the code they created.

I don’t know the logistics of how it would be implemented, but an idea might be to take some “applications” for a first line of approved reviewers and narrow down who has the skills and technical knowledge/programming knowledge to effectively review changes.

That way Kevin isn’t going about this alone, he can concentrate on the ones that have already passed the first past reviewers.

The big issue (and maybe solution) with that is, Klipper is kind of a beast behind the scenes. So many things interact it’s hard to jump into it and get a good “feel” for how it works without slogging through it.

So possibly if there was a small set of approved reviewers there could be a direct line of Q&A with Kevin for finer details so he’s not just answering questions from the random posts/PRs/FRs etc.

All that has it’s own layer of complexity as well.