Architectural thoughts on the addon mechanism — API versioning, lifecycle states, manager behaviour

Development
1

Dear Devs,

Recent work on the addons-source PR backlog (lint cleanups, dependency declarations, Py2-era import migrations on a handful of unstable addons) has given me a sense of some architectual issues with the addon mechanism. I’ve seen similar discussions by others here, so I thought the subject warrented a bigger discussion. This is some thoughts I’ve had how it could be improved and I would appreciate the input of the community that has far more experience with it then me.

Some issues I’ve seen in the past couple fo weeks

  • My lint backlog campaign uncovered a class of addons (~13 directories) that are status=UNSTABLE, include_in_listing=False — neither published nor maintained.
  • Every Gramps minor release triggers a cycle of bumping gramps_target_version across ~140 addons. Most of those bumps are no-op code-wise; just a string change. They’re busywork, and authors who’ve moved on can’t do them, so addons rot.
  • I’ve seen questions from new addon writers how they should go about it and a suggestion to move to a similar HACS plugin process for the repo

As far as I can tell from CONTRIBUTING.md and MAINTAINERS.md in addons-source I couldn’t find some of the suggestions here. With that in mind, I would like to suggest following improvements

Introduce an API version

Today an addon declares a gramps target version (e.g. 6.0, 6.1) tying it to a release band. What the plugin actually depends on is the API contract of the plugin framework plus the bits of gramps.gen.* and gramps.gui.*; if the contract changes, the addons don’t need to be touched either. I’d therefore propose to use something like addon_api=“>=1.0,<2.0” with the contract semver-ed:

  • major bump on breaking changes,
  • minor bump on additions,
  • patch bumps for bug-fix-only changes

Each Gramps release declares which addon-API version(s) it provides. The Addon Manager (AM) (and valid_plugin_version) checks the addon’s range against that, instead of literal version-string equality. The hard part isn’t the metadata — it’s that adopting this requires Gramps core to commit to a stable, explicitly-defined addon API surface, with semver discipline. Is this possible?

Lifecycle states

The status= field today (UNSTABLE / EXPERIMENTAL / BETA / STABLE) is overloaded. The way I understand it, it signals both code quality (UNSTABLE = experimental, in-development) and maintenance state (UNSTABLE = abandoned, needs porting). Same flag, two different intents.

Combined with include_in_listing=False, we end up with addons in an ambiguous state. Is this one being actively ported, or is it dead? The CI can’t tell. The AM UI can’t tell users. Maintainers can’t tell what to do with it. I’d suggest an explicity lifecycle field that disambiguates this

  • INCUBATING — new addon under development, or an existing addon being actively ported. Not in user listing. CI gates it (active dev needs feedback). Time-bound: a year in INCUBATING without graduation triggers a review.
  • PUBLISHED — released to users, actively maintained. Full CI, full listing visibility.
  • DEPRECATED — still loadable; AM UI shows a banner (“going away in version X”).
  • ORPHANED — This builds on the existing maintainers= field and the Addons MAINTAINERS wiki page, rather than inventing new metadata. Stays loadable for users who already have it; UI shows “no longer maintained, looking for adopter”; CI gating relaxed (warnings, not failures); explicitly catalogued so adopters can find it.
  • ARCHIVED — moved to addons-source/archive//, removed from build, source preserved for history.

The maintainer-departure → ORPHANED transition is the one that needs process, not just metadata. Possibilities: a periodic cron checking last-commit date plus the maintainers= field, opening a tracking issue when an addon hasn’t moved in 18 months and no maintainer responds. Decision still human; just surfaced.

Addon Manager behavior

With API-versioned compatibility, an addon at addon_api>=1.0,<2.0 works on every Gramps release that exposes API 1.x — no per-release re-targeting, no per-Gramps-version listing files, no “you upgraded Gramps, here’s the gramps61 version of FooAddon.”

What’s still needed:

  • Cross-Gramps-version persistence of installed state. If the user has FooAddon installed and upgrades Gramps, the addon stays installed and working — no re-discovery needed.
  • Lifecycle banners surfaced in the UI. DEPRECATED addons show a notice with removed_in= so users know to find a replacement; ORPHANED addons show “no longer maintained, looking for adopter.”
  • Declared migration paths for addon major versions. When an addon goes v1.x → v2.0 with breaking changes (DB schema, config format), declare it: migrates_from=[“1.5”,“1.6”] with a named hook function. PostgreSQLEnhanced is the textbook case where this is currently ad-hoc.
  • Addon update notifications independent of Gramps update. Periodic listing checks, prompt user when a newer addon version is available.

If you see value in following the direction, I’d be happy to write up a more concrete details and necessary PRs. Thoughts?

2

The addon status was discussed when I implemented the Addon Manager. It is intended to indicate the lifecycle state of an addon, and will typically progress forwards form Unstable to Stable. It is roughly equivalent to the addon rating in our wiki and took inspiration from the mediawiki extension status.

Unstable = In the early stages of development.
Experimental = Usable, but expect many bugs.
Beta = Ready for wider testing.
Stable = Complete.

I considered adding “Unmaintained”, but concluded that the new maintainer fields were sufficient. We can review this though.

The include_in_listing field indicates the published/unpublished state of the addon.

I can’t remember discussing the end-of-life of an addon before. We rarely remove addons, but when we do we just delete them. Anyone interested in finding them would have to look in the git repository.

Gramps core doesn’t use semantic versioning, but does use something similar with our major, feature and maintenance releases. I’m not sure that a separate API version would gain us much since it would likely increment for every feature release.

We already have this functionality, but I didn’t enhance it when I created the Addon Manager.

We also need to think about how we handle running more than one version of Gramps. Especially the case where an addon changes between versions. Are we going to keep the versioned directory structure?

3

Maybe the MAINTAINERS registry could be like the wiki’s Portal:Translators listing. Where “Vacancy” can be ALSO used for addons actively needing an adopter. And the MAINTAINERS_EMAIL could link to a page about how to adopt.

4

Excellent insights @eduralph! There is also a discussion on moving to a semvar version matching for addons here: Refactor addons to use semantic versioning · gramps-project/gramps · Discussion #2297 · GitHub

5

Thanks for the insights. There are currently 12 addons that would be excluded by the CI process because they are flagged “Unstable” and principally would need an either-or. By extending the process into an EOL statuses, we could make sure these addons get cleaned up in an orderly fashion. So adding the statuses

  • DEPRECATED — still loadable; AM UI shows a banner (“going away in version X”).
  • ORPHANED — This builds on the existing maintainers= field and the Addons MAINTAINERS wiki page, rather than inventing new metadata. Stays loadable for users who already have it; UI shows “no longer maintained, looking for adopter”; CI gating relaxed (warnings, not failures); explicitly catalogued so adopters can find it.
  • ARCHIVED — moved to addons-source/archive//, removed from build, source preserved for history.

would really help to ensure that the addons repository doesn’t collect effectively dead code and help maintain overall quality. The CI process I’ve been working on highlighted that with several of those items popping up with lint issues or even incompatiblity because they haven’t been updated from gramps 5.

The benefit would be that the need for maintaince of the add-ons would be much more explicit.

Let’s say the current Addon API interface were 1.6.4; now the gramps project moves from 6.0 to 6.1, adds some minor enhancements - like some I’ve submitted - which don’t really change the basic mechanism so it gets the bump to 1.6.5 in Gramps 6.1; because the addon is compatible with 1.6 it does not require any changes and both gramps versions 6.0 and 6.1 continue loading the addons without any touching of the addons repository.

Let’s say for 6.2 the gramps project decides to change the API Interface, add some new functionality, etc. Depending on the kind of changes you would then create a API version 1.7, signaling to all addons that are compatible with 1.6.5 to check if they are compatible - we can even do that in an automated fashion and notify the maintainers that there had been changes and their addons would not load for 6.2 unless they actually touch it. If the API version moves on and the maintainer does not do something about it, it moves to status “Deprecated”.

If you then say for 6.3 there’s a major overhaul of the API and the version gets bumped to 2.0, you could build a fallback system and have logic loading for 1.X series or 2.0 - if you ever wanted to do something like that.

It just makes the management of the addons much more decoupled and explicit from gramps. Obviously you would most likely do a bump to 2.0 in gramps 7.0 but the point is the API versioning decouples those two efforts.

In that case it would only depend on the API version, not the gramps version itself. It would pay into the discussion dsblank started as well:

What I see there is a discussion around when we would bump the versions under which circumstances which is essentially what a seperate versioning could take care of. It would decouple the logic of where the add-on is from the gramps versioning. Gramps would have to take care of the API interface, i.e. changes to that area need to be considered under that premise, but you have that issue right now anyways.

6

Thanks for the pointer! I see people have already been thinking along those lines. :grinning_face:

7

OK. I think I can see what you are trying to achieve here.

When the API changes we need to indicate to the user that the addon has not been checked for the latest version. They may still choose to install it or continue to use it, and it may work fine, but they may encounter problems.

After the maintainer checks the addon, they may move the status back to Stable if everything is OK. The maintainer or perhaps a core developer may fix a bug at this stage. If the addon has bugs that are not fixed then we may need to indicate that the addon is broken.

Perhaps we could add “Unchecked” and “Broken”?

An addon could become unmaintained/orphaned at any stage in the lifecycle.

I would suggest that “Broken” is treated like “Unstable” and only available when Gramps is running in development mode.

8

Yes. I could see that happening occasionally.

OK. That is worth discussing further.