Proposal: process for upstreaming Web API improvements into Gramps core

Development
1

Hi Devs,

I’m happy about the progress the Gramps ecosystem is making this year. Lots of improvements in Gramps itself and also in Gramps Web, plus Gramps Web API (the REST API Gramps Web uses as backend) is finally growing the ecosystem that was intended from the start. In addition to Gramps Web and the Web Sync Addon, there is now also Gramps Connect, GrampsWebApiDb, and several Gramps MCP projects building on it.

Progress on Gramps Web API is accelerating by a concurrence of ever growing AI coding capabilities and increasing popularity of Gramps Web (more than 2000 deployments, double the Github stars of Webtrees), both effects increasing the number of contributors. For illustration, the last four releases of the Gramps Web frontend each had two new first-time contributors!

What I’d like to discuss against this background is how to better define the scope of Gramps Web API vs. what I call Gramps core (essentially, the gramps.gen library and the core database plugins). Gramps Web API essentially has two separate roles to fulfill:

  1. Wrap Gramps’ genealogical and database functions in a way to make them accessible via HTTP requests
  2. Provide scaffolding that is needed for a web-based application - authentication, permissions, background tasks, search indices, caching, etc.

What was never the goal was to reimplement the genealogical and database functionality in Gramps core - in fact, I’ve always seen it as the big plus of Gramps Web API compared to other web-based genealogy apps that we are building on a well-tested foundation and can focus on the two points above. What I absolutely do not want is for Gramps Web API to become a web wrapper around a Gramps fork!

While that sounds good in theory, in practice there are parts of Gramps core that don’t/didn’t work well for a web-based app; for instance, there used to be for-loops over SQL queries that were reasonably fast on SQLite-on-HDD, but prohibitively slow when used with a database over the network. Since Gramps Web API has always had a very different release cadence than Gramps core, in the past we’ve often patched things in Web API, not always contributing the changes/improvements back to Gramps core. Examples include Web API’s ModifiedPrivateProxyDb or CachePeopleFamiliesProxy, which are not much more than ugly monkey-patches.

The problem with each performance optimization or feature addition that shadows one of Gramps core’s database or genealogy functions is that it leads to drift, which generates a maintenance burden and potential security risks. I noticed that already any time there was a major/minor version bump in Gramps and I had to adapt the patches accordingly.

Triggered by the fantastic efforts of @dsblank to remove performance bottlenecks in both Gramps core and Gramps Web API, I think it’s time to formally define the scope of the two libraries and to define a process to upstream improvements from Gramps Web API to Gramps core.

Ideally, I think all database specific code should live in Gramps core, except for the parts that are Web API specific (that is, not needed for Gramps desktop). As for defining what is Web API specific, I think we shouldn’t be too strict: for instance, Gramps desktop might not need the get_dbname or get_summary methods on PrivateProxyDb, but adding them doesn’t hurt and doesn’t require a lot of maintenance (compared to patching the class in Web API just for this purpose).

What is more important, in my mind, is to define how changes that should live in Gramps core should be contributed to Gramps Web API, without having to wait for a new minor release of Gramps core. Here are the requirements I think such a process should have:

  • It should not require the Gramps project to change its release cadence or development process
  • It should allow timely adoption of changes by the Gramps Web API project
  • It should not inflict an undue maintenance burden on the Gramps core maintainer or the Gramps Web API maintainer
  • It should not inflict an undue bureaucratic burden on the contributor or the maintainers
  • It should ensure long-term drift between Gramps Web API and Gramps core is minimized for database-access and genealogical functions

Here is the process I propose to meet those requirements:

  1. Contributor opens a proposal (could be in Discourse or the Web API repo - let me know what makes most sense in your opinion) requesting an improvement to a database or genealogical function that benefits Gramps core and Gramps Web API
  2. The Gramps core maintainer and the Gramps Web API maintainer independently confirm they agree with the proposed change. If the Gramps core maintainer disagrees, the change can still be reviewed as a normal contribution to Gramps Web API.
  3. The change is implemented as a patch of the current Gramps core version and contributed to a new module, gramps_webapi.future_gramps. A PR is opened against the next Gramps minor version.
  4. The change goes live in the next Gramps Web API minor release
  5. When the change in Gramps core is merged and released, the patch in Gramps Web API is removed by the original contributor.

I think this process would be better than things we did in the past - such as monkey patching without upstreaming at all (my own sins) or moving changes into an external library to keep them out of Web API (as we did in PR 913), because neither of those prevent the drift that I’d like to minimize.

One possible issue with the approach could be that the patching in step 4 might not be feasible if the change in core is too extensive. But I suspect this wouldn’t be a problem most of the time.

Let me know what you think.

2

Thanks @DavidMStraub for the continued thoughts.

For some database-specific things (like finding relationships) it might be easier than you describe. We’d need to:

  1. Create some “business logic” functions (like find relationships) in the basedb with default implementations. This idea hasn’t been rejected outright, and we’ve had some discussions.
  2. Override the business logic functions in subclasses (SQLite, PostgreSQL, etc)
  3. Move the database layers out of core and into addons
  4. When we build the core packages, grab the addons so that they are always packaged (not sure exactly how updated addons would be handled, maybe they wouldn’t be updateable)

Then addons can continue like they do, and the code can live where it belongs—with the database. There are some details I glossed over (proxies and filters) but I have ideas.

Other contributions, like a SQL-based query-language… I am not sure. That seems fine to be a stand-alone package in gramps-project.

Of course, I am open to any amenable solution.

3

Just so I understand what you mean: are you saying for some improvements no process is needed because they live in Addons in addons-source that have their own release process? If so, I agree of course.

But if it’s about modifications of the basedb, that sounds again like a Gramps core change. If it’s something that will land in, say 6.2, and we can wait for it, there’s no problem anyway and no need for a specific process.

To be clear: the process is meant for changes that belong in Gramps core long-term, but should land in Web API sooner.

And the “not rejected outright” and “I am not sure” part is what I hope would turn into a transparent decision process.

4

Yes, they can be released fairly quickly, thank you @GaryGriffin !

I hope we could fast-track a change in gramps-core if it unblocked gramps-web-api.

The first is a gramps core decision (approval for business logic in the db) is for @Nick-Hall. The second (can an addon override a built-in) is a technical question (I will explore).

5

@dsblank I’m still not sure we’re on the same page.

I’m proposing a process that will allow changes like Relationship lookups are exponentially slow · gramps-project/gramps-web-api · Discussion #965 · GitHub to be accepted in Gramps Web API.

Now you’re arguing I don’t need a process because things can be fast-tracked in Gramps core. So are you suggesting the process should instead be to reject all changes to Web API that are in Gramps core’s scope?

6

I’m not arguing, but merely pointing out some technical changes that could:

  1. alleviate the need for a formal process
  2. re-arrange components to live in the database layer

That would still require relatively minor changes in gramps-core, but that is something that could be considered more like a bug fix than a feature (and thus could appear in the next release).

But exactly how we get to the above-described organization may take some time. But before we start planning, we need to know:

  1. Can we change the philosophy in gramps-core to allow “business logic” methods in the database? These would have a base implementation in Python, but subclases could override them (eg, use SQL).
  2. Can we make a PR for gramps core to add the initial method get_relationship(table, handle1, handle2) into the base database? To make a very small change, gramps-core need not use the method initially.

If the answer to those is “yes” then we can make that be the goal, but have intermediary transitional code in gramps-web-api until that is deployed.

7

Yes, but the way we’re arriving at this “yes” or “no” and the implications that follow is not sustainable for me as a maintainer right now. It takes way too much effort & time. I want to reduce the effort on all sides and maximize the benefit.

8

Yes. We should be able to discuss this without reference to any specific change.

From the beginning I was aware that we may be putting things into the Web API that would better belong in core. Your proposal seems to be reasonable to me. If we agree changes in advance and the patches are relatively small and the license is GPL2+ then this could be a way forward.