I made a thing: GrampsWebApiDB

Development
,
1

BETA BETA BETA BETA BETA BETA BETA BETA

I made a database addon called GrampsWebApiDb that does something a little unique: it allows you to use Gramps on the same database that you are using with gramps-web. Why? I mean you can already use the GrampsWebSync addon right? True, but this is instant, all of the time. At least that is the idea. It needs more testing and could eat your data.

You can read more about it here:

Once you have defined a GRAMPS_WEB_API_KEY, some things you can do:

  1. script some changes for your gramps-web database using standard gramps
  2. you could share a DB with a friend without gramps-web, using gramps-web-api directly + gramps. You could use a postgresql db on the web, but this respects gramps-web-api permissions and accounts
  3. automatically download your gramps-web data nightly

Probably some other things.

Why did I make it? I was thinking about making an SDK for gramps-web-api but wanted to use all the standard gramps code. There is a gramps-web-api-client that is similar, but it operates with native gramps-web-api JSON data.

2

This is a really good idea.

3

Thank you for that comment: I really wasn’t sure if others would find this useful. It was really a great learning exercise for me to get to understand gramps-web-api better. I’ll report back when it is out of beta.

4

I’ve added “live-updates” to the gramps db addon, and to gramps-connect. That means if there is any change to the database from anywhere (for example another browser, gramps desktop, or a script) then you will “instantly” (within 10 seconds) see the page update.

Perhaps better to see it in action. On the bottom left, there is Gramps running with the new addon GrampsWebApiDb, top left, a script (shown below), and on the right, the latest gramps-connect. Both the gramps addon and gramps-connect now update when changes are made elsewhere.

In the video, I run the script and the apps both update:

Screencast from 2026-08-05 13-26-19

Here’s the script (simplified):

import tempfile
from gramps.gen.db import DbTxn
from gramps.gen.db.utils import make_database

handle = "E04KQC637O9JLP5PNM"  # I0553, John Adkins in example.gramps

# Requires GRAMPS_WEB_API_KEY to be set
db = make_database("grampswebapidb")
db.load(tempfile.mkdtemp(), callback=None)

person = db.get_person_from_handle(handle)
new_given_name = "JJ"
with DbTxn(f"Set given name to {new_given_name!r}", db) as trans:
    name.set_first_name(new_given_name)
    db.commit_person(person, trans)
db.close()

What is cool is that Gramps app records the change for undo/redo, and changing anywhere affects the others.

This addon may be more useful than I first thought. :man_shrugging:

PS: @DavidMStraub it may be worth adding live-updates to gramps-web. I did it via polling the history endpoint, so that I didn’t need to change gramps-web-api, and the stragey works in all scenarios (eg, desktop gramps).

5

I investigated that - it’s not worth it as long as we don’t switch to an async framework, which would not only be a complete rewrite, but also break the automatic upgrade path for every existing deployment out there. Polling is the right call for now.

EDIT: To be clear, I didn’t investigate that just now, I investigated it a few months ago because I wanted to add WebSocket support, and this is the conclusion I came up with.

6

That’s what I meant. :smile: We don’t have polling-based updates in gramps-web right? (Not “live”, but they don’t exist at all.)

7

We don’t have polling in the frontend except for background task progress bars.

But any edit re-fetches the metadata endpoint, which leads to an automatic cascading update of all components that need fresh data.

I’ve been hesitant to add additional polling - in my experience, most trees can go weeks or months without a single edit, so polling for updates without any reason to assume there has been an update is very expensive, especially given that many users are running Gramps Web on hardware that doesn’t allow running more than 1 or 2 sync workers.

8

If I understand you correctly, with the new GrampsWebApiDb you are updating both the local database and the remote database at the same time. How do you handle two-phase commit?

9

I’m not sure I am answering your question, but as far as I understand it, Gramps core has no two-phase commit at all — it’s a plain single-database local transaction model, and there’s nothing distributed for it to coordinate.

GrampsWebApiDb (which does have two participants, the local mirror and the remote server) can’t just reuse Gramps idea of a transaction: core’s model has no concept of “prepare on two sides, commit only if both agree.” The addon layers its own best-effort push-after-local-commit logic on top precisely because core gives it nothing here — which is the “not real 2PC, optimistic-concurrency-plus-resync instead” design covered in the previous answer.

Having said that, I did just make some changes to handle conflicts:

Before: if you edited a person on your desktop while someone else had already changed that same person on the server (or another device), the server would say “no, that’s stale” and reject your edit. Gramps would quietly grab the server’s latest version and overwrite your local copy with it — your edit just vanished from the server’s point of view, even though it briefly looked like it had gone through.

Now: when that rejection happens, Gramps still grabs the server’s latest version first (so it’s not fighting over stale data), but then it automatically tries your edit again, applied on top of that fresh version — like reapplying a sticky note to a page after someone slipped a newer page underneath it. Most of the time this just works, because most conflicts aren’t really about the same field — you edited a birth date while someone else added a source citation, say.

The one thing GrampsWebApiDb still doesn’t do: if someone genuinely edited the exact same field you did (e.g., you both changed the same person’s death date to different values), there’s no “merge” — whichever edit gets replayed last simply wins, with no popup asking you to pick. That’s the deliberate simplification: solving the common “different things changed” case automatically, while leaving the rarer “we both changed the same thing” case as a known limitation rather than building a full compare-and-merge dialog. If necessary, we can add that as well.

I also added a safety valve so it only retries once — if your edit conflicts again right after the retry (a genuinely fast-moving edit war on the same record), it gives up and logs it rather than looping forever.

Thanks for the question @csam, and hopefully this answers it.

10

On a related note, I see that @DavidMStraub has updated the web sync addon UI:

Note that this addon is really a Gramps Web API sync, and useful for any gramps-web-api based system.

12

Hi @dsblank, can you please make two changes to GrampsWebApiDb? I can also submit PRs if you agree.

  • The XML export endpoint is hit synchronously via GET - this blocks a web worker until the export is finished, which can take a while. The GET endpoint only exists for backwards compatibility and I think we should actually remove it. The Gramps Web frontend uses POST, which starts a background task.
  • Polling is done at a fixed interval. Although it’s fairly cheap, it currently even polls when the Gramps window is minimized or hidden. Can we increase the interval under some conditions, e.g. when the windows don’t have focus, or when there is no activity?
13

Thanks! I made a PR to address these, and related issues:

@GaryGriffin, would it be better to discuss addons-source in a repo issues, or discussion github section?