6.1 FamilySearch schema changes - bugs and suggested simplifiction

Development
1

Hi devs (especially @Nick-Hall & @SourceAirbender),

sorry for bringing this up only now, but I did a thorough analysis of the interaction between the FamilySearch-related schema changes in maintenance/gramps61 and Gramps Web API as well as Gramps Web Sync and found a few bugs - but more importantly, the fix for one of them, namely making the sync object nullable in the schema, makes it possible to greatly simplify 6.1 by completely skipping the database migration. There is no need to rewrite every person record in every Gramps installation on the planet just to add an empty field. Full report below.

Proposals

These changes are small, but they are only cheap before 6.1.0 ships. Once released, the data shape is fixed in every user’s database, and changing it later needs another migration.

P1. Make Person.familysearch_sync nullable (None by default)

Currently, every person carries an empty FamilySearch object, and the v22 upgrade writes one into every record.

  • Optional feature, optional data. The integration is off by default, and earlier in this thread there was agreement that users should be able to disable it completely. A disabled integration shouldn’t leave FamilySearch data in every person, every export and every API response.
  • It’s metadata, not genealogy. FS status writes intentionally don’t update change, which is correct: change should track genealogical edits only. Integration metadata outside change tracking shouldn’t be a mandatory part of every Person.
  • Code and schema agree. Gramps’ JSON schemas declare no required properties, so a Person without this key is valid according to the schema, yet loading one crashes Gramps today (bug 1). Making the field nullable brings the code in line with the schema. It’s also what allows 6.0 and 6.1 to share trees (P2).
  • Sets the right pattern for future integrations. Earlier in this thread, one sync object per integration was suggested as the way to support other tools. If each one adds a non-null object to every person, each needs a full rewrite of the person table and makes every record bigger. With nullable fields, adding an integration costs nothing for users who don’t use it.
  • Small change, existing precedent. Only the Person class, its FamilySearch mixin and the method that clears the status need to handle “no value”. The FamilySearch feature code itself doesn’t change, because it reads and writes the status only through two database methods. Nullable structured values already exist: MediaRef.rect uses "oneOf": [{"type": "null"}, …].

P2. Release 6.1.0 without the schema bump (stay at v21)

With P1, records without the key are valid, so gramps_upgrade_22 no longer serves a purpose.

  • No forced migration for every user. No upgrade dialog and no rewrite of every person row, which takes noticeable time on large trees, for a feature most users never enable.
  • 6.1 stays reversible. A schema bump is one-way: after the upgrade, the tree no longer opens in 6.0. Without it, users can go back if they hit a 6.1 problem, and they can share trees with people still on 6.0.
  • Seamless server upgrades. Server setups like Gramps Web host many trees. A schema bump means a database migration for every hosted tree as part of the upgrade. Without it, moving a server to 6.1 is a plain software update.
  • Beta testers are covered. When 6.1.0 opens a v22 beta database, it replaces the empty FS objects with null, keeps real FS data, and sets the version back to 21. For trees that do contain FS links, a small 6.0.9 change to json_utils lets 6.0 read, edit and preserve them. Older 6.0 versions fail on those persons instead of losing the data, and this only affects users who enabled the integration.

P3. Ignore familysearch_sync in the database diff

diff_items() in gen/merge/diff.py already skips change. It should skip familysearch_sync too.

  • It’s metadata that changes constantly. FS status is updated on every FamilySearch refresh. The database diff, used by sync tools, Import Merge and the Database Differences report, shouldn’t report that as a genealogical difference.
  • Required for P2. Once some persons have the field and others don’t, the current diff fails with a KeyError whenever only one side has it.

EDIT: Sorry, I removed the section “Bugs” that was here before because I realized it was misleading/wrong.

I’d be happy to submit PRs for the proposals if you agree.

2

Hi @DavidMStraub

So P2 requires a Gramps v6.0.9 release, is that correct?

Whatever decision is made, it’s great that you’ve raised this while 6.1 is in beta. Thank you.

3

Corrected the description, it was partly misleading.