Desktop: Gramps v5.2.3 on MacOS Ventura 13.7
Server: Gramps Web v5.2.3 (API 2.5.0, FE 24.10.0) on Fedora Linux 40 SE
Once in a while when syncing from desktop to Gramps Web I get a strange error, where the sync says a number of objects on remote site has been deleted and or changed.
I only make changes on the desktop, so this is quite strange. When this happens I just select sync mode “Reset remote to local”, so no harm.
Hi, interesting. Do you always sync from the same desktop computer? Can you please check whether on one of the machines, the time is not set correctly? Sync uses timestamps to decide which side is newer. (The time zone doesn’t matter, only UTC time.)
Both machines uses automatic time setting, though they are not using the same time server (NTP). Both are using the same time zone (CET).
I have been running Gramps Web since late May this year. First time I saw this problem, I thought it was related to I had added a mother with a child without a father, but later occurences of the problem has happened without such special data.
Is it possible to have a sync log ?
The best way to diagnose the problem is to wait until it occurs again, abort the sync, and then export a Gramps XML from both sides and look at the difference between the offending objects in the two XMLs. Also look at the last sync timestamp in the webapisync.ini in your Gramps directory. With this information, we can find the root cause.
It happened again!
Looking at the list in the Gramps Web Sync the IDs listed as deleted on the remote side are all IDs that were added during the previous sync. So my immediate guess is that the previous sync didn’t finish correctly.
I have made a Gramps XML export on both sides.
They were added on desktop and shown as deleted on remote, correct?
That sounds like a timestamp issue then, where the diff correctly identifies the different objects but identifies the wrong side as more recent.
Can you please give me the change timestamp of the offending objects as well as the line containing timestamp in ~/.gramps/gramps52/plugins/webapisync.ini?
They were added on desktop and shown as deleted on remote, correct?
Yes, they were added on desktop and synced to remote on previous sync, but I didn’t check wether they were actually stored on remote
From webapisync.ini:
;; Gramps key file
;; Automatically created at 2024/11/19 17:51:04[credentials]
timestamp=1731694151.061122
From gramps XML on desktop:
<person handle="_fbf0c40b73e2f634cf27472b559" change="1731322631" id="I0684">
This id is NOT in the gramps XML ‘web’
I ran a diff on the two gramps XML files, and it looks like the ‘change’ values are different for all objects:
< <event handle="_7140e784-b2e2-4188-8003-7a379e5c4531" change="1718127215" id="E0709">
---
> <event handle="_7140e784-b2e2-4188-8003-7a379e5c4531" change="1718127020" id="E0709">
Is this expected?
Oh, that’s different, and was the previous sync on November 18, 21:15 UTC?
No, the previous sync was probably on November 15
Ok, because that’s the ini file claims the last successful sync was on November 18. So the mismatch must come from the fact that the sync addon erroneously thought that the November 18 sync was successful. Now we need to find out why.
No, the webapisync.ini file is created as soon as you have logged in and the checking of the two instances are done.
So the webapisync.ini file was from the aborted sync due to “deleted” records on the web side.
Well, it shouldn’t. At least that’s not what “the author” intended 
So it’s a bug. The intention was that the timestamp is only updated after all changes have been successfully committed, or if the databases are identical
Sorry, you’re right. The webapisync.ini is created if not existing after login and with timestamp=0. And the timestamp=0 is left as that, when aborting.
Yesterday I ran a “Reset remote to local” sync.
After a few updates in Gramps desktop today, I started sync.
In the “Final confirmation” I now have “Remote changes: Deleted (14)”
The webapisync.ini:
➜ plugins cat webapisync.ini
;; Gramps key file
;; Automatically created at 2024/11/22 13:41:21
[credentials]
timestamp=1732211649.144118
and the value of the timestamp:
➜ plugins date -r 1732211649
Thu Nov 21 18:54:09 CET 2024
Question: How is data inserted on the remote database? Could it be that a ‘commit’ transaction is missing? Or what happens if the remote database does a rollback e.g. if a long transtransaction is detected?
Ran “docker logs grampsweb” and see various errors, e.g.:
[2024-11-19 10:20:22 +0000] [15] [ERROR] Error handling request /api/search/?sort=-change&query=change:%27-1%20year%20to%20tomorrow%27&locale=da&profile=all&page=1&pagesize=8
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/gunicorn/workers/sync.py", line 134, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python3.11/dist-packages/gunicorn/workers/sync.py", line 177, in handle_request
respiter = self.wsgi(environ, resp.start_response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1498, in __call__
return self.wsgi_app(environ, start_response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1476, in wsgi_app
response = self.handle_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/auth.py", line 44, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/views.py", line 110, in view
return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/views.py", line 191, in dispatch_request
return current_app.ensure_sync(meth)(**kwargs) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/webargs/core.py", line 657, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/search.py", line 154, in get
total, hits = searcher.search(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/search/indexer.py", line 317, in search
results = search.query(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/sifts/core.py", line 335, in query
result = conn.execute(fts_query, params) or []
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: change
also a lot of this error:
ERROR:root:Error parsing list of recent DBs from file /root/.gramps/recent-files-gramps.xml: no element found: line 1, column 0.
This might indicate a damage to your files.
If you're sure there is no problem with other files, delete it, and restart Gramps.
Do you think I should recreate the gramps web database?
Just a follow up.
On November 23 I did a ‘docker compose pull’ and all images except grampsweb was updated with new version.
I also ran a number af database checks, rebuilded indices etc.
I almost do a daily sync, and haven’t seen the problem since the 22nd Nov.