Checking, Repairing, or Removing Forms data

Development
1

When I look at the backup files that I received from a fellow user, who complained about the corruption of forms data, it is now clear that these data can indeed be corrupted by an operation that may look quite trivial, like the merging of citations. I tried that here, on an old back-up that he sent, and the effect is the same as what I saw after the analysis of a recent backup of the same tree.

Based on what I see in the data after merge, I think it’s safe to say that merging citations has disastrous consequences for forms data, and that users need a remedy, or maybe even more than one.

The 1st probably is prevention: When I run the merge duplicate citations tool, there is no warning about the effect on forms data, and it doesn’t detect the existence of forms data either, meaning that we now have corruption built in Gramps itself. And in this case, it doesn’t really help to see the usual warning about the deletion of history, and the advice to make a backup, because as a user I just see that as a routine notice, which I have learned to ignore, just like I ignore the kid crying ‘Wolf!’. Users are not made aware of the consequences, which seem to be quite bad, especially because they’re almost completely hidden. Users only see the corruptions when they edit an existing form and see that it contains a lot of unrelated persons, not the ones that they entered. And they are also not warned when they run check & repair.

The 2nd remedy would probably be repair, although I doubt that this is possible, which is why I put Removing in the subject too. And that is something that the user doesn’t seem to be able to do either, because the forms data is not visible in our editors, except in the forms Gramplet itself, which shows only a subset of the forms data, because it only follows the 1st link when multiple forms are attached to the same citation, which is exactly what happens after a merge.

When I look at the XML, I can see an easy way to get rid of all forms data, so that the user can make a fresh start, but I really dislike the idea that users may depend on such a trick to maintain data integrity.

So here’s the question: What can we do to help?

2

When I started using GRAMPS back in 2919 I read if you use the forms feature, only use it to edit data. That set a red flag for me, so I never used it.
If there is any feature in a program that corrupts data, it should be immediately removed IMO. To help with this recovery, tools should be made available to correct the data corruption.

3

Enno
Would it be too simplistic to suggest just remove “Merge Citations” from
the tools options or convert it to an addon that you load separately
with dire warnings about the consequences of doing so.
Which would mean it is unfortunate for those who have already suffered
but a short term? preventative.
Follow up work might then find a solution
Phil

4

Well, in this case I assume that the merge citations tool is older than the Forms Gramplet, so it sounds a bit weird to remove it. In this case, at least IMO, the real problem is the Gramplet, which stores data in such an odd place, that it won’t be noticed by any normal human being looking for places where the merge citations tool might cause damage. Spelled in other words: The people who worked on the Forms Gramplet should have warned other developers, that they relied on an undocumented restriction, and they didn’t, or at least, I didn’t such a notice.

As a developer, I really think that the Forms Gramplet needs a big red label that warns all users that it stores data in odd-places, and as a retired professional, my real opinion is that it should have never been allowed to added to our official tools, because it’s so bad. Support for better data entry and proper citation of source data is a good thing, if it’s accompanied by a proper change in the data model, and this Gramplet was allowed without such a change, breaking principles that have been part of computer science since more than 50 years.

As far as I can check, the merge citations tool is quite safe if you don’t use forms, like me, but it can be dangerous if you’re a bit careless with its options, for example when you let it merge citations with notes, or media, or weblinks for that matter.

And in fact, this weekend, I found a similar complaint on the RootsMagic Discourse forum, where a user ended up with mangled web links for the Find A Grave site, after merging citations:

And there too, users should have been protected against this kind of merge, because it creates a complete mess of their citations.

I’m using RootsMagic to synchronize data with FamilySearch, so for me, the program itself is not that essential, but I am in total agreement with user TomH on this subject.

5

I agree that something should be removed, but as a developer I think that that something is the forms Gramplet, because that is the thing that leads users into a dangerous path. And I call that dangerous, because it stores data in odd places, where they are hard to access.

Removing the merge duplicate citations tool may be a good thing for the short term, and I think that it can be changed to protect users, by ignoring all citations that have associated forms data.

And what I’ve seen from the data suggests that there is no easy way to correct the corruption, not even a great reset, because the forms data can’t be removed either, unless you know how to hack Gramps XML. It might also be possible with the Supertools, but that’s also a specialists’ job.

6

Perhaps with the new Addons categorizations for 5.2, the Forms addon can be recategorized? It could be marked as Expert User and maybe pushed back to Development stage?

7

Enno
I confess I came to GRAMPS later into its development cycle so Forms was
available and it fitted my way of thinking so I started using it
extensively with Census Data with apparently no real issue I have tried
using it for other data without the same level of success and indeed
have backed away from it somewhat.
So on balance like the people who use the Merge Tool but not Forms I
would need to keep Forms available for some time and not use Citation
Merge until I have moved completely away from Forms.
So I think the powers that be have a conundrum to be addressed.
The bonus is that I am on Ver5.1.6 and not likely to move to 5.2 on its
release as there are enough differences between the two I would need to
carefully consider that move when it comes.
Phil

8

Hello Phil,

To me it looks like you’re pretty safe if you don’t merge citations, but your source data is stored in a place where it’s close to impossible to ever transfer it to another program, if you so wish, meaning that we have a sort of vendor lock-in, just like you have when you use other programs that deal with forms data, like Clooz and Evidentia. And they are better in so far, that it seems to be easier to access the data in those, and because they’re commercial, they also know the consequences when something goes wrong.

Technically speaking, I think that it’s quite simple to add some protection to the merge citations tool, and I think that it should be addes a.s.a.p. And there is no need to wait for 5.1.6 for that.

The other issue, and way more important to me, is that an unknown number or users may already have corrupted forms data, and to me it looks like that can’t be repaired, and if that’s true, it means that we may have a few users who don’t trust Gramps anymore.

And at the moment, they don’t have a way to start over either, because there is no way to get rid of that corrupted data.

As a user, I can fully understand that people like to use the Gramplet for data entry, and it could do a great job for standard forms too, for those who live in countries where censuses are rare. But in its current status, I wouldn’t even use it if those forms were available, because of the way it stores data.

Regards,

Enno

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.