Corruption when opening FORMS data in PEOPLE screen

Help
, ,
21

When I start with a person, for whom I found a changed citation reference, the Forms Gramplet quickly shows that things are wrong, because it shows census forms that don’t have any reference to the person at all. And that’s an error, because when you fill in a form, every person listed there is either a new person in the database, or an existing one, for whom an event will be created, with a citation that links back to the form. In other words, when the form is created, the circle is closed, always.

This means, that it’s also possible to check for corruptions, because that is a situation where no row in the form points back to the person.

But now the big question is: Is there a way to repair things? Can the forms data, which seems to be all intact, be used to reconnect the form to the proper person(s)?

My first instinct says that this should be possible, because the event references are all there, and they not only have all the row data, but they also still point to the right person.

Can you elaborate on this @Nick-Hall , or can the Gramplet creator jump in?

22

The event references are actually contained within the person objects. The Form addon just allows you to edit attributes in the event and event references quicker than using the standard editors.

The form editor finds it definition file from the citation attached to the event. The source should contain an attribute with the key “Form” and the value containing a valid form identifier.

If you remove the link to the form definition, then your data is still safe. The event will no longer be recognised as a from.

If you add a link to an event created without using the form editor, you can still get the form editor to recognise it. There is nothing special about the data in a form.

Don’t merge citations. You won’t lose any data doing this, but you may well prevent the form editor from accessing the event correctly.

23

@SteveY that’s just temporary, right? I mean, the Gramplet’s not updated, but nothing gets changed in the database. You just don’t see the right data.

24

Thanks for confirming that. It is exactly what I see in the new database, where citations that link to 1 person or event in the old one, link to dozens in the new one, and many forms show census data not associated with the persons they were once attached to.

25

That’s correct. It’s a display glitch only. The underlying data remains fully valid and correct.

Thanks
Steve

26

@Martnal I did a quick check with the April version of your database, and when I did a merge citations in that, using the options that I use for that in my own database, which has no forms at all, I got the exact same mess for person I0000 as I found in your October tree, meaning that 4 out of 5 forms had been replaced by other forms that don’t mention this person anymore. This suggests that your mishap may have been caused by such a merge.

@Nick-Hall assuming that this is the case, and knowing that you warned against merging citations, is there any way to repair this, other than restoring from back-up? And does it make sense to file a feature request so that merging citations either ignores citations with forms, or puts up a large warning against merging for users with forms?

27

Nick, I’ve been busy looking at various backups. Could you expand on the following, please:

"The form editor finds it definition file from the citation attached to the event. The source should contain an attribute with the key “Form” and the value containing a valid form identifier.

If you remove the link to the form definition, then your data is still safe. The event will no longer be recognised as a from.

If you add a link to an event created without using the form editor, you can still get the form editor to recognise it. There is nothing special about the data in a form.

28

Here’s an illustrated overview of how Form data is stored by Gramps. This is based on my own understanding. Whilst I’ve made some minor edits to the underlying Forms code, I don’t claim to have the same depth of knowledge as @Nick-Hall. Hopefully he can correct any mistakes I have made.

The starting point is the Source. The Source record for a Form has an attribute called “Form”. The value of the attribute is the ID of the Form to use for that source.

image
image735×354 42.2 KB

A citation has a reference to a Source.
image

Finally an Event has a reference to a Citation
image

So for a given Event, we can check the Citation (if any) and then the Source (if any) to determine the ID of the Form to be used for display.

So we now know which form to use, next, we need the data to populate the form with. This data is stored in two locations

  1. Person specific \ Family specific data (“Details”) is stored in the event reference Attributes
    This person is the 2nd person on a census form
    image
    Family specific data is stored in the

  2. Form specific data (“Headings”) is stored in the Event attributes
    image

Additionally,

  • The Form reference is stored in the Citation Volume/Page field
    image

  • The Form Date and Place are stored in the Event Date and Place fields
    image

  • The Form Gallery is stored in the Citation Gallery
    image

Here’s what the above data looks like in the Form

image
image1260×369 32.2 KB

So from the above you can see that only the Form Reference and Gallery data is stored in the Citation itself. Given you merged citations, can it be assumed that these citations also have a common value for the Reference and Gallery fields? If so, I agree with Nick’s earlier comment; your Form data is still in your database.

How to fix?
I’ve not tested this but I would try the following

  1. Make sure you have a backup before trying this
  2. Open one of your events where the Form is not displaying correctly and switch to the Source Citations tab
    image
  3. Make a note of the “Source” - it’s in the Title column in the previous image
  4. Delete this citation (click the ‘-’ button)
  5. Add a new citation (click the ‘+’ button) and choose the Source that you noted in step #3
  6. Click OK to save the new citation
  7. Click OK to save the event

This event now has a new (unique) citation. Hopefully the Form for this event now displays correctly.

@ennoborg if the above steps work, then I think an automated fix is to identify all events sharing the same citation, where the citation has a reference to a Source and that Source has the Form attribute. For each event, replace this shared citation with a new citation having the same Source value.

I hope this helps move you a step closer to recovering your data
Steve

1 Like
29

Steve, that looks very helpful. Bedtime here, but I will be on it in the morning.
Martin

30

Hello Steve,

It looks like you’re right. I checked the forms data in the event references for one person, and they were still there, which was to be expected, because only the citations were merged, not the event.

For 4 out of 5, the forms Gramplet showed the wrong data, and when I edited these events, removed the citations, and replaced them with new ones, attached to the same census, the forms Gramplet showed the right data again.

This suggests that an automated repair is possible, with the right tools.

And knowing that these citations can easily be recognizef by the Form attribute in the source, the whole thing is quite easy to prevent too.

Things might not have happened if each citation had a unique vol./page field, but those are quite likely to be not unique enough for this purpose, because if you enter the name of the census page there, including street address, district, etc., it will still be the same for all persons in a particular family, so it will still be seen as mergable, right?

thanks for investigating,

Enno

31

A form citation will be shared by the event and all event and event reference attributes.

32

OK, good to know. How does that affect its reference count?

When I browse the citations, in the source tree view, the merged ones have references to multiple persons and events, and what you wrote suggest that a proper forms citation is only referenced by a single event.

And that suggests that the problem citations and events can be easily found by using a filter that selects those that don’t have a 1-to-1 relationship.

33

Firstly more thanks to Steve, Nick & Enno. I am reading and rereading these recent posts. I am being very careful, and I have lots of backups.

It’s ironic, I’m long retired now, but for much of my career I had to understand the intricate workings of niche-market software for which I had no data. Now I have lots of valuable data, but I lack the intricacy knowledge of the software.

Finally, a question for Enno, are you meaning by your line “And that suggests that the problem citations and events can be easily found by using a filter that selects those that don’t have a 1-to-1 relationship.” that I should be able to easily identify the corrupt records? Should I be creating the Filter in Events or in Citations, or doesn’t it matter? I am getting well out of my depth here.

Martin

34

Yes, that’s indeed what I mean, because Nick wrote that when you create a new form, an event and a citation are created, and as a developer I interpret that as 1 event and 1 citation, so I feel safe to assume a 1-to-1 relationship.

This is an assumption, because I never created forms in my own tree, so I need to rely on what Nick wrote. And that is supported by his advice to avoid merging, and his comment that when a citation is linked to more than one census event, the form editor picks the 1st, which is a fair choice when the normal situation is that a citation is always linked to a single census event.

Under these circumstances, one can create a custom filter that shows citations with more than 1 reference, and I tried that already, just to be able to find citations that may be connected to more than one person and/or event. For me that is way easier than creating one for events, but a specialist may be able to write that too. I rely on the one that I made for citations, and that one already shows a few that look quite suspicious, like one that is also referenced by a birth event, with a date that is not available in any form.

35

On closer inspection, it seems that a filter by reference count is a bit crude, because in your file, it shows many citations that are linked to one census event and a couple of other events, like births, that you may have linked to census citations because the census gave you an age, from which you could calculate an approximate birth year. And when I check some of the associated persons, their forms are OK.

This means that what you really need is a filter that selects all census events that share a citation, because they are the ones where the forms Gramplet can get confused, just like Nick wrote.

And that’s something that I can easily write in SQL, where it’s a join on two instances of the events table, where the event IDs are different and the associated citation ID is the same, but I can’t write that with the regular filtering tools.

It may be possible with Isotammi Supertools, but if you want that, you need another expert than me.

36

Yes, SuperTool can be used to build a new Custom Filters where there is no appropriate Rule. That might be a good way to go for a search you might have to re-use.

However, that is a section of the SuperTool user guide that needs an expansion and an example.

37

Well, when I look at it, I see no easy way to build such a rule, because all rules that we have examples of are based on the comparison of data with a fixed value. And that is not available here.

38

The Source will all have a Form attribute. It is needed to utilize the gramplet. The type of Form is then defined by the values. ie “US1950”.

Not sure if this helps but it would be a starting point to find the errors. I avoid the Form gramplet and have it hidden on my system.

39

I know what you mean, but the form is not the real problem. It works well, if the forms Gramplet can find a unique census event that is linked to the citation that was created with it. And that uniqueness is destroyed when citations are merged.

This means, that we need a filter that finds all census events that share a citation with another census event. And that citation is not defined by an ID, but by the simple fact that it is referenced by two or more census events. And the only thing that we have now is a filter that can find citations by reference count, and that is not enough.

It is somewhat similar to a filter to find persons that have the same surname, because in that case too, there is no specific surname to search for. And all filters that we currently have rely on the comparison of objects or attributes with a specific value.

In SQL, a search for people with the same surname is very simple:

select a.name, b.name
from t as a, t as b
where a.surname = b.surname and a.id < b.id

And what we need is the same, with a more complex where clause.

40

It would probably be easier if I modified the editor to edit an event rather than a citation. I don’t see a problem in relaxing the unique citation requirement.