Yes, I like that, especially because most of the data is in the event and event reference attributes, and not in the citation. The latter can have some string for the census district, page, line, etc. in the vol./page field, but that field is often empty too, or the same for multiple citations.
And if you need a volunteer for testing, count me in. I have half a dozen versions of Martin’s database to see what works, and what not.
Nick, I today took a copy of my most recent known good .gramps file. I
checked the integrity of a record which I knew had previously been
corrupted last month. (It was correct in my most recent known good
gramps file, of course). This was a Person record with several Census
events, created using the Forms gramplet.
I ran Tools - Family Tree Processing - Merge Citations, accepting
‘Ignore Date and Confidence’ for the ‘Match Threshold’ option and I left
‘Don’t merge if citation has notes’ checked.
“787 citations merged.”
I then examined the Person record mentioned above. It was corrupted in
exactly the same way as before.
Martin
@Martnal when I look at the code, and the comments made by Nick and Steve, it is obvious that we have a bug in the forms Gramplet, and no real corruption of data. It is all there in the event and event reference attributes, and the only real problem we have is that the forms Gramplet can’t find it after citations are merged, because it relies on the assumption that every census event has a unique citation. And that assumption is only right when you never use the merging tool.
A quick search in the forms code shows, that when it builds the list of forms associated with a person, it knows the events, and a quick glance at your tree shows that the list that you see doesn’t change after merging. The only thing that often changes is the form shown when you try to edit a form, because at that time, the Gramplet just opens the first form it can find for that citation, and after merging, that is most often the wrong one.
Nick and Steve know about this, and because the Gramplet ‘knows’ the events when it builds the form list for the active person, it can also be adapted to use that to open the right form, when you click edit. The only thing that we don’t have yet, I think, is a bug report and some sample data to test with.
Would you be able to make an extract of your tree that is big enough for the merge citation tool to find duplicates and cause some ‘corruption’, and attach that to a bug report?
Enno
I would argue the opposite there is nothing wrong with forms for example
if you take my birth certificate it has one unique citation number which
then appears 4 times
Which is what I would do manually if I was not using forms for this purpose.
So the assumption of one citation per census is correct the complicating
factor is then, that this same citation will correctly appear many times
in the database with different people
So the question is why would would you want to merge citations? When by
nature of the beast any one citation gets used multiple times.
The only merging I can think of is if I enter a birth cert (say) twice
and create 2 different citations. Which is why I have to differentiate
between a birth cert (at birth) and a later re-registration (adoption)
Phil
Enno
An issue for me which I have tended to ignore but will now consider is
I use the format “PRO Ref RG09 Piece 2878 Folio 139 Page 42”
for my census citation on the older census where there is more than one
family per page this means it is possible I have 2 families with an
identical citation which is again exactly what I would have done manually.
Phil
@ennoborg,
I agree with Phil - the Form Event Citation can be referenced multiple times; this is not an error. I do the same as Phil, adding for example a Form census Event citation to each household members name and their birth, occupation, residence, … Events.
In my earlier description, I missed out that the event’s event type field must match the type defined in the form definition for it to be correctly detected as a form.
The one thing you must not do is add the citation to another event of the same Event type. That confuses the forms gramplet. For census events, I’ve not found this a problem but for other event types it can be. As an example I was just transcribing a persons second marriage certificate as a form. The bride was described as a widow which is evidence of there being an earlier marriage. I cannot add the citation for the 2nd marriage to the 1st marriage event without “breaking” the Forms gramplet.
I still think my suggested approach of “unmerging” the citations is the right way to fix this data. In pseudo code:
For each citation, c, that has a source that has a valid Form attribute
Get the event type, t, of the form
Get all events, e, of type t that reference citation c
For events 2…n in e
create a new citation c’ that is a copy of c
replace the event link to citation c with a link to c’
If the form event citation is referenced in more than just events, as I do, the fix gets a lot more complicated. You would need to start checking all data for each person on the form, replacing citation c with c’.
Steve
I add the schedule number to avoid this problem e.g.
Department: RG, Series: 13, Piece: 3856, Folio: 115, Page: 9, Schedule: 47
It also helps find the correct household again in future. Especially useful given the atrocious handwriting and image quality of some pages!
To reduce typing I modified the Form gramplet to include this boiler plate text in a new Form. As an example, this is what I see when I create a new 1901 England census form
This change is pending review for inclusion in the Forms gramplet. @Nick-Hall can you suggest someone that can review this PR? Form references[v52] by stevenyoungs · Pull Request #265 · gramps-project/addons-source · GitHub
Steve
Steve
Whilst I could do that for the very small number of times it happens and
given that the Household Schedule Number appears as an item under
Headings. Then I am not sure there is any great benefit. It can go on my
ToDo List.
Incidentally I use custom.xml not the standard Forms.xml. I chose not to
follow the boilerplate you suggest but keep my Citation Blanks in a
Spreadsheet where I can adjust them easier if required and then simply
copy and paste the one I want. Saves having to update the .xml
Thanks for the comments
Phil
Hello Steve,
I can see what you mean, but I have also looked at the code, and I know that Nick was right when he wrote that there is no need for the citation restriction. And that is, because the (census) event is known by the Gramplet, and through the event, it is also possible to find the proper form, because that is stored in the attached source, which can still be found when citations have been merged. And that is, because citations can only be merged, when they are linked to the same source.
Another thing is, that even when you think that citations should have unique dates and volume/page entries, that does not mean that Martin and I have no right to keep some of these empty, or use the same volume/page for different families. One can then argue that one should add a line number to make sure that the volume/page field is unique, but that is no more than a personal choice, and not a restriction in Gramps, at least not one that can’t be repaired.
I hope that Martin can provide a small file that can be attached to a bug report, and may write that report myself, later this week.
Regards,
Enno
Responding in reverse order:
I don’t in fact think that all citations should have unique combination of (source, volume/page, date) so apologies if my earlier reply was not clear on that. I have examples in my own tree which are not unique. I’ve just been fortunate enough not to merge citations.
I agree. The event object can be passed in to the EditForm class. This removes the need to call find_form_event. As you know, it is this function that, given a citation attempts to find the correct event for the form. Where there are multiple form events sharing a citation, it returns the first one it finds. I’m assuming that is what Nick was suggesting?
Steve
And I’m assuming the same thing, after reading the code.
And just so you know: My own citations are quite a mess, because some still originate in Brother’s Keeper, where you had to put all information in the source title, or in notes.
Enno
This PR removes the find_form_event function in the way we have discussed
Improve the robustness of finding form events by stevenyoungs · Pull Request #582 · gramps-project/addons-source (github.com)
I’ve only given it a very basic test so make sure you have a backup.
I’m very interested to know if this improves the behaviour on Martin’s tree
Steve
Hello Steve,
This is nice, thanks. I was so obsessed with the discussion, and some other issues, that I couldn’t find the peace of mind to hack my own code, so I’m glad that you did.
And my first checks suggest that they work, meaning that I see no difference between the before and after backups that Martin sent me. Every form that I checked shows the right persons here now.
Regards,
Enno
Hi Enno,
That’s great. Many thanks for taking the time to test the change.
We’re hopefully another step closer towards “recovering” Martin’s data 
Steve
Hi Steve,
Yes, I hope so. It looks more like uncovering, now that we know that there was a problem with finding the data after the merge, and it was all still there.
There may be more users with the same issue. who don’t see it, because they never look at old forms. And I assume that we don’t have reports that show forms data. Is that right?
I haven’t contacted Martin yet, and I’m not sure whether it is a good idea to put some instructtons on-line to let people download your fix, and install it in Linux or Windows themselves, for testing. It will be installed anyway, when your PR is approved.
Regards,
Enno
The form editor is a transcription tool. Once the form is transcribed there isn’t really much need to use it again.
I also expect that most people don’t merge citations. They should usually be unique for a form anyway. The exception will be two families on the same page in a census and where the citation doesn’t include a line number.
There is a census report which displays the census data in a similar format to the layout in the census return. It only works well for UK census data which has a relatively small number of columns.
I mainly use gramplets to compare certain attributes for a person over time. This is useful to check that birth details and perhaps occupation and residence are consistent.
Users shouldn’t encounter problems with these even if citations have been merged.