Convenient and Efficient Exports/Updates from Gramps to RootsMagic

Ideas
1

I want to share my idea of how to conveniently and reliably perform periodic exports from Gramps to RootsMagic. I apologize in advance if this is already common knowledge, but I haven’t found a clear solution for this process. I don’t think I’ve discovered any groundbreaking technique, but unfortunately, I haven’t come across a detailed description of how other researchers handle this process. Therefore, I’m offering my own perspective on it. My proposal might contain some mistakes, so I welcome any constructive criticism.

The Task

Since Gramps is my primary software for genealogical work, I often need to export data to online services like FamilySearch. RootsMagic provides useful tools for working with FamilySearch, so I decided to use it for this purpose. However, I believe this approach could also work similarly for importing data into other programs.

The Problem

The initial export from Gramps to RootsMagic is usually straightforward. However, subsequent exports become more complex because data changes in both programs, and RootsMagic may not always accurately match individuals across different versions of the database. This complicates and slows down the update process.

The Solution

To ensure accurate matching of individuals during future export-import cycles, I propose the following approach:

1. Generating _UID before export:
  • Before each export from Gramps to RootsMagic, create a 32-character UUID using the SuperTool script and add it to the _UID attribute for each individual.
  • This ensures the uniqueness of each person in the GEDCOM file.
2. Importing into RootsMagic:
  • Import the GEDCOM from Gramps into a new database in RootsMagic.
  • Then, use the File → Tools → Compare tool to compare the two databases.
  • The comparison will be based on _UID, ensuring accurate matching, even with significant changes in the records.
3. Manual Merging:
  • After the comparison, manual confirmation of changes for each individual will be required.
  • I haven’t found a way to perform batch merging, so this process will take some time. However, the procedure is simplified by the fact that matching is reliable, and you only need to confirm changes.
4. Exporting to FamilySearch:
  • After merging the new and updated individuals in RootsMagic, you can export them to FamilySearch one by one. This also involves non-intellectual work, but it probably cannot be fully automated.
  • Afterward, the new _FSFTID identifiers can be manually added to Gramps. However, this step can also be automated.
5. Automating the process of creating _FSFTID in Gramps:
  • To automate the transfer of new _FSFTID into Gramps, you can create a script that:
    • Compares two GEDCOM files (one from RootsMagic and one from Gramps) and finds new _FSFTID identifiers that are missing from one of the files. Similarly, unnecessary ones can be removed.
    • Creates an array with pairs of gramps_id - _FSFTID.
    • Using another SuperTool script, adds these new _FSFTID to Gramps.

Benefits and Summary

This approach reduces the amount of manual work and ensures accurate matching of individuals between Gramps and RootsMagic during periodic updates. The bulk of the work will be automated using two SuperTool scripts and a separate script for comparing GEDCOM files. Manual work will become less intellectually demanding due to the shared _UID between the two desktop applications.

2

I also tested comparing with/without _UID. Everything looks as described above.

And also, @ennoborg thank you for your supporting everywhere ))

1 Like
3

So you admit that this thread is yours, right?

I had some thoughts about it already. :slight_smile:

1 Like
4

From the Gramps wiki

Mentioned on Feature Requests 0009691: Create UUIDs unstead of hashes for handles

5

In my current process, I treat the other databases as throwaway items, meaning that I don’t see a need to merge in RootsMagic, or Ancestral Quest for that matter. I like the latter, because it’s faster for larger imports from FamilySearch, and it never shows the kind of errors that RootsMagic has. And when I want, there is an ever faster alternative in Legacy 10, which is completely free, and works great to download new branches from FamilySearch, where I know where to start, and don’t have to import a GEDCOM from Gramps. Legacy’s GEDCOM import is quite slow, so I prefer to use it for new data only. Legacy 10 can import 20 generations in less time than RootsMagic needs for 10, and works great for experiments. It looks awful though, as if it was designed for Windows 3.

My works is made easier by the fact, that I did not want to wait for the UID discussion, and created my own tool for that, in pure Python. An earlier version of that had some traces of AI generated code, but that was so bad that I removed it, so now it’s all mine, and it uses the official _UID checksum as defined by the LDS, who wrote a PDF with an example program in C. These UIDs are stored in attributes, just like the UIDs that I got when I migrated from PAF to Gramps, when I expanded the GEDCOM importer to import those. And that imports FSFTIDs too. Both import hacks are in the official repo since long, so in my database (and yours) all new persons imported via Ancesttal Quest, Legacy, and RootsMagic already have a UID, and an FSFTID when downloaded from FamilySearch.

With this, there is one problem left, in my process, and that’s how to figure out which persons added or changed in either program are worth importing back into Gramps. That is a manual process now, based on notes, for which I first import the tree from AQ, Legacy, or RM into an empty tree in Gramps, for quality control, and then export a selection to be merged into my main tree.

This last process would be much nicer with an import and merge for GEDCOM files, based on the UID. That should be quite easy when you make sure that all persons have one before export to that other program, and it can do all sorts of interesting things, like add FSFTIDs for all persons matched with FS, and show which persons are new, and how they relate to existing ones. It can also show new data to be merged, just like the import and merge for Gramps XML already does.

There are a few pitfalls, especially when people are merged:

  1. In Gramps, this adds both UIDs to the new person, and most other programs don’t understand that, and throw away one UID, at random,
  2. On FamilySearch, the new person gets one FSFTID, but the site knows the old one, and that sometimes means that the other program must replace the one it knew with the new one. I’ve seen that on AQ and RM, where it can be a bit clumsy to deal with, and it means that the FSFTID may need to be replaced on import,
  3. Sometimes, merges on FamilySearch result in a totally different person being linked to an FSFTID, And that can be so bad, that I just tell RM to unlink the current one.

That’s all for tonight.

1 Like
6

Maybe anybody needs to generate UUID attributes for Rootsmagic or any other genealogic application to sync data - here is Supertool script:

[Gramps SuperTool script file]
version=1

[title]
Add _UID Attribute to Persons

[description]
This script checks if each person in Gramps has an attribute named "_UID" for RootsMagic. If not, it generates a 32-character unique identifier and adds it as the "_UID" attribute.

[category]
People

[initial_statements]
import uuid

[statements]
# Retrieve the list of attributes associated with the person
attribute_list = obj.get_attribute_list()

# Flag to check if _UID attribute exists
uid_exists = False

# Iterate through each attribute in the list
for attr in attribute_list:
    key = attr.get_type()

    # Check if the _UID attribute already exists
    if key == "_UID":
        uid_exists = True
        break

# If _UID attribute is not found, generate and add a new one
if not uid_exists:
    # Generate a new 32-character unique identifier
    new_uid = uuid.uuid4().hex

    # Create a new attribute for _UID
    new_attr = Attribute()
    new_attr.set_type("_UID")
    new_attr.set_value(new_uid)

    # Add the new _UID attribute to the person
    obj.add_attribute(new_attr)

    print(f"Added _UID: {new_uid} to ID: {gramps_id}")

[filter]

[expressions]

[scope]
all

[unwind_lists]
False

[commit_changes]
True

[summary_only]
False
7

Looks like this is impossible to automate this part because Rootsmagic changes INDI (for all people) and removes _UID for updated people. At least in gedcom export. So, I can not compare two gedcom files.

8

You can, if you use the right _UID, and this one’s wrong. Mine was wrong too, because I allowed ChatGPT to write the checksum code, and it took years before a fellow user told me that they were changed. I hadn’t noticed that, because most of my UIDs were imported from PAF, or AQ, or RM, and none of these programs tell you anything when an imported UID is wrong. You only notice that when you see that the comparison fails.

If you add this checksum, your UID will be OK:

1 Like
9

I have done a little testing to add to the conversation. It would be nice to have an update/merge feature that works.
I have Ancestry, Find My Past and Family Search accounts.
I assume the Family Search ID does not change because it is a global tree. Enno knows more about the imports from Root Magic and FS than me.
On Ancestry the only thing linking it to the person is the ID number which can’t be viewed via the profile browser, however gramps uses this as the gramps record ID. This would stay the same as long as you continued using the same tree that you created.
FMP does have a UID in the gedcom file and it stays the same in each gedcom creation. Gramps does create it as an Attribute.
Enno created a program to check and add a UID if missing and that works on the FMP records. Therefore presently you could keep FMP and gramps in sync.
I guess you could export your gramps DB to FMP and then download it back to gramps. Then you would have the two in sync.
You could then export it to Ancestry and then do an import which would give you a record ID. Then use Root Magic to find the matches on FS.
All of the above steps done without making any changes to the databases between steps.

1 Like
10

I just did a test with Ancestry, and found that in a direct export from the site, there are UID lines, not _UID, that have the same value as I have in the _UID attribute, except that the 4 character checksum is missing. And when I download that tree from Ancestry with RootsMagic, and export a GEDCOM from that, I see _UID lines, with checksums, which are totally different from the ones in both the Gramps and Ancestry checksums. What I mean is that they don’t match for the part without the checksum, which is 32 characters.

Ancestry downloads with RootsMagic have another ID too, called AMTID, also exported with an underscore, as it should, which looks a little different, for which I don’t know how persistent it is.

A quick test with FMP shows that they export UIDs as _UID with 32 characters in the official format of the UUID with 5 groups of characters divided by dashes. Without those, the value is the same as exported by Gramps, but again without the checksum, which is right, because the checksum is an LDS construct.

This suggest that these UIDs can be used for matching, because they are persistent. And I know by experience that they are persistent between Gramps and RootsMagic too, and also between Gramps and other programs like Ancestral Quest and Legacy.

The IDs that we see as a user are not stable, and can never be trusted. All programs that I know change them when you import the same person twice, which is something that they have to do, for a couple of reasons, one of which is that the IDs must be unique at the GEDCOM level. This is also true for the handles inside Gramps, which look like UUIDs, but must also be unique, and get a new value when you import the same object again. This means that in most software what we call UID is the only thing that is really persistent.

FamilySearch, Geni, and WikiTree all have their own IDs which are somewhat persistent. And I write somewhat, because when persons are merged, one of these sort of disappears. And I write sort of, because on FamilySearch and WikiTree, they can still be used in a URL, with the server redirecting us to the merged person. I have no idea how Geni does that though.

The original UID with checksum was introduced by the LDS in 2000, and the first time I saw it was in 2004, when I migrated my late father’s work from Brother’s Keeper to PAF.

1 Like