Places in backup and xml exports

Hi,

I’m not sure of that but it seems to me that unused places (in events) aren’t exported in backup nor xml exports since some recent version of gramps (since 5.1.1-1 may be previous version, I don’t know). I’ve lost thousands of them (from 23000 in August '19 to 1800 now!) using exports and backups these past weeks/months to recreate my database from time to time.

Do you experiment this too? Are your places always here? Are backup/export secure?

Expecting to retrieve my places, I’ve found an old backup with my 23000 places in it. I want to know if I can merge places from two xml, this old one and current one with 1800? Extract everything between < places > and < places /> from the first one and put it in the 1800 one. Can I do this? What do you advise to me?

It’s scary if backup loose data.

You mention that you have used Exports. There is a setting in the Export Options >> Reference Filter >> Do not include records not linked to a selected person.

With this option, any record that is not linked with a person record will not become a part of the export. This would include place records that you have created but not yet attached to an event.

1 Like

You want to create an export with just place records and that is NOT easy.

In the family tree that you want the place records exported from, create a Person filter >> General Filters >> Everyone. In the Define Filter screen, check the box to Return values that do not match the filter rules.

Create the same type of filter for Notes.

Making the Gramps XML (family Tree) export, put this filter as the Person Filter. Set the Note filter. Make sure that the Reference Filter is set to Include all selected records.

The export should be everything with all the people (and families) and Notes already deleted.

The only way to get rid of all the other parts of the database (Sources/Citations, Media, etc ) is to manually delete those components. The goal is to get down to a database that only contains the place records

Something you could try is making all the other records Private. I discussed it here for a specific event type.

Event Handling…

Going this route, you could make all the events with the same Type before making them all private.

As I said, it is not easy to strip out everything leaving just place records.

Backups do not lose data, but settings in Export can prevent their inclusion in the export file.

1 Like

Hi Dave!

Thanks ever so much, wohhh!

Please forgive me for being totally lost - I thought your answer was going to be something like:

  1. Find this file (this is where it might be (i’m on Linux Mint btw)), open it in (appropriate program you might recommend knowing about code when I don’t);

  2. Find "Phrase (whatever you want to call it tantamount to “this is where the place data starts”)

  3. Copy it;

  4. Create new family tree in GRAMPS, let GRAMPS generate new XML file, paste copied text back in?

Without meaning this at all bluntly or intended to sound at all rude, why is it so much more complicated please?

Thanks!

Why is this so hard? Because places in Gramps often refer to other things, like other places (enclosed by), media, citations, notes etc. If you were to simple copy in the section about places in the XML, it would be missing many of these references. Importing would actually work, but you would end up with references that were created to fill in the gaps, they would be otherwise empty.

Also note that when you do a normal import of an XML into your tree, even if the import contains elements that are already in your tree, they are NOT automatically merged. You would end up with duplicates for any elements that were already there. So you would have to merge duplicates one at a time.

If you are trying to restore from an XML backup, and the trees are related (current one was derived from the backup), you might be able to use the “Import and Merge tool” addon tool. This is designed to spot differences between XML trees, and allow you to select which parts to import, or reject. I think you could select the ‘added’ places only, and they will be added to your tree. Unfortunately, you have to do this one place at a time.

One last possibility, would be to use CSV export and import. The CSV importer is designed to overwrite matching elements (matched by their Gramps ID). If you exported from a tree made from your backup, edited the CSV file to remove everything but the places, and imported that to your current tree, I think this would get what you want. It would lose citations, media, and notes attached to the places though. There might be some issues if you have added new places since deleting some of the ones in the backup, this would overwrite them if they ended up with common IDs. And it would fail miserably if you had used the “Reorder Gramps ID tool”.

But be absolutely sure to make a backup of your current tree before you try any of these. And check you data carefully to make sure it is what you want.

2 Likes

You’re right, I’ve surely must do something like that.

Today I have juggle with old and new backups and excel vlookup to find xml handles not in my current database but in old one. I’ve retrieved a lot of things, re-imported them in a merged database. Not all is correct at this time. I must proceed another excel search.

After that I must compare data between current and recovered database as I’d done a gedcom import. Only then I’ll use this new db.

Sure I’ll take care of options I use when I’ll export my db in the future.

Thanks to all who answer to me today.

The best way to handle “only” places if you dont have a lot of notes and other relations/links on them, is to use the CSV export, and store a copy of all your used/unused places in a separate Gramps Database… As long as you keep the original PlaceID’s the places will merge if you for some reason want to reimport them using CSV, and its easy to add more places in a CSV using any spreadsheet or even a text editor… VS code has add-ons for CSV tables…

CSV export/import is also a good way to wash your data with i.e. OpenRefine…

Just a tips…

Edit:
Some small typing errors!

While checking recoveries I’m still making, I’ve seen some strange things even if I’ve used the Do not include records not linked to a selected person option:

  • A citation’s transcription note attached to the citation and to an event have been preserved but citation nor its source hadn’t been exported. It’s not directly linked to the person but some kind of links is there anyway.

Thank you for this idea. I built all these places to export and share them. That idea facilitates really me to do what I want.

What you can do if you have trouble exporting “only places”, is to do a export with no filter, then after the CSV are completed, just clean up the CSV by removing the “tables sections” you dont want, Gramps will import only the Place table as long as you keep the format and the culumn names…

I recommend to deselect “Translate Headers” when Exporting so that you keep the english headers if you use any other language for Gramps…
It do work with native languages to, but it might be you get som trouble if importing to Gramps with English as language (i’m always keeping an updated version both in my native language and English (its easy to make 2 or more sheets in a spreadsheet, each with its headers in the language you like, then you just export the sheet you need…)

This information was not good for my place project that also contain multiple alternative names for the same ID…

My stupidity to not testing it… my whole project just went down the drain…

Would it be possible to do some changes to the CSV import/Export, so that if a CSV hold multiple lines with the same ID and different Names, it will add all of those names as alternative names for the first that use the ID, and if the ID already exist in the database, it just add the name in the CSV as an Alternative Name instead of overwrite, and that it add enclosed by and dates to the existing ID?

For this kind of lists, the CSV format is actually way better than the xml or json, because nearly every geo spatial data provider will supply a csv in addition to any geospatial json or GIS format…

And using OpenRefine, Excel or LibreOffice to merge or alter those lists is really simple, but its really difficult to get the data from any of those tools out to a xml or json that Gramps read…

If its a big job to change it, can you point me in the right direction, so that I can do it for my own data…

I can add the data directly to the mongodb via import, but it might be others that also makes this kind of place lists outside of Gramps… and as long as this data is in BLOBS, its not easy to add them directly to the database either.

just a question…

I have added some comments on a few of the CSV related feature requests thats in Mantis now…

Hopefully that will bring some light on some usage…

And as I write, CSV import is an advanced feature so a big Warning when using it, would be reasonable… “Do not use if you don’t know what you are doing” in red, type of warning… and a chckbox “I understand”…