Importing Places CSV

Help
1

Regarding Exporting Locations and Geocoordinates which is talking about an alternative place name restriction, is there any other restrictions?

I’m trying to import a places csv that I’ve completed with GPS locations* but if my CSV export and my CSV import files are the same excluded added GPS coordinates; importing do nothing. Console gives me an error each line it tries to import:

image
image1015×565 77.2 KB

Is it possible to import places data from a csv to replace missing GPS coordinates? What I’m doing wrong? I’ve deleted blank lines, is it the issue?

* completion workflow:

  • export of place display as CSV file
    image
    image1940×1247 301 KB
  • fetch missing GPS locations from google using OpenRefine
  • create a new CSV file with the same columns than exported one
    image
    image1870×746 174 KB
  • Import the new CSV file into Gramps
2

Can you change format midstream like that?

The exported latitude & longitude data was a weird text format (weird in that they contain 2 double quote delimited text strings) and the updated lines have decimal format.

(I don’t quite understand why the export is generating Lat/Long data that probably couldn’t be directly pasted into the GUI parser either. Decimal export seems like it would’ve been a much more universally digestible format.)

3

Assuming you are using a recent (5.1.x) version of Gramps, you want to set up your CSV table to look like the place lines from the CSV Export (NOT the place view export).

If formatted like the following, when you read it back in it should UPDATE the Latitude and Longitude (if the column,s row data is changed), leaving any columns row data without changes in the CSV alone.

Lieu ,Titre,Nom ,Type ,Latitude,Longitude,Code,_Partie de,Date
[P0000],Canada,Pays ,56.13,106.34,
[P0001],Quebec,Province,[P0000],

I used the French export for this example, but you can also use the English:

Place,Title,Name,Type,Latitude,Longitude,Code,Enclosed_by,Date
[P0000],Canada,Country,56.13,106.34,
[P0001],Quebec,Province,[P0000],

4

Thanks @prculley !

I’ve tried that but it don’t export all titles. Is it an option to add them (and why some of them and not all) ?

image
image1384×1185 162 KB

image
image1986×471 36.9 KB

5

I had to drop using the CSV import/export because I found that it Overwrote anything Place I had in my database with new data if there was same ID’s, Unless someone have done MAJOR changes to the import, you might end up losing connections between places and other information you already have in your database.

i.e. if you can not import places that are connected to multiple hierarchies, because only the last line in the CSV with a given ID is used, all others are overwritten.

Hopefully someone can prove me wrong, because I have nearly 800k lines in a CSV with place names and multiple hierarchies…

And the Save to csv in the list view, they are great, for getting data out in lists, to use in other types of software, analyzing, reports, etc.

6

I’m working with a test DB but l’m trying to compare gramps xml files before and after places import. I’ll post results her

7

Some checks between gramps xml files before and after changes:

City renamed into Ville (city in french),
Deleted place date (it don’t appear in the CSV file):

image
image1940×250 62.3 KB

Date ranges not properly included in CSV file: no way to make difference between from-to and between-and dates

Before and After dates are in english (bef and aft) in CSV file and not reimported properly, I’d to change them in Avant or Après, their french equivalent

Some strange changes like this (I had to check my own changes in OpenRefine):

image
image1937×165 38.5 KB

8

Approx. two years ago there was a long discussion in the user mail list on SourceForge about this…

Some of us use Openrefine to clean up data, and found that there was a lot of problems reimporting the data… a few things got fixed, but it is still a lot of problems with the csv import…
As I wrote, I stopped using it for anything other that the first import to a new Gramps database to get a set of basic objects into the database…

1 Like
9

The place title in the export CSV is the old manually entered place title, used when preferences display Place automatic title generation is NOT used. Looks like the this did not get modified in the code when we switched to automatic place titles.

It shouldn’t be necessary to use or modify the place title to get the coordinates. Except perhaps as a convenience to more accurately locate the place from your external data.

10

I’d to modify them to remove the code added in second position but it’s easy to do.

11

New try using Sqlite export, OpenRefine modification of place table then reimport of Sqlite DB into a fresh new Gramps DB. Really successful.

Process and results details in that Tweet : https://twitter.com/plegoux/status/1384435962869059584?s=20

To Gramps devs: Note that export and import of Sqlite are missing (or are not reimported) Bookmarks and Namemaps tables found in bottom of xml export file. There: two XML compared files (left: before Sqlite export, right: after Sqlite import):

image
image1935×1340 475 KB

Thanks for your help.

12

You should try sqlite export and import.

13

Yeh, I have looked at it, but there is other limitations inn it, I don’t remember what it was, but there was some references I didn’t find or something, I found that I couldn’t use it for my usage creating network graphs without some changes, but its sql is a good format for a lot of things

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.