Personal introduction - steers sought on programmatic interation

Help
1

I installed GRAMPS (AIO64-5.1.5-1) yesterday, and it looks as if it will be very helpful in a project on the history of my village, where I have numerous lists of people who lived there (since the 16th century) and family relationships between them. I’m used to coding in Python, and could immediately use it to create CSV files to import this data, assuming I can use the same format as the CSV files exported.

I’d like to be more flexible than this, and use Python to add or edit data within GRAMPS. I’d hope to work with XML like files, but when I see the export in this format there are attributes with automatically generated values, evidently designed to track changes, so I’m wary of messing with them.

I’d also like to be able to get data out of GRAMPS. Is there an SQLite option to do this?

Is all this possible? Where do I learn?

1 Like
2

Bienvenue !

You should try to export your database to SQLite, use this exported db in OpenRefine to integrate your csv data then create a new Gramps database using the SQLite db as input

2 Likes
3

Welcome to Gramps!

Yes, you could use CSV in the same format as the export to import data. It is also possible to edit the Gramps XML directly, but you have to be careful.

Alternatively, you could use our database API. The objects returned are described in our technical documentation. Have a look at the gramps.gen.lib module.

Our Getting started with Gramps development guide may also be useful. It contains further links.

1 Like
4

There is also an Import Text Gramplet add-on.

Once you had a sheets with recognized header for each object type, you could do all sorts of fast imports.

For instance, a hierarchy of Place objects for “Baldwin Township, Allegheny County, Pennsylvania, United States of America”:

Place,Title,Name,Type,Latitude,Longitude,Code,Enclosed_by
[P00001],United States,United States of America,Country,39.76,-98.5,,,
[P00002],"Pennsylvania, United States",Pennsylvania,State,40.8,-77.7,,[P00001],
[P00003],"Allegheny, Pennsylvania, United States",Allegheny County,County,40.46883,-79.98119,,[P00002],
[P00004],"Baldwin Township, Allegheny County, Pennsylvania, United States of America",Baldwin Township,Township,"40°22'48.54""N","80°00'51.44""W",15234,[P00001]

If you wanted to fork the Import Text Gramplet and hack the fork to recognize tab delimiters as well as comma delimiters, you could copy’n’paste directly from a spreadsheet. (With comma delimited, you have to do the “Save As” in the spreadsheet and then the “Family Trees → Import…” file shuffle with all the dialogs in & out. That makes a LOT more steps.)

Serge recently added a GitHub PR that patches the View export to allow choosing Tab delimiters. (Thanks again @SNoiraud !) Paul suggested in another thread that I patch the Import Text gramplet… but my ambition exceeds my Python skills. (I am clawing my way up from Python Novice a millimeter at a time.)

2 Likes
5

The following add-ons may be of interest:

2 Likes
6

Also, another experienced Gramps user and programmer (but new to Python and the Gramps codebase) just started getting with Gramps development. He decided to learn by helping with the Code Review of our backlog of patches and improvements. He learns Gramps and helps the community at the same time!

In the “Is there a way I can help some?” thread, our benevolent dictator by acclaim (Nick) is leading him get started, including setting up a Gramps development environment.

2 Likes
7

Assuming you’re the same Nick as in “benevolent dictator by acclaim (Nick)” [ (C) @emyoulation, so humbly begging you to pass on thanks to all responding here. I can see myself using the Import Text, SQLite Export/Import and QueryGramplets before long, but I want to get more of my data together first

2 Likes
8

Yes. Feel free to ask if you need any further help.

1 Like
9

Hi @Tim, to get data programmatically in and out of Gramps, another option might be to try out our REST API.

4 Likes
10

I would like to help with Code Reviews until I figure out how to run a development environment. I probably want to just run from a command prompt. My son who has used python says you don’t have to compile it. So with some research I’ll figure it out. In the meantime, I could do code reviews.

2 Likes
11

Quick update on this. Thanks to responses to some more specific questions, I cannow see my way to getting data out via SQLite export, which is good enough for the time being, and doing most bulk imposrts from CSV.

While I can use Python, it’s not going to be at the level of developers here. Is there a package I could install and use with iPython? This wasn’t clear to me when I explored a bit, and it’s not my highest priority. That is more just to learn how to enter and maintain data manually, especially the more complex, such as relationship. Ultimately I’d like to persuade other people to use GRAMPS, who will not want to do any coding at all

12

I’ve got the Import Text Gramplet working, and can see this is going to be very useful. To take an instance, I imported people who lived in the village according to the 1841 census, including a note of this fact. However, I’d prefer to have the 1841 census as a citation, and have this against all these people. How could I do this? I had thought to export everything to SQLite and run some SQL there, but I’m not sure I’m up to that - how are all those 25 character PK ‘handles’ created?

13

You can (and should) omit the handle on new objects created via the import. Only worry about those for doing updates to objects via Import Text. When omitted, the handles are automatically generated.

Note that the CSV does NOT support all field names. If there’s one needed, it is very straightforward to hack the CSV import library to support and additional attribute.

1 Like
14

Easy for me to hack, or you? I can have a look, but doing it a first time will be a bit scary.

Tim

1 Like
15

I didn’t say “easy”, just straight forward. One 1 module to hack and all the includes should already be there.

It just requires finding the call to write the missing piece and the most appropriate header name.

Then duplicate a similar section and adapt it.

16

If you’re more comfortable coding SQL than Python, it might be worth a try.

The handles that Gramps creates are merely a combination of a timestamp and a random number, which you can achieve with SQL functions (see discussion here).

But the handles you create need not be cryptic, just unique. If you had a sufficiently long list of unique items (e.g. words from a dictionary, used singly or in random combinations, somewhat like what3words.com does for locations), I imagine those would serve just as well, if you took care to use each word or combination only once.

Import your results to a new Gramps tree and run the various tools and utilities for verifying the data.

1 Like
17

You might not need to adapt either. If you look at the People CSV block, it already includes an option to do a source title for the following Events: Birth, Baptism, Death, Burial, & Occupation. And also for the Person themselves or a defined attribute.

18

Just remember that with CSV import, you cannot import a place multiple times with different “Embedded by” values, the place is overwritten by the last instance imported, unless someone has made a change the last year.
So, you can’t build a place database in Excel, LibreOffice etc. and and get it imported with multiple “embedded by” places.

1 Like
19

It looks as if the ImportText gramplet can accept XML, so maybe that lets me import what I want without any code tweaking.

I’ll test in the next few days, but if I can use this sort of thing as a model

image
image1096×442 19.3 KB

I think I’ll be OK. I’ll not include handle or change attributes, and id attributes only when I’m adding data to a person or other entity which I’ll not want to have to merge later. For links to citations, events, sources, places and families, I’m going to need to store their handles, but that’s easy enough.

If manual XML works, then I’ll be able to write code to generate it from archives some volunteers are digitising for me. Obviously there is a manual step, for me initially, to look up those id attributes, and so avoid having to merge later. I’ll probably work one ‘citation’ at a time, since this will correspond to how volunteers digitise records.

Snags? If this isn’t going to work, let me know. Better ways of doing it?

Tim

20

Sure, writing Gramps XML is an option. I use it occasionally to clone an object Rather than learning to write well-formed Gramps XML, you can filter to export 1 person (with all the necessary Sources & Places) to an uncompressed Gramps file. (Drag the Person to the clipboard and use the right-click context menu to create a custom filter.)

Then you can tweak that XML for that person to create clone profiles.

But… if you’re going the direction of actually WRITING the encoded data from source data, you might find writing GEDCOM a easier approach. There are are lots of tools that write GEDCOM already. Include those that convert excel sheets to GEDCOM

1 Like