Gramps workflow (roughing in a tree)

[geophile]

Gramps 5.1.6 on Ubuntu 20.04 (hosted by Digital Ocean).

I have Gramps Web running, and I’ve been able to enter some data. I’m learning my way around, and starting to plan my workflow.

I have a Word document with a couple hundred people (name, birthdate, marriages, places) that I would like to enter, and I’m trying to figure out my workflow. From playing around, I can tell that I really don’t want to enter a couple hundred people and families using the web UI. I am very comfortable working with text data (CSV or other formats). I have an extensive background in programming, databases and data modeling, so I am leaning to creating the bulk of data manually, e.g. as a CSV file and importing it. The structuring using identifiers like [I0001] and [F00000] is pretty obvious.

Questions about this approach:

• General advice? Is this sane?

• Is there any reason to prefer .ged to .csv? The .csv format looks simpler to me.

• I’ve only entered information on four people so far. Before importing the entire data set (as .csv or .ged) is there some way to wipe data and start clean? In the worst case, I will just delete the four people (and one family) I’ve created manually. Alternatively, is it okay to import data into a non-empty database?

[emyoulation]

CSV is a good ‘roughing in’ approach using the Desktop version. Use the Gramps Web Sync to start with a Desktop and Web database in sync.

Then, on Gramps Desktop, begin experimenting with the CSV import. (Make a backup before each import.)

I like using the Gramps 5.2 beta because @SNoiraud added the option to select the delimiter. (Changing delimiters is a bit of an Easter Egg – the GUI is hidden in the Edit → Configure… of views with tabular data. ) This means that you do not have to Save a spread sheet to a CSV file, then import the CSV file. With a ‘tab’ delimiter instead of ‘comma’ delimiter, you can simply copy the spreadsheet Cells to the OS clipboard and paste them into the Import Text gramplet.

Note, @dsblank recently posted a patch to the CSV importer plug-in to support Tagging and eliminate unintentional Given Name overwrites.

[emyoulation]

Take a look at:

Tab Separated Value parser to supplement CSV?

I was experimenting with transferring my Place hierarchy using the CSV view export & the Import Text Gramplet. But one common step is to clean up the data a bit in Excel. I was hoping to be able to just copy filtered chunks of spreadsheet & have it be parsed in a variant of the Import Text Gramplet … without messing with double-quoting Place Titles & names. And without applying an Excel formula to concatenate each row with comma delimiters or having to write the rows to a subsequently importab…

[emyoulation]

@geophile

I’ve only entered information on four people so far. Before importing the entire data set (as .csv or .ged) is there some way to wipe data and start clean? In the worst case, I will just delete the four people (and one family) I’ve created manually. Alternatively, is it okay to import data into a non-empty database?

You can create a new tree for experimenting or testing imports. The easiest way to start clean is to delete the tree and create a new one. (Or to be safe, start a new tree first and only delete the old tree when you’re confident that will not be a mistake. Rename the old tree so you do not confuse it with what is current.)

[geophile]

Thank you for those ideas.

Pardon a newbie question: What do you mean by “Gramps Desktop”? Is that Gramps running locally in Docker?

Fixing data in Excel sounds like a nightmare. I am proficient in Python, and have homegrown tools for operating on CSV and other forms of data. These have proven very effective in a variety of projects.

So what I’d like to do is to treat my CSV files as my primary source that I operate on while cleaning the data. Then periodically, I would wipe the Gramps database and import all the CSVs.

• How can I wipe the Gramps database? I would think that there are Postgres tables like Place, Person, Family, etc., and simply writing a psql script to truncate those tables (leaving admin tables in place) would do the trick.

• Or perhaps wipe through the API? That seems to be oriented exclusively to working on one object (Person, Family, etc.) at a time. So wiping that way seems like it would be quite slow.

(Just saw your last response)

Wiping the tree sounds like it does what I need. How do I go about doing that? I looked around the web UI and didn’t see anything that looks appropriate. Is a Tree the same thing as a Family Tree? If my CSVs are my primary source of data, deleting a tree is a low-risk operation.

Any pointers to documentation on these basic concepts would be appreciated. I haven’t been able to find such documentation so far.

[emyoulation]

@geophile

What do you mean by “Gramps Desktop”?

Gramps Web is a young (July 2021) project created for collaborative work on web servers. Since 2011, Gramps been a single-user database python application which runs on Linux, macOS and Windows desktop/laptop computers. Having to distinguish between the local desktop-based and hosted/online server-based variants of Gramps is a new problem.

This question was submitted in the general “Help” category, not the “Gramps Web” category. So I assumed that you wanted to take advantage of the more mature and feature-complete Desktop version. Then use Gramps Web Sync to push the data up to the Gramps Web instance being hosted on Digital Ocean.

I only mentioned a spreadsheet be cause it is an easy way of manipulating CSV data without hassling with formatting problems: strings that include commas, dates. And to take advantage of tabbing between cells, fill-down, pasting blocks of cells, etc.

This data (4 people: the home person of the example.gramps tree, his spouse and parents) was exported to CSV and imported (as Text fields) into LibreOffice calc. Then pasted here into Discourse. But it would be just as easy to have pasted it into the Text Import gramplet with TSV delimiting.

Place	Title	Name	Type	Latitude	Longitude	Code	Enclosed_by	Date
[P0957]	USA	USA	Country
[P0000]	OH, USA	OH	State				[P0957]
[P0007]	CA, USA	CA	State				[P0957]
[P0022]	WA, USA	WA	State				[P0957]
[P0029]	TN, USA	TN	State				[P0957]
[P0032]	SC, USA	SC	State				[P0957]
[P0036]	AR, USA	AR	State				[P0957]
[P0039]	IN, USA	IN	State				[P0957]
[P0041]	MS, USA	MS	State				[P0957]
[P0042]	NM, USA	NM	State				[P0957]
[P0059]	SD, USA	SD	State				[P0957]
[P0062]	LA, USA	LA	State				[P0957]
[P0066]	MT, USA	MT	State				[P0957]
[P0070]	ID, USA	ID	State				[P0957]
[P0239]	Humboldt, CA, USA	Humboldt	County				[P0007]
[P0266]	Greene, AR, USA	Greene	County				[P0036]
[P0316]	LaPorte, IN, USA	LaPorte	County				[P0039]
[P0409]	Twin Falls, ID, USA	Twin Falls	County				[P0070]
[P0433]	Washington, LA, USA	Washington	Parish				[P0062]
[P1131]	Aberdeen, WA	Aberdeen	City				[P0022]
[P1132]	Bogalusa, LA	Bogalusa	City	30.7910204	-89.8486858		[P0433]
[P1388]	Portsmouth, OH	Portsmouth	City				[P0000]
[P1398]	Albuquerque, NM	Albuquerque	City				[P0042]
[P1414]	Eureka, CA	Eureka	City	40.8020712	-124.1636729		[P0239]
[P1435]	Great Falls, MT	Great Falls	City				[P0066]
[P1494]	Knoxville, TN	Knoxville	City				[P0029]
[P1498]	Paragould, AR	Paragould	City	36.0584021	-90.4973286		[P0266]
[P1623]	Michigan City, IN	Michigan City	City	41.7075394	-86.8950297		[P0316]
[P1643]	Aberdeen, SD	Aberdeen	City				[P0059]
[P1661]	Myrtle Beach, SC	Myrtle Beach	City				[P0032]
[P1678]	Twin Falls, ID	Twin Falls	City	42.5629668	-114.4608711		[P0409]
[P1691]	Picayune, MS	Picayune	City				[P0041]
Person	Surname	Given	Call	Suffix	Prefix	Title	Gender	Birth date	Birth place	Birth source	Baptism date	Baptism place	Baptism source	Death date	Death place	Death source	Burial date	Burial place	Burial source	Note
[I0044]	Garner	Lewis Anderson	Anderson	Sr		Dr.	male	21 Jun 1855	[P1435]	Baptize registry 1850 - 1867 Great Falls Church				28 Jun 1911	[P1678]		1 Jul 1911	[P1678]
[I0106]	Garner	Robert W.					male	24 Apr 1826/7 (Julian)	[P1131]					3 Feb 1916	[P1388]		5 Feb 1916 (Mar25)	[P1494]
[I0045]	Martel	Luella Jacques					female	23 Jan 1852	[P1414]					28 Apr 1921	[P1661]		30 Apr 1921	[P1661]
[I0107]	Zieliński	Phoebe Emily					female	12 Apr 1827	[P1132]					7 Mar 1882	[P1398]		9 Mar 1882	[P1623]
Marriage	Husband	Wife	Date	Place	Source	Note
[F0017]	[I0044]	[I0045]	1 Apr 1875	[P1498]
[F0018]	[I0106]	[I0107]	4 Oct 1849	[P1498]
[F0019]			3 Jan 1823	[P1691]
[F0306]
[F0307]			4 Apr 1840
Family	Child
[F0018]	[I0044]
[F0019]	[I0106]
[F0306]	[I0107]
[F0307]	[I0045]

[DavidMStraub]

Is this a Gramps Web question or not? If yes, we should move it to the appropriate forum.

Gramps Web currently indeed doesn’t support “wiping” the tree through the API. If you want to start over, just delete the database.

Since you say you have extensive programming experience, you could have a look at a new tool I built a few days ago to do batch operations with Gramps Web API - it is just a thin wrapper around requests but makes the authorization part a bit less painful: GitHub - DavidMStraub/gramps-web-api-client: A simple Python client to interact with a Gramps Web API instance.

Be warned though - this is definitely experimental and not suited for average users.

But you could use it to walk through your CSV in a Python script and create the people programmatically using the format defined in the API docs.

[geophile]

Well, it is an ignorant question, as the difference between Gramps and Gramps Web is just now becoming apparent. Let me take a look at the desktop version and revise my thoughts about my workflow.

Thanks for your code, that may come in handy.

BTW: I’ve worked with many open source projects over the years, and I have to say that the support I’m finding here is amazing, on a par only with Postgres.