I’m finally looking to start work on integrating Betty with Gramps’ Python APIs directly instead of parsing Gramps XML.
From an end user perspective, what I would like to achieve is:
read existing family trees from a Gramps installation on the system, unrelated to Betty
read files of any of the types that Gramps supports (GEDCOM, Gramps XML, etc)
From a development perspective, I would like to achieve, for each of the above:
for Gramps to handle all file interactions, and for Betty to no longer parse any genealogy files itself anymore
for Betty to be able to interact with Gramps family trees through Gramps’ own Python data model
Based on that, I think the questions I should ask you are:
How do I ‘bootstrap’ Gramps in Python, to be able to call Gramps APIs? I assume there is some setup involved because as an application Gramps probably requires some environment variables, such as where to save its data.
Once bootstrapped, how do I create a temporary family tree, tell Gramps to load a file into it, get the Python model for that family tree, and when done, remove the family tree?
How do I bootstrap Gramps so that it can read the family trees from a Gramps installation elsewhere on the system, separate from Betty? And once bootstrapped, how do I load a family tree, e.g. by its name?
I believe the answers to these questions will provide me with good starting points for my work, and allow me to progress from there. I’m a proponent of type safety, and Betty is fully typed, so I am expecting to be able to contribute PRs to Gramps that help with your recent efforts to introduce typing.
@pytest.fixture
def database():
# Open the GRAMPS database (replace with the path to your GRAMPS database)
db = gramps.gen.db.utils.open_database("mini-test-6_0_0")
# db = gramps.gen.db.utils.open_database("work/test/data/6.0.0")
# db = gramps.gen.db.DBLink('work/test/data/6.0.0')
yield db
db.close()
As already mentioned the gramps.gen.db.utils module is a good place to look:
You can call the make_database method directly and open an in-memory database which may be useful.
db = make_database("sqlite")
db.load(":memory:")
Importing a file takes a few steps, but have a look at the cl_import function in the command line interface.
Our importers are plugins, so you have to create an instance of a plugin manager and then get a list of importers. Loop through these and choose one with the extension of the file you want to import. Then its just a matter of instantiating an importer and calling it with a database, file and User class which is need to provide feedback to the user such as progress.
More seriously, I see that you are parsing Gramps XML with lxml lib. Does find() [or findall()] “consume” a lot of memory? Can Betty support large content or data into memory (Gramps XML)? I remember some issues around lxml and ressources, by loading large Gramps XML, in the past. You are also using asyncio (await & co), I suppose it is related to this type of performance issue?
There are no (significant) memory issues in this area, but I’m sure loading data through Gramps directly will speed things up somewhat (although loading data is only a tiny part of what Betty does so it won’t impact the overall experience much)
Slight bump in the road: the Gramps package requires several packages that require C code to be compiled, as well system packages to be installed. That’s not a end user hurdle I am willing to add to Betty (especially considering the current situation allows for the loading of Gramps data without additional dependencies). Would it be possible (in a near future release) to be able to install Gramps’ APIs without being forced to install all dependencies that are not prebuilt for different platforms? For example, cairo is unneeded in my case, as Betty does not need Gramps to manipulate graphics.
E.g. considering Gramps 6 is not (fully) stable yet, one option would be to make all of these packages optional dependencies. That way someone running pip install gramps would only get the Gramps code and dependencies for which cross-platform builds exist (or that are pure Python), and if they want full service they must instead run pip install gramps[cairo] for cairo support (and then manually take care of the additional system requirements).
That is indeed how pip install gramps works. What doesn’t work is that some modules are imported when they aren’t needed. We’re discussing making rearrangements in 6.1 or later.
It would technically be a BC break to make pip install gramps install fewer required dependencies in 6.1 than in 6.0, albeit not one that many people will run into. Can you confirm that that is a change you as a team are willing to make? If not, then now, before 6.0.0 is released, is the time to reduce the number of packages installed by pip install gramps (because expanding it later would not break BC and is perfectly fine to do).
Wait, just to be clear, are you talking about importing modules (a runtime penalty), or installing packages (an install-time issue)? I’m talking about the latter.
There is currently in Gramps 6.0 setup exactly one install_requirement: orjson. That is needed to load the database. (Techincally, you can use json, but the codebase uses orjson).
Gramps 6.0 will be released any day now, and no such changes will be made.
Of course, we can’t guarantee that 6.1 will be able to work as a library (without GUI stuff) but there are a few of us committed to that goal. It shouldn’t be controversial, as it is just a rearrangement of imports.
These are related. Currently, pip install gramps only installs itself and orjson (and you can even install with no dependencies with a pip flag). But that doesn’t mean that gramps will work. There are imports of assumed-installed items. We need to rearrange those so you don’t encounter imports (like cairo) if you don’t need them.
Yes, excellent. That’s indeed my problem and we were indeed talking about the same thing. Also what I said was slightly confusing because you’re right, none of these packages are indeed installed as dependencies by default, it’s that they’re all being imported. Sorry for the confusion there!
What you described sounds like it would make Gramps ready for a use case like mine
Yes, exactly. And other uses like gramps-web. Currently it runs in a docker container because (at least partially) all of the gramps GUI stuff is needed, and can’t easily be installed.
(I did this exercise once before about 17 years ago because I wanted to use a different GUI (tkinter). But that was too many changes for the time. However, the time is right )
Are you intending to just use the import/export plugins with the Gramps engine to expand Betty from Gramps (.gramps/.gpkg) and GEDCOM (.ged) to more formats?
“Betty visualizes and publishes your family history by building interactive, encyclopedia-like genealogy websites out of your Gramps and GEDCOM family trees.”
Hah, I understand the burden of things done in times long gone.
Would you be able to give me a timeline for when such changes may be merged and become part of a release? That would help me plan my work to occur around the same time and provide feedback or code contributions.
If possible, yes, I’d like to use this new integration to allow Betty to load data from any of the file formats Gramps can load data from. At the moment Betty does not actually parse GEDCOM files, but instructs users to load those into Gramps first, then export them as one of the Gramps formats, so these changes would be a huge step forward without adding a massive maintenance burden.
The goals for a release are always a combination of high-level goals (set by @Nick-Hall ) and things that individuals are willing to tackle. This last time around, serious changes started around Nov 2024 and the final release is now being readied (Mar 2025). So I would estimate that later this year is reasonable to expect a 6.1.
If @Nick-Hall approves it, I’d be glad to help make a gramps-as-library happen then.
Unfortunately I don’t. When I started digitizing my genealogy I went straight for Gramps because it could do so much more than basic GEDCOM. And Betty’s feature scope is publication rather than data collection, and I’m trying to keep it that way for now because there’s plenty of work to be done already
Say, hypothetically, that some of these improvements would be relatively minor changes (move a few imports around here and there, for example). Are those changes that could possibly make it into minor releases?
