Yes, and the example by @bugbear (where a Person 2-stage custom filter runs faster then the 1-stage Event custom filter it references) might be a good place to start visiting.
or:
0012034: Overhead of Filter Gramplet seems excesssive in large Trees
Would it be possible to share the 38,655 person family tree mentioned in 12034 with me?
I’ll have to dig out the Laptop (which is on its last legs and may have to be nursed along) and export the 2020 backup with Privacy exclusions. Then eMail to you. Might take a day or 2. [sent 3 days later.]
Revisited as part of the Gramps 6.0 enhancements.
See
For all of you that have been advocating for a graph data structure in Gramps: you were right.
I have been working making the new Gramps Connect HTML-based app easy to use with family trees with up to 100k people. But using the get-relationship functions in Gramps core it just isn’t feasible (as many of you already know), and relationships are slightly important in a genealogy program. ![]()
I tried many things to speed-up the function, but the solution was, of course, a graph data structure and algorithm:
This drop-in replacement for the functions (designed for gramps-web-api) builds a new virtual table, childOf on the fly. This is a graph. This only takes milliseconds to run through the entire 100k people’s families, and once that is done, relationship-type queries only take single-digit milliseconds. It implements the private proxy as well.
This speeds up regular relationship queries on the 100k people tree that were anywhere from 15 seconds to over 3 minutes, into near instant response. If you want to know more about the algorithm, check out the code, or ask me for an overview.
(I’m not sure it makes sense to make the childOf table persistent… it would take some work to properly maintain it, and we’d need one for each proxy you want to support. But it is fast enough to just recreate as needed.)
So, thanks to all of you who kept pushing on the graph idea, and to the idea that we don’t need third-party tools to implement it.