It is probably related to fetching 1000 rows at a time.
From Perplexity.ai :
The 1000 row limit in Python when using databases is not a limitation of Python itself, but rather a common default setting in database drivers or libraries used for interfacing with databases. Many database connectors, such as those for SQLite or PostgreSQL, have default fetch sizes to prevent excessive memory usage when retrieving large datasets. This limit can usually be adjusted by configuring the fetch size in the database connection settings. Python’s database access layer is seen as less developed compared to technologies like JDBC and ODBC, which can also contribute to perceived limitations in handling large data volumes efficiently
Okay, it could be if I want get ALL table records (all people). But I use search conditions. Sql query will look like this:
select * from people inner join ... where ....
In this case I dont expect a lot of records. It will be several records only, or even 1 record only. You are telling about case when my 10000 records are chunced by 1000 and then python will iterate all them and find what I need. But I tell you that search should make not python. This is the job for DB, because it work fast.
Lets approximate the search perspectives accorting to my measurements:
6000 people - 0.6 sec
11000 people - 1.3 sec - 1 sip of beer
22000 people - 2.6 sec - 2 sips of beer
33000 people - 3.9 sec - 3 sips of beer
44000 people - 5.2 sec - 4 sips of beer
55000 people - 6.5 sec - 5 sips of beer
66000 people - 7.8 sec - 6 sips of beer
…
… more people … died of an overdose
Thats because seacrcing makes python, but not DB. Right?
I have a complex filter that took 178 seconds yesterday. Granted, it is not a filter I run often. But the times you posted I do not see as unreasonable. It probably takes you longer to enter the search parameters for your 11K people at 1.38 seconds’ example.
So, as I originally posted… Relax. Before you give yourself an ulcer. How much more genealogy work would you get done if you got back those 1.38 seconds?
Do you value your time? Judging by your words - not very. OK, let’s count.
In 2024, 365 days, I will spend 1.32 seconds searching. Let’s say I have 100 such searches per day. We have 132 seconds every day. For the whole year, I will spend 800 minutes.
In 2025, the base will increase and the time will be 1600 minutes.
In 2026, the base will increase and the time will be 2400 minutes.
In just three years, I will spend in waiting 800+1600+2400 = 4800 minutes = 80 hours. Now multiply this time by the number of GRAMPS users whose time you don’t want to respect.
When filtering on people, Gramps will fetch objects from the database using something like:
SELECT person_object from Person;
Then our person filter will apply functions to the objects retrieved which return either a True or False value. Multiple functions, sometimes quite complex, can be applied to each object.
The performance issues will be due to the object serialisation time, but we do get some advantages to compensate for this:
Users can use their existing filters
The filters will work on any database backend (e.g. MongoDB, Neo4j, OrientDB …)
Only python is necessary to write new filters
The best suggestion to enhance our filters that has been proposed so far is to use the backend to narrow the results before applying our existing filters. The database already contains columns and indexes to allow a prototype to be written.
Okay, it could be a “golden mean”. But only why do you need any DB like Mongo and others? It makes everything difficult with no benefits. The users need one good working and fast DB but not possibility in potential Mongo (or others) connection.
@Nick-Hall what are the chances of Gramps getting an update that will improve search performance? Obviously, the Gramps community does not want to do this because they consider it the norm for themselves and other users. If so, then this app is not for me.
I have written many times about various improvements, acceleration, usability and so on. But in 2 years, all my offers were rejected. And all because everyone is focused on some new interesting features and addons, and nobody is interested in improving the old code (often not their own code).
Can I be honest? Gramps has a huge amount of interesting functionality. I probably still don’t know about half of its possibilities. I think it is TOP-1 in this sense. And for that I love him very much. But at the same time, most addons and features have some problems. Or performance, or usability, or something else. And this really annoys me (only me?). This makes it difficult to enjoy using it.
I guess my last suggestion would be:
I suggest including half updates and half new functionality in each new release. The old must be improved. It might make sense to create a forum to vote on new features and various minor improvements. Those with the highest number of votes will be included in future editions. There is currently no voting, and any requests for improvements are rejected at the discussion stage. But what if many users have the same request? Voting individual features would help clarify this.
Thank you guys for everything!
And Im sorry. I don’t want to offend anyone, I’m just explaining what’s wrong with the app, for its sake.
But where is the fun, enjoyment or intellectual challenge
in that.
So if you do a DNA test on Ancestry put you name as the linked home
person on a tree of 1 and sit back until your tree goes back to the
Neanderthal 3rd cave on the left and think you have got value for money
fine.
But not my style.
phil
If you really do a lot of different searches, it may be faster to export your tree to RootsMagic, and use that as a reference tree. I have such a tree, with my own research, and a big GEDCOM that I once found on the web, with more that 600,000 persons, and searches are fast, because RootsMagic has a full index on names, which are stored in a separate table.
RootsMagic Essentials is free, and works well in Wine too.
This subset of secondary objects would be the same as if the Export was selected for that Person filter. (Another option might be to make People who are in Ancestor and Descendant Families of the Home Person)
There are no rules for Ancestor/Descendant families of a Person or Person filter. So since I am the Home Person on my Tree, so the captures below manually select my Parents family ID.
I have made some experiments with a different approach: I wrote a “Fulltext search” addon that can make quite fast searches across the whole database (well, on selected attributes only). See gramps/addons/Fulltext at master · kkujansuu/gramps · GitHub.
It may be possible to integrate this also with the sidebar Filter gramplet - at least for searching people by name.
I’ll investigate to see if I can suggest something simple that will give immediate benefits.
Our developers are all volunteers. I can’t dictate what they choose to work on.
We could create a category in this forum where users can post feature requests. People could vote with a thumbs up (and possibly also thumbs down) on a topic. Unfortunately, the Topic Voting plugin for Discourse is not available on our free plan.
If/when the blobs are replaced with json, would it be possible for some new python code to use the existing custom filter details (which are stored as XML) to generate pure SQL queries containing all the necessary joins and “where” clause? The SQL could be generated on the fly (single use), or perhaps stored as custom “view” statements. Either way, it should leverage existing indexes.
A user on Facebook was wanting to plot birth places of a particular generation. So we suggested a 3-stage custom filter. However the 1st stage (finding 16 people 5th generation of the people is… ssslloooooowww … 55.30s).
