While looking at a new feature for Gramps , it struck me that the add-on was storing CSV data in a Gramps Note and some of the bugs were standard parsing problems. And that this new Note type probably needed to embed headers for future-proofing. Particularly since the DNA segment map data could come …

[image] Obstipatus: Because I have other services and processes running, This is not a problem, even if you have only 4 cores without threading, you can create 30 threads. It will work, but only 4 in parallel will be running. It will be less performant. that’s all.

[image] SNoiraud: It will be less performant. that’s all. The whole point was the performance and to limit the resources so that it was possible to run other multiple other services in parallell not in serie.

Another thing, when I load my 32k place database it take 28 to 30 second to start Gramps, That is way to long on a new Ryzen system with NVME SSD’s both as system and data storage. Multi-threading would have reduced that load. I’m not looking forward to the day I shall load my 800k + Place Name da…

[image] Obstipatus: Multi-threading would have reduced that load. I am not sure. The loading is not multithread when loading libraries or the database.

SQLITE can work in multithread mode, same can any other database server, and mongodb also support multithreaded clients

Still it is multi-threaded and when the sqlite database is on a SSD, the single thread doesn’t utilize the read speed of the SSD, but a multi-thread will.

[image] SNoiraud: The loading is not multithread when loading libraries or the database. This is something I really wonderabout, why is it that the loading of the Python Libraries isn’t loaded in a multi-threaded way? Why can’t the libraries after initializing be loaded in parallell into mem…

With my tree that ish 300 people, so small one, loading straight in to it takes maybe 6-8 sec to load, no single core get much load or anything, so that time is most likely software limited. Not representative for bigger ones most likely tho.

[image] Mihle: Not representative for bigger ones most likely tho. I have tested this on 2 systems, and one virtual machine, mostly because I use the mongodb connector that are much slower, so I wanted to see how much. I have tested 3 different databases (one with 891 people, a few hundred p…

Datamining standard libraries for Gramps?

Development

SNoiraud November 11, 2020, 6:57pm 31

Yes, but the problem is we have only one file for the database and this file is on the single disk. You can put 50 threads to read the database, you will not run faster.

Topic		Replies	Views
About Gramps usage and organization to store and research Help	27	1035	November 14, 2020
Threading (or multiprocessing), performance and refactoring Development third-party-addon , performance , quick-view	20	408	September 28, 2025
Gramps API integration Help third-party-addon , websolutions	31	482	April 14, 2025
JSON inside table Development third-party-addon , tools , data-import	33	433	December 16, 2025
What its the best way to get a family tree from TMG to Gramps in Windows? Help	39	1517	October 29, 2020

Datamining standard libraries for Gramps?

Related topics