# Threading (or multiprocessing), performance and refactoring

**URL:** <https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527>\
**Category:** Development\
**Tags:** third-party-addon, performance, quick-view\
**Created:** [September 8, 2025, 12:10pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527 "2025-09-08T12:10:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 8, 2025, 12:10pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/1 "2025-09-08T12:10:51Z")

</div>

Hello !  
I am starting and looking at a little bit refactoring and performance issues on an old 3rd party addon. It was generated some years ago and tested with old hardwares and few resources. So, the idea was to generate (Yet) another alternated interface to Relationships Calculator module and to provide an output (_via an hack of standard way for OpenDocument Spreadsheet exporter_).

 ![RelID_menu](https://global.discourse-cdn.com/free1/uploads/gramps/original/2X/2/21ebb4e3dae479c2616f49cb2835f7f9e6608425.png)

 ![RelID](https://global.discourse-cdn.com/free1/uploads/gramps/original/2X/e/ef5a8142e7ed4370c09ccf807f2473a42d7ab9c8.png)  
Performances sounded good by using `example.gramps` data (less than 5 secondes for calculations and display) with default values (generations depth).

As filter rules performances have been improved on Gramps 6.x, (and before making migration from 5.2 to 6.0), I started to look at ways for improving this addon tool. As it re-uses many times the `Relationships` module and as this could be expensive (memory, calculation), maybe the first step might be a quick refactoring. Also, the more number of columns will increase, the more it needs time for calculations and display. So, maybe something around this too.

The first exploration (on my side) is to try to look at `threading` and `multiprocessing` modules. It makes sense for more modern configurations and more and more large dataset. It is the first time that I use/call it. The basic test was to replace some possible expensive sections and put it them to their threads.  
Well, the total time process increases from 5 secondes to 12 secondes with a simple database and default values (~2000 individuals)! I did not test with large databases. So, is it normal or I generated more mistakes with threads? Here a link to these first changes (tests, experimentations):

> **[Comparing...](https://github.com/gramps-project/addons-source/compare/maintenance/gramps52...romjeromealt:addons-source:maintenance/gramps52)**
>
> Contributed 3rd party plugins for Gramps. Contribute to gramps-project/addons-source development by creating an account on GitHub.

Something looks wrong to my thread usage and implementation.

```diff
- filter.add_rule(related)
- self.progress.set_pass(_('Please wait, filtering...'))
- filtered_list = filter.apply(self.dbstate.db, plist)
+ t_filter = Thread(target=self.t_filter_rules(related, plist))
+ t_filter.start()
+ t_filter.join()

+def t_filter_rules(self, related, plist):
 + """
+ """
+ self.filter.add_rule(related)
+ self.progress.set_pass(_('Please wait, filtering...'))
+ self.filtered_list = self.filter.apply(self.dbstate.db, plist)

```

Should I rather try to make a pool for threading? Or should rather improve the monitoring[1]? It looks like iteration and filtering might be different on Gramps 6.0, but I need to first improve it on 5.2.x, for understanding ways and modifications.

[1] `$ gramps -d “relation_tab”`

Best regards,  
Jérôme

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 8, 2025, 1:20pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/2 "2025-09-08T13:20:09Z")

</div>

Maybe, I should also add a close event to threads?

> <https://github.com/gramps-project/gramps/pull/1055/commits/8c61a0d01e33b1dc92cb1accb7fd85f0cbf5287d>

> <https://github.com/gramps-project/gramps/pull/1055#issuecomment-653806309>
>
> Threading is not generally used in Gramps because our code is not thread safe. …Not even a little bit. We don't have any way to lock our modifications to the db that would allow other threads to continue seeing a coherent view of the data. For instance a change to an object that might refer to other objects doesn't require that the other objects even be present; our standard code would eventually get around to creating the needed objects in the db, but another thread would not know when it was safe to examine the db. Our GUI uses signals from the db to try to remain consistent with the db, but is has taken literally years of effort to keep the GUI somewhat up to date. We are still finding issues there. Adding in changes from other threads would make this even more 'interesting'.

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 8, 2025, 2:55pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/3 "2025-09-08T14:55:47Z")

</div>

> [@romjerome](#):
>
> Well, the total time process increases from 5 secondes to 12 secondes with a simple database and default values (~2000 individuals)!

It seems that I should not use `return` into the thread section dedicated to rank calculation?

```python
def t_rank(self, dist, max_level):
    self.rank = dist[0][0]
    if self.rank == -1 or self.rank > max_level: # not related and ignored people            
        return

```

This might be into the loop (with `continue`) to avoid extra calculations (not related and ignored people).

---

<div class="post-metadata">

**Author:** ![dsblank](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/dsblank/32/3649_2.png) [@dsblank](https://gramps.discourse.group/u/dsblank)\
**Post date:** [September 8, 2025, 10:41pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/4 "2025-09-08T22:41:08Z")

</div>

There are better ways of handling threads than just starting 3. But in general, you’re going to run into issues with incompatibilities in accessing the database from different threads. I worked on trying to have a generic interface for parallel processing a couple of weeks ago, and I don’t think this is a good solution in general. Not to say that we can’t do parallel processing, but we have to do it in an abstract manner so that each database backend has options.

In any event, using Gramps 6.0.5 is going to be faster than any parallelism because of the Optimizer. You’ll want to move to it to get ready for 6.1.

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 8, 2025, 11:12pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/5 "2025-09-08T23:12:34Z")

</div>

> [@dsblank](#):
>
> There are better ways of handling threads than just starting 3

Do you mean a Pool or a high-level stuff like `concurrent` module?

> **[concurrent.futures — Launching parallel tasks](https://docs.python.org/3/library/concurrent.futures.html)**
>
> Source code: Lib/concurrent/futures/thread.py and Lib/concurrent/futures/process.py The concurrent.futures module provides a high-level interface for asynchronously executing callables. The asynchr...

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 8, 2025, 11:25pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/6 "2025-09-08T23:25:44Z")

</div>

> [@dsblank](#):
>
> But in general, you’re going to run into issues with incompatibilities in accessing the database from different threads

The primary idea was to move some basic calculations to functions and their related threads. I made a mistake on one of the function and the test for limiting entries was broken into the loop, generating extra calculations and increasing time process. So, testing my primary idea around threading should be now complete. After looking at some samples, I just understood that dealing with lock() and playing with location of join() might not provide the expected result. The thread on filter could be ignored anyway as it is outside of my iteration and loop.

I suppose that I got some of these incompatibilities…

```python
Exception in thread Thread-56:Traceback (most recent call last):
File “/usr/lib/python3.6/threading.py”, line 916, in _bootstrap_innerself.run()
File “/usr/lib/python3.6/threading.py”, line 1182, in runself.function(*self.args, **self.kwargs)
TypeError: ‘NoneType’ object is not callable

```

```python
Exception in thread Thread-58:Traceback (most recent call last):
File “/usr/lib/python3.6/threading.py”, line 916, in _bootstrap_innerself.run()
File “/usr/lib/python3.6/threading.py”, line 1182, in runself.function(*self.args, **self.kwargs)
TypeError: ‘NoneType’ object is not callable

```

`File “/usr/lib/python3/dist-packages/gramps/plugins/db/dbapi/dbapi.py”, line 1002, in _get_raw_data`  
`self.dbapi.execute(sql, [handle])`  
`File “/usr/lib/python3/dist-packages/gramps/plugins/db/dbapi/sqlite.py”, line 136, in execute`  
`self.__cursor.execute(*args, **kwargs)`  
`sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140130013738816 and this is thread id 140129322649344.`

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 8, 2025, 11:34pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/7 "2025-09-08T23:34:33Z")

</div>

> [@dsblank](#):
>
> we have to do it in an abstract manner so that each database backend has options.

Nothing currently common (python, others applications, etc.) or a limitation to gramps database backends?

> <https://github.com/gramps-project/gramps/pull/2101>
>
> This PR builds on your excellent SQLite optimizations by adding complementary ba…ckend-agnostic improvements to the DBAPI base class. I've kept all your valuable work while organizing things so SQLite-specific code stays in sqlite.py and generic improvements benefit all backends.
> 
> \## What This Adds
> 
> \### 1. Real Cursor Support (5 lines, huge memory savings)
> \- Returns iterators for backends that support real cursors (PostgreSQL, MySQL)
> \- Gracefully falls back to regular lists for SQLite/BSDDB
> \- Reduces memory usage by 10x for large databases
> 
> \### 2. Lazy Loading (Optional, 50% memory reduction)
> \- Returns a proxy that loads person data only when accessed
> \- Dramatically reduces memory for operations that don't need all object data
> \- Completely optional - existing code continues to work
> 
> \### 3. Improved Prepared Statements (Backend-agnostic)
> \- PostgreSQL/MySQL get real prepared statements
> \- SQLite gets your cached query strings
> \- Same API for all backends
> 
> \### 4. Batch Operations (Using executemany)
> \- Uses executemany() when available
> \- Falls back to individual commits for other backends
> \- 100x speedup for bulk imports
> 
> \## What I Preserved
> 
> All @dsblank 's proposed SQLite optimizations remain intact:
> \- WAL mode, cache settings, memory-mapped I/O
> \- Connection pooling
> \- Your bulk insert/update methods
> \- Performance indexes
> \- JSON query functions
> 
> \## Organization Changes
> 
> I moved SQLite-specific code to the SQLite class:
> \- VACUUM now lives in SQLite.optimize\_database()
> \- Base DBAPI.optimize\_database() only does ANALYZE (widely supported)
> 
> \## Testing
> 
> All changes include graceful fallbacks and maintain backward compatibility. The improvements are additive - existing code continues to work exactly as before.
> 
> \## Impact
> 
> Combined with your optimizations:
> \- Your changes: 2-10x performance improvement
> \- These additions: 10x memory reduction, 2x query performance
> \- Together: Makes Gramps viable for 100,000+ person databases
> 
> Let me know if you'd like any adjustments or have questions about the implementation\\!
> 
> Best,
> Greg

About Gramps 6 installation, I need a large OS upgrade, before…

> [@Pas de langue française dans Gramps 6 installé depuis Flatpak](https://gramps.discourse.group/t/pas-de-langue-francaise-dans-gramps-6-installe-depuis-flatpak/7351/9):
>
> Not tested… $ flatpak config languages: \*unset\* (default: fr) extra-languages: \*unset\* $ flatpak install flathub org.gramps\_project.Gramps 1. org.freedesktop.Platform.GL.Debug.default 24.08 i flathub \< 551,3 MB 2. org.freedesktop.Platform.GL.Debug.default 24.08extra i flathub \< 551,3 MB 3. org.freedesktop.Platform.GL.default 24.08 i flathub \< 156,8 MB 4. org.freedesktop.…

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 10, 2025, 11:46am UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/8 "2025-09-10T11:46:14Z")

</div>

Well, after some refactoring, I looked at additionnal (or advanced) features like family network centrality, or shared subtree size… It sounds very good!

 ![imp](https://global.discourse-cdn.com/free1/uploads/gramps/original/2X/1/1e9ec8a7b443adf0ed62eb75c476ac13694fada7.png)

(_sorry, I did not force english locale for the screenshot_)

You can see two additionnal columns. That’s something that I cannot add alone on one day (maybe on one week!)…

I was behind instructions to the “copilot” and the spirit and ideas are still present.😉

Maybe I can polish the RelID (ID Rel) map design. Anyway, by re-using most core modules from gramps, we could quickly go very far on relationships analysis. I will not make a PR. This analyse (analyze) tool has some custom behaviors like asking to select a folder for the save .ods action. Sure, after refactoring, the code is more pythonic and clean, but some sections are very experimental or still pending (DNA stuff?):

```python
name = name_displayer.display(person)
# pseudo privacy; sample for DNA stuff and mapping
import hashlib
no_name = hashlib.sha384(name.encode() + handle.encode()).hexdigest()
_LOG.info(no_name) # own internal password via handle

```

the model should very close to the TreeView, but I cannot maintain a new View (e.g., Relation Views Category) or make some deep interactions with filter rules (e.g., like on some Graphical Views).

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 11, 2025, 10:41am UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/9 "2025-09-11T10:41:11Z")

</div>

Note, during experimentations, I also got a ProgressMeter window via CLI…

`$ gramps -O ‘example’ -a tool -p name=relationtab -d “relation_tab”`

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 11, 2025, 12:40pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/10 "2025-09-11T12:40:38Z")

</div>

> [@dsblank](#):
>
> In any event, using Gramps 6.0.5 is going to be faster than any parallelism because of the Optimizer. You’ll want to move to it to get ready for 6.1.

I made a draft Pull Request against gramps60.

> <https://github.com/gramps-project/addons-source/pull/769>

> Copyright (C) 2000-2006 Donald N. Allingham

It was the plugin environment (and global gramps application).

> Copyright (C) 2008 Brian G. Matherly

It was around the GUI stuff, maybe the hack for the folder selector (gtk2 to gtk3).

> Copyright (C) 2010 Jakim Friant

It was the filter rules handling into the tool.

> Copyright (C) 2012 Doug Blank

Might be related to the TreeView model, the ODS file format support or the Tools options.

I kept way to generate issues around threading, via a dedicated function! So, it will return errors (console) without crashing the tool. The addon is not listed (_include in listing = False_) and there is an additional file (`import number` ; `numbers` already exist as built-in python set).

---

<div class="post-metadata">

**Author:** ![dsblank](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/dsblank/32/3649_2.png) [@dsblank](https://gramps.discourse.group/u/dsblank)\
**Post date:** [September 11, 2025, 2:16pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/11 "2025-09-11T14:16:14Z")

</div>

@romjerome I am not sure what you are doing. At one point you have:

```python
1. thread = Thread(target=self.long_running_task, args=(default_person, person,))
2. thread.start()
3. dist = self.relationship.get_relationship_distance_new(
       self.dbstate.db, default_person, person, only_birth=True)

```

where line 1 calls `long_running_task` that does this:

```python
        dist = self.relationship.get_relationship_distance_new(
            self.dbstate.db, default_person, person, only_birth=True)

```

But you call that again on line 3. I don’t think this code is doing what you think it is doing. And this doesn’t look like proper thread management. But even if it were done properly, Gramps makes no guarantees that you can handle database actions in a thread.

---

<div class="post-metadata">

**Author:** ![dsblank](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/dsblank/32/3649_2.png) [@dsblank](https://gramps.discourse.group/u/dsblank)\
**Post date:** [September 11, 2025, 2:36pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/12 "2025-09-11T14:36:58Z")

</div>

BTW, there is a “git blame” button and cli:

 ![image](https://global.discourse-cdn.com/free1/uploads/gramps/original/2X/f/fac45b6f7962b6e8403c002fe710abc3cd39eb35.png)

But in general it is impossible to associate a copyright with particular lines (because they may be revised or removed).

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 11, 2025, 5:57pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/13 "2025-09-11T17:57:16Z")

</div>

> [@dsblank](#):
>
> But you call that again on line 3. I don’t think this code is doing what you think it is doing. And this doesn’t look like proper thread management. But even if it were done properly, Gramps makes no guarantees that you can handle database actions in a thread.

Yes, it was the plan! I just call it twice to keep a trace of the incompatibilities you warned me and also listed on:

> [@Threading (or multiprocessing), performance and refactoring](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/6):
>
> The primary idea was to move some basic calculations to functions and their related threads. I made a mistake on one of the function and the test for limiting entries was broken into the loop, generating extra calculations and increasing time process. So, testing my primary idea around threading should be now complete. After looking at some samples, I just understood that dealing with lock() and playing with location of join() might not provide the expected result. The thread on filter could be…

Something like a testing without crash because the thread section does not really generate a useful data. The second one does. Sure, a final version (polish outside draft PR) should remove the thread “lines”.

I remember that one script on experimental `Gram.py`could generate something very close (at least the primary idea).

> [@New Gram.py Script for 6.0](https://gramps.discourse.group/t/new-gram-py-script-for-6-0/7192/18):
>
> Relationship list: from gramps.gen.relationship import get\_relationship\_calculator rc = get\_relationship\_calculator() columns("Relative", "Relationship, of", "Active Person", "Dist org", "Dist other") for person in people(): (relationship, \_ga, \_gb) = rc.get\_one\_relationship( self.db, active\_person, person, extra\_info=True, ) if relationship: row(person, relationship, active\_person, \_ga, \_gb)

By looking at some comments on code, history and documentation, there was some performance issues in the past (e.g., SQLite backend 30% slower than bsddb3 backend). As it was some years ago, I just try to improve a little bit any possible way to limit any slow down or extra time process with large database.

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 11, 2025, 6:32pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/14 "2025-09-11T18:32:44Z")

</div>

> [@dsblank](#):
>
> But in general it is impossible to associate a copyright with particular lines (because they may be revised or removed).

It seems that I just added a copyright when I looked at an other module, or section (piece or part) of code into Gramps, then re-used it into the addon.

Maybe it makes sense for the core set of plugins. Does it still make sense when we create an addon? It seems that I re-used the GUI logic, Plugins classes for Tools, part of the OpenDocument logic and just hide my name when it should be added. For me, it is a plugin for gramps. So, the licence, maybe the copyright, are related to development around gramps. As you pointed out `git blame` (or `svn blame` !) could find the history of commits and authors.

My problem is now on coding Policy, AI, copyright & co. I should be able to quickly add advanced calculations (via new columns on table) related to Relationships, DNA, Families, Surnames, Statistics, etc. Most of them will be basic calculations (few lines), but what about any addition from copyright(ed) code and provided by an AI. Do not worry, I checked before adding any lines from the AI suggestions. 😉 The problem might be to include a “real” algorithm. I remember the `Lunisolar calendar` issue (e.g., for Chinese dates, see bug tracker). It was not included, despite some patchs (even by myself!). Today, any AI will include it (at a glance).

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 11, 2025, 10:05pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/15 "2025-09-11T22:05:25Z")

</div>

> [@dsblank](#):
>
> I don’t think this code is doing what you think it is doing.

Is it dangerous to keep it? Can this crash more than Gramps? Can this corrupt the database (in read only state/mode)? Python uses its pseudo-sandbox (GIL environment), isn’t it?

I saw some projets like this one:

> **[GitHub - colesbury/nogil: Multithreaded Python without the GIL](https://github.com/colesbury/nogil)**
>
> Multithreaded Python without the GIL

but I thought that warnings were only for printing informations and threads were closed before.

**update** : I suppose that I understand now what you wrote about a proper implementation… For testing threading, I should always (or often) skip and limit use of `return` into the related function and prefer `yield` for performances issues.

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 14, 2025, 4:38am UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/16 "2025-09-14T04:38:58Z")

</div>

An ambitious feature could be: “_Person Comparison: Add a feature to compare relationships and metrics between two specific individuals_.”

So, it does not assign UUID for records into a Big Tree, like FamilySearch or Ancestry. It only checks our Relationships maps. I am not sure that we could fully hide surnames, maybe the soundex could be an alternative. Anonymized matrix(es) means decentralyzed stuff and less ressourses too. In the 90th (1990), ‘Tafel Matching System’ was very popular, at least in France, and ‘Geneanet’ more and less has improved it.

> <https://github.com/gramps-project/addons-source/pull/769#issuecomment-3286420004>
>
> Some suggestions for additions to improve it:
> 
> \* Add Graphical Visualizations:… Integrate libraries like Matplotlib or Plotly to create graphs of relationships and metrics.
> 
> \* Export to Other Formats: Add the ability to export data to formats like CSV, JSON, or PDF.
> 
> \* Advanced Filtering: Allow users to filter results based on specific criteria, such as gender, period, or number of generations.
> 
> \* Person Comparison: Add a feature to compare relationships and metrics between two specific individuals.
> 
> \* External Data Integration: Allow importing data from external sources like FamilySearch or Ancestry.
> 
> \* Enhanced User Interface: Add customization options for the interface, such as the ability to change the language or theme.
> 
> \* Performance Optimization: Improve performance by using techniques like multithreading or caching.
> 
> \* Enhanced Documentation: Add detailed comments and comprehensive documentation to facilitate understanding and usage of the script.
> 
> \* Unit Testing: Add unit tests to verify the proper functioning of different functions and methods.
> 
> \* Enhanced Logging: Improve logging to include more detailed information and different logging levels (DEBUG, INFO, WARNING, ERROR, CRITICAL).
> 
> Here's an example of how you could add a person comparison feature:
> \`\`\`
> def compare\_people(self, person1\_handle, person2\_handle):
> """
> Compare relationships and metrics between two people.
> Args:
> person1\_handle: Handle of the first person.
> person2\_handle: Handle of the second person.
> Returns:
> dict: A dictionary containing the differences between the two people.
> """
> person1 = self.dbstate.db.get\_person\_from\_handle(person1\_handle)
> person2 = self.dbstate.db.get\_person\_from\_handle(person2\_handle)
> 
> comparison = {
> 'name': {
> 'person1': name\_displayer.display(person1),
> 'person2': name\_displayer.display(person2)
> },
> 'relationship': {
> 'person1': get\_relationship\_between\_people(self.dbstate, self.relationship, self.default\_person, person1),
> 'person2': get\_relationship\_between\_people(self.dbstate, self.relationship, self.default\_person, person2)
> },
> 'surname\_diversity': {
> 'person1': FamilyPathMetrics.calculate\_surname\_diversity(self.dbstate.db, person1\_handle, generations=5),
> 'person2': FamilyPathMetrics.calculate\_surname\_diversity(self.dbstate.db, person2\_handle, generations=5)
> },
> # Add other metrics to compare here
> }
> 
> return comparison
> \`\`\`
> This function compares names, relationships, and surname diversity between two people and returns a dictionary containing the differences. You can then use this dictionary to display the comparison results in the user interface. \_~codestral Agent\_

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 22, 2025, 8:45am UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/17 "2025-09-22T08:45:25Z")

</div>

> [@dsblank](#):
>
> In any event, using Gramps 6.0.5 is going to be faster than any parallelism because of the Optimizer. You’ll want to move to it to get ready for 6.1.

Doug, I suppose I get it more clear and clean now (yes, it is possible!) and I could provide a simple test or proof-of-concept of these testing.

How make such code fit for a Family Tree with more than 200 000 individuals ?

```python
from gramps.gen.filters import GenericFilterFactory, rules
filter = FilterClass()
self.filter = FilterClass()
default_person = self.dbstate.db.get_default_person()
plist = self.dbstate.db.iter_person_handles()
if default_person: # rather designed for a run via GUI...
    root_id = default_person.get_gramps_id()
    ancestors = rules.person.IsAncestorOf([str(root_id), True])
    descendants = rules.person.IsDescendantOf([str(root_id), True])
    related = rules.person.IsRelatedWith([str(root_id)])
    self.filter.add_rule(related)
    _LOG.info("Filtering people related to the root person...")
    self.progress.set_pass(_('Please wait, filtering...'))
    self.filtered_list = filter.apply(self.dbstate.db, plist)
    for handle in self.filtered_list:
       ...

```

Someone reported that this (or on an other piece of code into the addon) could take more than 2 hours.

Sure, to call Relationships will use some ressources, but the poor timer issue seems (for me) rather on the filter rule.

Checking and retrieving informations for all filtered individuals via person\_handle iteration (without filtering), seems rather to take around maximum 15 minutes. Does ‘related’ filter rule have been improved/optimized on Gramps 6.0.5?

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 22, 2025, 1:29pm UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/18 "2025-09-22T13:29:39Z")

</div>

> [@dsblank](#):
>
> @romjerome I am not sure what you are doing.

Right, it seems that I did some strange experimentations!

I just see now, that the `IsRelatedWith` filter rule is already using a pseudo-parallel processing!

I was wondering why I get some blocs of records after the filtering pass. In my mind, the filter rule was only for limiting the dataset. Currently the progress meter seems (for me) to list a part of the filtered people while (or during) iteration on the first matching handles! Looking at the recursive code on filter rule I cannot really monitor this from the addon, which is only an interface. This cannot explain the performance issue with large database (I suppose), maybe it just only explains why I wanted to explore the threading during the iteration pass.

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 24, 2025, 9:54am UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/19 "2025-09-24T09:54:43Z")

</div>

> [@romjerome](#):
>
> prefer `yield` for performances issues.

I get more difficulties to properly implement custom filter rules support on tools, than use of yield and generators on code!

---

<div class="post-metadata">

**Author:** ![romjerome](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/romjerome/32/1489_2.png) [@romjerome](https://gramps.discourse.group/u/romjerome)\
**Post date:** [September 27, 2025, 7:25am UTC](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527/20 "2025-09-27T07:25:19Z")

</div>

Finally, the workflow, for human with or without “machine” support, did not really change! And I still find possible issues never reported before and outside AI monitoring or check passes…

> [@Possible issues on 'Relationship' module](https://gramps.discourse.group/t/possible-issues-on-relationship-module/8603):
>
> Hello. During testing performances for an addon[1], I got a strange return under Gramps 5.2.5 (I do not have installed Gramps 6.0.5 yet): File “/usr/lib/python3/dist-packages/gramps/gen/relationship.py”, line 1623, in \_\_apply\_filter childrel[0][1], IndexError: list index out of range Well, the code (and line) seems to be the same on master branch (and I guess on 6.0 one) So, maybe an issue (corner case?), but I am not able to provide a good unit test or real testcase. [1] [https://www.gr…](https://www.gramps-project.org/wiki/index.php/Addon:Display_relations_and_distances_with_the_home_person#Test)

Sure, AI’s answer sounds good and solution seems logical. I skiped now the threading experimentations, even indirectly tested with `yield`, `stack`, [lists](https://docs.python.org/3/tutorial/datastructures.html#using-lists-as-queues) and gtk events. So, code should be more pythonic and modern, and there is no real features addition, except maybe more columns.

One issue is still pending. It is specific to GUI (Gtk TreeView model). No crash or error via CLI. It is on gtk window used for displaying the list of results. As a pseudo-sosa/kékulé numbering was expected, an experimental numbering for descendants or cousins has been tested. The main problem is to have a real number, not too complicated to understand at a glance. Maybe a positive one, not used by sosa/kékulé (so, not from 2 to infinite) and uniq. Maybe something around 0 to 1? To have a float value will crash the gtk model

```python
self.model = ListModel(treeview, self.titles)
for entry, sort_key in batch: 
    self.model.add(entry, sort_key)

```

It is a design issue. So no more related to refactoring, polishing or cleanup, but still around performance and maybe threading with gtk window.

ps : there is also a cache issue, which is for re-using a list. This might display an incomplete list (signal, active person and database, etc.)

[Next page](https://gramps.discourse.group/t/threading-or-multiprocessing-performance-and-refactoring/8527.md?page=2)