Performance Issues with Gramps

Development
,
63

The large number of links is causing the problem. In v5.1, links in notes were only soft links, but in v5.2 we store them in a reference table.

I think what we do is delete all the links for a note and then insert them back again. So when you edit your note it is probably doing 200 inserts. I’d have to look at the code, but it may well be possible to optimise.

Update: After looking at the code, we should use executemany rather than execute in _update_backlinks to insert multiple rows into a table. There are other places that could also benefit from this enhancement. I’ve already written a patch, but haven’t timed it on large numbers of links yet.

64

And also in my case I didn’t have links. It was transcribed audio. There was about 2-3 A4 pages of text I think. And I had the troubles not on saving but on letters typing/removing. The page was broken on each small change. It was Gramps 5.1.5. I dont know if this is still repro. All my audio transcribations are in text-files for now. But I already have middle-size notes with to-do links. In the future I would not like move them to files. Files are not a proper place for these to-do-s.

65

That seems overkill for what I am currently doing.

I did some more testing with multiple steps:
When I copied and pasting the note content in to a new tree the time to close became 1-2s instead of 10s. When I went back to the slow note it was somehow faster than it was before.

I think I figured something out after connecting dots with my use Notes Gramplet in notes view:

Plenty of the slowness I have experienced seem to be because I also use the Notes Gramplet in Note view. When that Gramplet is selected when I close the note, it is very slow, but when I have another Gramplet selected instead, for example filter, its back to closing in 1-2s.

I tested this because when selecting the note with that gramplet selected, it was slow to get selected and preview to show in that gramplet. But I didnt think there may be a connection between the two before now when I noticed it was faster when a different one were selected.

I find it kind of weird that one is slow to show its preview, as just opening the note, is fast.

Hope I didn’t cause anyone to do spend time looking for code that didnt matter.

EDIT: I rewrote this post since I posted it first.

66

But, SQL can utilize Views in addition to indexes, so a query can be executed for that view inside the DB engine instead of using wildcard queries against multiple tables or indexes, or trying to use a multi-wildcard query in the first place.

A View can also be used on a JSON enabled table, selecting only parts of the JSON serialized data and combine it with data from other tables?
…and the Python scripts in Gramps could utilize those Views instead of creating new tables?
… the views could give subsets from a query, done in the database engine, while the Python scripts/filters running RegEx could query those subsets of data and run the RegEx filters and rules on that subset

67

By all means, you may not need this, but it was a tip on how you and others could get Gramps to work a bit faster and at the same time have good control over all your research notes and even expand the notes with graphic elements such as diagrams and graphs or a map of the area you are researching.

You don’t necessarily have to use all the functions in a markdown editor/notebook even though there are many addons…

My thought was really just that if you had so much text in a note that it took maybe up to half a minute every time you opened and closed that note, then in the long run it would be significantly faster to use an external notebook, at least until you were done with the research task you were working on, and since it is easy to link these external notes to Gramps, you will be able to have full control over them and you will be able to continue further research based on these notes by simply opening the note files.
This applies of course regardless of which program or solution you choose, as long as the external program is file-based.
It will of course also work if the program uses a database and provides URL-links/software-links to the internal notes/documents, but then some of the purpose of having it externally is gone, I think.

In addition, in most of these MD notebooks you will also be able to use internal wiki-links between different notes and in this way find new relationships and links to objects you are researching, be it relationships between people in the form of colleagues, witnesses, friends, upcoming family members, places where people have lived, events they may or may not have been involved in etc., which is almost impossible to find in a table structure…
(This is just a tip based on my own experience where I have found some relationships between sailors, I would never have seen without a network graph).

I have no problem understanding that not everyone works with or wants to work with data/information that is outside lineage-linked genealogy, so what I wrote about using Obsidian and linking to MD text files for research projects was only meant to be able to increase the speed of what you are doing, possibly until there has been a solution to what now seems to have become a kind of bottleneck in many cases.

It is easy to copy the entire text in an MD note, even the formatted text via the built in view in most of this software, over to a Gramps Note when you are done with the research you are doing to be able to write it out to any reports etc. or even create a pdf document of this.

Another thing with e.g., Obsidian or Foam, is that there exist addons for transcribing from video and sound to text or PDF extraction etc., something that can be of great help for genealogists as well as for other researchers/scholars.

It was only a tip to get things to work a little bit faster.

68

SQLite actually has a REGEXP operator that can be used in a similar way to LIKE.

69

I was just thinking about not pushing too much into the database engine if at some point in time Gramps also would adapt to other types of database-backends.

70

Yes. We may wish to explore other types of database backends in the future. Some of the new hybrid models look interesting.

71

One more problem is discussed here: Find duplicates feature is too slow - #6 by RenTheRoot

74

I just ran into a Gramplets bog down yesterday.

Every time the OK button was pressed when adding a Person or Family to my Tree, Gramps would stop responding. Not for seconds, but for more than a minute. The “Gramps is not responding” dialog would appear.

image

Restarting did not help. So I decided to try clearing the Gramplet overhead. I was not in “Troubleshooting” mode but in “Productivity” mode. That meant a diagnostic workflow (clearing Gramplets one-by-one [or even splitbars of view-by-view], restarting and testing an “Add”) was not an option. Even filtering through the Gramps User Directory’s gramps52 subdirectory for sidebar .ini files and deleting them was a surprising amount of effort.

But clearing the Gramplet burden also cleared the stoppage.

So I need a process for isolating the bad actor. @SNoiraud created 2 variant for Freezing signals (or freezing them from responding) to the splitbars. Maybe these could be used for isolation?

  • FR10850 V1: Refresh button in the gramplet bar. PR#698
  • FR10850 V2: freeze/unfreeze button per gramplet PR#699
GRAMPS: 5.2.1 
Python: 3.11.8 
BSDDB: not found 
sqlite: 3.42.0 (2.6.0)
LANG: en_US.UTF-8
OS: Linux
Distribution: 6.5.12-100.fc37.x86_64

Individuals
  Number of individuals: 50012
  Males: 25889
  Females: 23995
  Individuals with other gender: 0
  Individuals with unknown gender: 128
  Incomplete names: 121
  Individuals missing birth dates: 9561
  Disconnected individuals: 292
  Unique surnames: 21950
  Individuals with media objects: 25
Number of families: 18567
Number of events: 100352
Number of places: 4239
Number of sources: 223
Number of citations: 9793
Number of repositories: 14
Media Objects
  Number of unique media objects: 37
  Total size of media objects: 27 MB
  Missing Media Objects
    Users/DistrictSupport/AppData/Roaming/gramps/Gramps51.log
    Users/DistrictSupport/AppData/Roaming/gramps/gramps51/new_addons.txt
    ../gramps.ini
    ../gramps.ini
    ../gramps.ini
    C:/Users/DistrictSupport/Downloads/MapSeries_LayoutManager_Pro24.zip
    RobertAllinghamSr.jpg
    Robert Allingham Sr..JPG
Number of notes: 4663
75

Did you check what’s in the bottom bar? Mine is configured to show the ID and the name of the selected person, but it can be configured to show the relationship between that and the home person. And that can be quite slow.

76

The settings for the Status Bar were not changed when I cleared the splitbar ini files. But the response time returned to instantaneous for Adding a Person/Family.

It was (and continues to be) set to “Relationship to Home Person”. So, while that might contribute a fraction of a second to the refresh overhead, it does not seem to be the big offender.

77

H’m, weird. I’d expect it to be a problem, even in a tree much smaller than yours, and always set it to ID and name. And here, big delays often come from the Deep Connections Gramplet, or the ones showing ancestors or descendants of the selected person. All three only affect things when they’re on the foreground.

Can you list the Gramplets that you have, in either panel?

78

Unfortunately, no. I flushed the splitbar .ini files.
We may need a diagnostic tool to list all gramplets in all view modes before flushing them.

Consider that there 12 built-in views (33 view modes with 2 gramplet splitbars) in addition to the Dashboard. That makes 67 .ini files before add-on view modes.

It is beyond the possibility of troubleshooting effectively.

79

I don’t really get this, because I’ve always thought that the only active Gramplets are the ones in the foreground in any panel. And since most categories have 2 panels, one at the right, and one at the bottom, I would expect that you don’t need to look further than those, in the category that you work in.

Am I missing something here? And if yes, what is it?

80

Some progress

81

Is that 640k GEDCOM available? Can you point me to it?

82

And this one’s for all. I did an import of this, after merging it with my ancestors, in RootsMagic, and that import was about half way after 3 hours, on a PC with an i7, 8 GB RAM, and a 2 TB HDD, running on LMDE 6. I left the machine on for the night, and if the progress was linear, a wild guess would be that the total import needed 6 hours. It was complete when I got up, so it was definitely less than 10 hours.

The import of the original, which was once exported from PAF, into RootsMagic, takes about 5 minutes, running in Wine on the same hardware. And it’s even faster in PAF, even when I run that on an old Toshiba Satellite Pro 470 CDT, with a Pentium III and 64 MB RAM, running Windows 2000.

83

While working with Claude on a Phonetic grouping fork of the v.6.0.8 Grouped People view mode, we found a big performance improvement opportunity. Here is Claude’s summary:

Expand all Nodes in grouped person views: main loop stall from quadratic do_get_path()

Symptom. On a tree with 61,052 people in 8,654 surname groups, View > Expand all Nodes blocks the GTK main loop for 36–40 s, in both the core Grouped People view (39.4 s measured) and a derived addon view. The application stops answering the compositor’s ping, so after about 5 s the desktop shows “Gramps is not responding” with Force Quit / Wait. There is no progress feedback or way to cancel, and a user who chooses Force Quit loses the session.

Diagnosis. A debug watchdog made the stall visible. A GLib.timeout_add() callback records a heartbeat every 250 ms. A daemon thread checks it, and when the main loop has not beaten for 2 s, it prints the main thread’s Python stack (sys._current_frames() plus traceback.format_stack()), then repeats every 5 s until the stall ends. Every sample during Expand all landed in the same place: Gtk.TreeView.expand_all() → TreeBaseModel.do_get_path(), in its sibling walk (index += 1, get_cached_path()).

Cause. expand_all() asks the model for the path of every row it opens. TreeBaseModel.do_get_path() (gramps/gui/views/treemodels/treebasemodel.py) does not store row positions. It walks backward through node.prev until it reaches a sibling whose path is in lru_path, an LRU of BaseModel._CACHE_SIZE entries. During expand_all(), the paths of a large group’s children push the previous group’s path out of the cache. Each top-level group then walks back to index 0, so the total cost is O(n²) in the number of groups.

Fix.

  1. O(1) paths. The model subclass overrides do_get_path() and resolves each level from a per-parent dict {child_node_id: index}, built lazily from parent.children. The dict is dropped whenever the core drops its path cache (overrides of clear_path_cache() and clear_cache()). It is also rebuilt if the children list object or its length has changed. Reverse sort order is handled as len - 1 - index. Any unexpected state falls back to the core method. One caveat: PyGObject installed this vfunc override where core PersonTreeModel instances also reach it, so the override first checks isinstance() and passes other models straight to TreeBaseModel.do_get_path().
  2. No unbounded stall. open_all_nodes() replaces the single expand_all() call with one expand_row() per group. After 2 s, a ProgressMeter with a Cancel button appears. Its step() processes pending GTK events, which keeps the main loop answering the compositor, so “Not responding” never appears even on a slower machine or a larger tree. The loop stops if the model is replaced while events run. Groups on screen are opened first, and the original top row is restored afterward.

Result. On the same tree, Expand all Nodes takes 0.93–1.01 s with both fixes, against 36–39 s (with the “Not responding” dialog) in the core Grouped People view in the same session. With fix 1 alone, the run is short enough that the dialog never appears. Fix 2 keeps the window responsive regardless of tree size. Fix 1 would apply to core TreeBaseModel and so to every grouped view, including Grouped People.

84

The 0.2.0 version with that fixed fork (the Fuzzy Grouping people category view mode) is posted to my experimental GitHub repo.
Cross-compatible for 5.2+ and now compliant with the Gramps Agents.md guidance.

With this addon view mode, you can directly compare the performance of the original “Expand all Nodes” to the patched version.

If you cancel the initial indexing (or choose “none” phonetic encoding in the Configure options) for the grouping, then it behaves like the built-in Grouped People view mode… except for the Expand All Nodes being quick and having the Grouping Nodes sub-sorted by first name and with tie-breaking by birth date.

It integrates with the other phonetics options. So, if Flat Filter and Fuzzy Matching gramplets addons are installed, the gramplet selectors to change the fuzzy (phonetic encoding) system simultaneously for all 3. The “Restore to default gramplets” for the view mode with include those gramplets – provided the addons are registered and not hidden/deactivated. And the view mode recognizes the Fuzzy Rules addons for expanding those pop-up menus.