How to write up a performance enhancement request?

Help
1

While adding a new Tag to a large number (some 5,700 records) of Citations, I noted that the Citation view was refreshing the list and the Citation record noted in the Status bar after EACH addition.

It was processing no more than 3 records per second. But even at that speed, adding all the Tags calculates to take about 32 minutes to complete.

It seems likely the feedback updates are taking a LOT more time than the database update.

When processing a lot of records in another application, I had created a percentage progress bar that updated after every record. And it was processing hundreds of thousands of records. Just cutting that to only update every percent (integer of the total selected record count divided by 100) of the records massively improved the performance. I added a variable and a calculation but that overhead was insignificant compared to a refresh. In that application, the process went from 12 minutes to less than 15 seconds. And efficiency improved without reducing feedback quality.

Is there a suggestion for how to write up an Enhancement Request to do something similar for Mass Deletes and Tags?

2

How do you mean how? When I read this, the scenario is quite clear, at least to me.

Is this the flat citation view? I know that we had lots of speed problems when we migrated to GTK 3, but those mainly appeared in the person tree view, in which GTK 3 triggered way more callbacks than what we were used to with GTK 2. And I think that Doug Blank fixed that by caching a lot of person data.

Background info: When you scroll the tree view, what I would expect is that each time that new rows become visible, GTK triggers a callback to retrieve data for those new rows. In reality however, with GTK 3, I saw callbacks for all rows from #1 to the rows that showed up as new, meaning that when you scrolled down, things became slower and slower. This was not so with GTK 2, where only new rows were retrieved from the database. I know that, because I traced this with print statements, slowing things even further in GTK 3 (of course).

When I reported this to the gnome team, their only response was denial, which is why Doug fixed this in Gramps.

IMO: All you need to do is, to describe the scenarios where this happens.

3

Cool. It seemed liked it was not clear. And there is a real potential that I misinterpreted what is happening.

It was the default Citation view (tree, not flat)

I’m not sure how to categorize or title it either.

4

I didn’t try this on Windows yet, but with 5.1.5 on Linux Mint, I see that adding tags to my citations, also about 5700, takes about a minute, not much more. And I see no difference between doing that in the citation category, which is flat, or in the source/citation tree view, which is an option in the source category, not the citation.

Expanding the tree view and then selecting all, and delete, leads to an error here, so I can’t reproduce that for sources and citations. It is slow in the person tree view though.

5

Maybe that is a different between systems too? I am using a more-than-a-decade-old HP Pavilion g7 laptop with a HDD. Maybe you’re using something more powerful with a SSD?

When I was in software development, we had a rotating low-spec machine that each developer had to use for a couple days each month instead of their top-of-the-line systems… just to remind them of the pain entry-level users suffered with our product. (Although it also gave IT time to do regular maintenance on the powerhouse machines.) Agony in slow motion encourages optimization.

6

There is a difference indeed, because mine is a desktop PC with an i7, 8 GB RAM, and also a real hard disk. It’s almost 10 years old now, so that’s close to yours, but it may still be a couple of times faster.

I don’t see any updates in the status bar when setting tags though. It is only updated when I delete things, and in those cases, I also see less than a handful of deletions per second.

Are you running Windoze on it?

7

Yes, Windoze 10 Home with 2 screens. I had the statusbar Display preference set to show Relationship to home person. But that probably did not affect things.

However, there was a custom filter looking for records with less than 1 reference. That probably affected things a bit.

Device name Pavilion g7
Processor AMD A6-3420M APU with Radeon™ HD Graphics 1.50 GHz
Installed RAM 8.00 GB (7.48 GB usable)
Device ID 3090039F-31D8-45BA-A86B-92CCE2DAD509
Product ID 00326-10000-00000-AA457
System type 64-bit operating system, x64-based processor
8

H’m, I have a similar filter to select unused citations, and that makes things slower indeed.

In this case, I assume that you get the usual dialog that asks whether you want to confirm each delete, and I think that it makes sense that you suggest an enhancement that switches off filter evaluation when such a bulk delete is running.

My statusbar settings are the default ones.

9

Mass Tagging & Deleting are a couple of the many long operations that aren’t abortable/interuptible in Gramps. Essentially, the entire GUI becomes inaccessible while the operation is in progress.

So maybe the whole GUI should be dimmed except that dialog & only a progress-bar within the dialog should be updated?

(One of the lovely benefits of the FilterParams addon by @kku is that the Test Runs can be aborted. That’s in contrast to the Filter Test of the built-in Filter Editor. If that test falls into a loop or looks like it will take hours, your only other alternative is to kill the whole Gramps application.)

10

I just ran a few tests on my Windows laptop, with an i5, 8 GB RAM, and a hard disk that is much slower than on my desktop. I did that to make sure that I understand what you wrote, and some of that suprises me, so I’m asking whether we’re discussing the same thing.

Firstly, when I do a mass tagging of citations, I see a small window with a progress bar, which runs to 100 % in less than a minute, and then stays on screen for another 10 seconds or so, while Gramps is still frozen. Windows will then also tell that the application is not responding. I see the same when I tag all persons in my tree, about 10,000, where it takes a little longer for the progress bar to reach 100 %, so there is a progress bar there too, and your message suggest that it’s not there, because otherwise you wouldn’t suggest it, right?

Anyway, in this scenario, the mass tagging of persons has the same flaw as for citations, but the effect is much more annoying, because when the progress bar has reached 100 %, it stays there for a few minutes, and Gramps really acts like a zombie. I can’t close the progress dialog, so I just have to wait, and pray, hoping that the process will end some time.

Secondly, when I want to delete all persons (deleting citations doesn’t work here), my laptop is even slower than yours, deleting about 2 persons per second. Maybe that is because the hard disk is slower, or the laptop’s display (intel HD 4000), but in either case, I don’t think it’s acceptable, so it’s definitely worth an enhancement request.

I have more of these, because most of our competitors are way faster when calculating relationships, which is a function that I often use. And when I say way faster, I mean like 10 or a 100 times faster, or even more.

11

Thanks @ennoborg
Submitted: 2022-05-27 13:14
0012627: Mass Deletes or Tags bog down with excessive View content refreshes

12

I saw that, and I still can’t reproduce the slow tagging here, not even with a filter on citations. My earlier experiments were without filter, and I only see the view being refreshed during deletes. Am I missing something?

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.