Nightmare corruption recovery

Yesterday was a nightmare of a day, recovering a Tree with Gramps. And it might be because it is still a day before All Hallows’ eve, but I’m not confident that I’ve woken from that nightmare,

Note: This is not my main tree. Hopefully, it will be a future downloadable Tree for exploring some real-world data in Gramps.)

In the discussion from Can't export GEDCOM, can't import Backup .gpkg into new Tree: @ennoborg and @PLegoux identified a corrupt handle that was why Gramps crapped out exporting a Tree as a GEDCOM and created unusable Backup files.

So I double unzipped (once for the .gpkg, again for the .gramps) the XML file and then stripped the following offending tag out of George Herbert Walker Bush’s Person. (The NULL handled hlink tag was originally generated by a beta version of the Photo Tagging Gramplet.) That single fix allowed the Backup to load and the GEDCOM to work.

      <objref hlink="_">
       <region corner1_x="87" corner1_y="66" corner2_x="99" corner2_y="75"/>
      </objref>

The bad data had been cut out and the Tree was functional again… but it just seemed that way on the surface. The cancer went deeper.

Opening the Families view froze Gramps for a couple minutes. And opening the People Editor for G.H.W.Bush just spawned a 1 inch square blank window. The Relationships view was blank when George was the Active Person. However, the bottombar Gramplets showed data.

George’s family was corrupt, but how? (So many satirical political wisecracks clamor to be posted) It was a freshly imported XML file. Shouldn’t the Tree structure have been reconstructed and validated? But his family was still bad and Gramps gave no clue about the nature of the corruption.

Running the Check and Repair Database flagged some minor repairs. So I exported the Repaired Tree, make a new backup and imported again.

The same problems persisted. Gramps choked on George’s family.

Expanding the newly imported backup .gpkg created a file to inspect. But the 3MB XML file is a lot to swallow. Narrowing the inspection to JUST George did not reveal anything obvious. But even a minimal Person Profile spiders off to a multitude of secondary objects.

So I decided to treat George like a cancer. I export the corrupt individual off to a separate tree. That clone could be cleaned in detail.

(The clone had some empty families that were problematic to browse. The media chunk and any objref tags were eradication targets since Photo Tagging caused the cancer. <bookmarks> & <tags> chunks… gone. As were the tagref (tag reference) statements in the Person chunk. The cleaned clone passed an import and browse test in a blank tree.)

Once the clone was validated, the cancer-ridden Tree began treatment. Maybe the cancer was operable. Excising George seemed to make the views and George’s immediate relatives browsable again.

Remove Unused Objects… (twice) allowed interactive excision of George’s orphaned secondary objects (Citations, Events, Notes) and then their orphaned secondary objects (Citations). I made the mistake of leaving the orphaned Places, Notes markup for links and a Photo tagging rectangle.

After transplanting the cleaned clone via import, the families were stitched back together. Then the duplicates were tracked down and merged. The tree is in remission.

Note: My first transplant failed. The view problems recurred. The initial clone cleaning missed families. So the families chunk was lopped off in the XML before the next attempt. The next Transplant was more successful. But where the transplant would have created duplicates (Families, Notes, Places, Repositories, Sources, Citations, Media objects), the second transplant meant there were triplicates to be merged. I don’t know WHY the Places, Repositories, Sources or Media objects duplicated. They were identical Handles, IDs and content. I expected duplicate People, Notes, andCitations.

Looking back, I should have flushed the Person handle from George’s clone XML before importing. It would have been better if that object had been seen as a foreign and different person during import. The weird way he automatically spliced in among his parentless siblings while his parents had a childless marriage points to one (or both) of those family units being malignant. Maybe the George’s childref handle had not been excised when George was originally deleted?

The structure seems to be fixed now. But the Tree looked okay until I tried to open the Person Editor and Relationship view for George too.

Where else will this US Presidential History lineage be riddled with cancerous corruption? (Again, too many obvious mockery targets there. Maybe I just need to strip out the Living ex-presidents? The mere proximity of Tᴙump may be twisting reality.)

1 Like

Note:
The last of the Places has been cleaned and the only major data defects that remains are the Citation Vol/Page values that been saved as Person “REFN” attributes.

After fixing that, this data might be exported as a GEDCOM7 sample for inclusion 5.2 version.

Dunno whether the face detection metadata will translate to the ZIPped GEDCOM7 format.

When I repeat your actions in Linux, as far as I can understand them …, I see nothing like the problems that you reported here. That’s with a slightly modified 5.1.5, built from source, and I was lucky that I could recover your tree from an automated backup on exit, since I had already removed it, thinking that everything was OK.

This means that I have a backup in .gramps format, with all 4 media references still there. Switching to the families view is fast, also when I select papa Bush. I can also see that two media entries are removed when I choose to Remove Unused Objects.

I have not tried the same in Windows yet, but I can do that if you want. And I can also mail that .gramps backup if you want.

1 Like

A backup made by an admirer of Saddam Hussein hating G. W. Bush? :rofl:

This is a fresh re-install of Gramps core but with old add-ons and .ini files… It is likely that my laptop system is headed for a thermal failure. (The bearings on the fan have been complaining for a year and it is nearly a decade old.) So anything is possible.

The unloadable Backup file is attached to the bug report. I stripped the bad statement and imported the patched backup into new tree repeatedly with the exact same results:
George Sr. (and ONLY his profile) spawned the Person Editor on my primary screen in that 1 inch square blank window. Gramps was on my 2ndry screen and all other Person Editors opened there.

It could be a Windoze thing.

OK, I tried it on two Windows installations, 10 Home running in a VM inside Linux, with only 4 GB RAM, and 10 Home right on the same hardware (dual boot) with 8 GB RAM. No errors whatsoever. CPU is an i7.

1 Like

The only truly important one to replicate is: use Notepad++ to remove the 3 objref lines related the Null handle from the XML file. Then import and see if George Sr’s Person Editor misbehaves.

All the other steps are based on working around that problem.

I used Notepad++ to delete the 3 objref lines. Both Georges seem to behave appropriately in the Person Editor.

Edition: Windows 10 Pro
Version: 21H2
OS build: 19044.2130

Gramps is
GRAMPS: AIO64-5.1.5-1
Python: 3.6.4 (default, Jan 23 2018, 13:17:37) …
BSDDB: 6.1.0 (6, 0, 30)
sqlite: 3.21.0 (2.6.0)
LANG: en_GB.UTF-8
OS: Windows

Ok. That sends me back to add-ons. And likely to the thumbnails.

The Person Editor decides how to scale its dialog size by considering the thumbnail dimensions. Maybe bad data caused an error there? Either in generating the thumbnail or when finding the thumbnail. Maybe null dimensions or bad load just caused the Person Editor to abort building content?

That might explain the problem persisting. It could have persisted until something changed that forced regenerating a valid thumbnail … or a valid handle to a thumbnail.

1 Like

If that’s the case, you might be able to see some corruption in the thumbnail cache, and maybe even by seeing that Windows has problems creating thumbnails of thumbnails.

You might even add a part of that cache to the bug report, so that we can see how it might infect our setups. And then you can also find out whether clearing that cache helps.

I tried your test tree on my laptop too, which is also about 10 years old, a pre-owned Fujitsu, and I see no way to break it.

1 Like

I had removed the Media Objects from the SQLite Tree in one of my attempts to workaround the problem.

The Backup .gpkg won’t load so I pasted just the Media Object chunk back in via the Import Text gramplet. Of course, that did not restore the Rectangles to the Tree as they are linked from the other direction.

(And the error for George Sr. is not quite the same. Removing the Media Objects cleared the Rectangles out of the 44 other presidents but left George alone because it choked on the problem handle. So when I added back the Media Objects and they came back without the other rectangles, the old index would’ve been up around 40… with nothing earlier. So changing from the Media view to the Person view is giving the following error:

180093: ERROR: grampsapp.py: line 174: Unhandled exception
Traceback (most recent call last):
  File "C:\Program Files\GrampsAIO64-5.1.5\gramps\plugins\sidebar\dropdownsidebar.py", line 119, in __category_clicked
    self.viewmanager.goto_page(cat_num, None)
  File "C:\Program Files\GrampsAIO64-5.1.5\gramps\gui\viewmanager.py", line 789, in goto_page
    return self.pages[page_num]
IndexError: list index out of range

)

Still, that SQLite Tree still exhibits the problem when trying to open the George Sr. Person Editor. So that has be been ZIPped and added as another Note to the bug report.

Maybe the People Editor source could be changed to not choke on the missing handle for the rectangle?

Reminder:

This tree data HAS been recovered. So this thread is NOT about recovering this particular tree.

Instead, it is about the possibility of:

  • Expanding the Check and Repair Database tool to recognize and fix another type of corruption.
  • Changing the Import failure feedback to identify the bad line# or chunk so manual recovery has a target. (It doesn’t need to fix a problem… just also identify where the failure occurred in the data instead of just the import tool code.)
  • Changing the (GEDCOM) Export failure feedback to Identify the bad chunk and prompt the user to try database repair
  • Fixing Photo Tagging gramplet to not create the condition
1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.