[Photo Tagging gramplet] Gramps resists being a Photo detective

I was wistfully excited to receive a pair of old photos yesterday. An aunt just passed and her daughter shared a couple of treasured images.

But we were having to guess at the children and the year of one photo. So I wondered if Gramps could be used to eliminate possibilities.

It helped… but not willingly nor in a forthright manner. Gramps seemed to fight using its data in a forensic manner. It was really only useful in validating my theories… not in positing possibilities.


Here's the experiment with the "knowns" and "unknowns"

The photo had 7 people: 4 daughters or daughters-in-law of my paternal grandfather and 3 of his grandchildren. 3 adults were were known, a decoration set the photo around Christmastime at a particular home, and the grandchildren included a female of 5-8 years & 2 babies (one about 2 months & the other about 9 months). The remaining adult could only be 2 persons.

Photo Tagging seemed the obvious tool

Since I’m using the Windows version, the Photo Tagging gramplet doesn’t have the face recognition feature active. So the process started with Adding the photo (and setting a between Date span) as a Media object and manually marking face with rectangular marquees.

Photo Tagging was painful

Three of the 7 rectangles could immediately be associated with the 3 known persons.

Setting the associations this was a familiar task, nearly identical to sharing a known person in the Tree into another family. It used the Select Person dialog. This was the first problem. The selection dialog for an existing person in a 45,000 person hierarchical tree is agonizingly unresponsive.

There is no Search for the People Selection dialog. So it is necessary to either scroll or use the find-as-you-type. And the second feature, ‘Find-as-you-type’, has a timeout that tends to reset itself before finding anything in this large a set of records. So it averaged 3 times re-typing the same search term before Gramps actually scrolled to the record. Scrolling wasn’t any picnic either. The scrollbar graduation for a Surname cloud of 18,680 unique groups is still fairly uncontrollable.

Then a discovery… Although not in the documentation, the contributor of the tagging Gramplet fortunately included drag’n’drop support. ( big “thank you” is owed to Artem Glebov!) It is a bit of an odd implementation. You have to select the Preview row before dragging a Person to it. Even if you drop directly on another row, it changes the selected row. And if you mistakenly drag to the wrong thumbnailed row, you cannot drag to the right person and then fix the other. Because doing so would create a duplicate and the Gramplet demands that the wrong Marquee be removed. You cannot just make it undefined again. So lots of awkward caveats.

So, after detaching the Gramplet to allow flipping between Views with different researching options, I could go to the People view, filter to my paternal grandfather’s grandchildren, offspring and spouses of offspring. Then do drag’n’drop experiments to see who fit the photo’s “unidentified.”

Another useful feature was the experimental Profile Relationship view mode (by @cdhorn ) with a the Photo Tag gramplet to do sanity checks as to whether the the identifications are fit. Christopher is going to add a ‘age at the time’ calculations. That will be helpful since the Photo Tag only calculates how old the Persons was at time of death or are today. This new feature will help when trying to reverse-engineer a date for the Media Object.

Conclusions

All in all, I wasn’t able to use Gramps efficiently as a forensic tool. Biography data was too well was too segmented into information silos. Since this was a very simple case, it was a good test run.

The process of adding identifications was pretty painful. It seems like after doing one identification, the rest of the people are likely to be close associates. In my test case, the associations were well defined. But I still had to start winnowing down from the Whole Tree every time I wanted to define a new Rectangular Marquee. Even just trying drag’n’drop from immediate family using the Relationships view modes kept changing the Active Person focus. I had to fallback to the People view and apply a filter.

People naturally have a close group of contacts: those with an “Association” and immediate family: offspring, parents, siblings, niblings, in-laws, grandparents, grandchildren. And that is going to be the common selections when adding hyperlinks to to Notes too. Because the filter for these people is slow due to complexity (and kept re-filtering when data changed or was sorted) it was more efficient to copy them to the Clipboard.

(Since you cannot drag’n’drop an extended selection from a View directly to a Collections Clipboard, you can clear the Gramps main clipboard for temporary use by offloading its contents to a Collections Clipboard. The drag’n’drop of extended selections will work between clipboards.)

So I’m looking for suggestions on a better workflow. When I actually NEED to use Gramps forensically it would be nice to be able to do so.

I wish I could help but I had a similar experience. I have my g-grandfather’s photo album and other collections of photos. One in particular seems like I ought to be able to decipher it, but alas. The photo is a young man in Canadian Army Expedition uniform (WWI) and the notation on the back refers to “Aunt Lou*”. I know who Aunt Lou was and I have a pretty good idea of the age of the man in question (and thus his birth date) and that he was alive somewhere between 1914 and 1918. WWI Canadian military records are available on the internet. I found it difficult to get a decent list of candidates out of Gramps. Maybe if I knew more details about filters it would have all become clear. It seems like it should not be that hard.

Craig
*It is possible he called Lou his aunt but the relationship was something else. Say Great Aunt or second cousin once removed, etc.

I yet uses Picasa (it’s not distributed anymore but still working fine - distribution on my dropbox) to scan my pictures. It regroups them to make me naming the people. If I don’t know I name that person X1, X2, … and when it find it again it display it in the good (theoretically) person directory. Then I create some John Doe persons in Gramps with pictures found by Picasa about them.

Edit:
And, I don’t know if it’s your case, on some pictures I’ve the photograph name and address, so I try some filters to find persons I don’t know. I’ve wrote a little tutorial on this, if it can help.

I installed Picasa yesterday and applied names to all my modern family photos… just to see how it worked. It definitely became ‘smarter’ with more data points.

Unfortunately, Picasa saves the Face data as XMP metadata and Gramps only has Exif Metadata tools.

Since it is my standard test file, I added Picasa ID’s & face rectangles for this 1913 Family reunion JPEG picture.

(The picture was originally laid out for a Cascading Style Sheet toy. Where the image lived in a webpage frame that was half the width of the image. So it normally showed only the left half. But on a mouse hover, the frame origin slide to the right and showed just the right side.)

Well, THAT experiment was an excellent example of why you should not trust cloud services.

I understood from the start that Google had migrated from Picasa to Google Photos in 2016. But Google Photos didn’t recognize the XMP Face metadata created by Picasa! The only metadata it found compatible was the Caption.

It was expected that the IDs wouldn’t survive.

It did not not even recognize the Face rectangles existed.

I wonder if that is due to the linkrot in the DTD URIs? A number of the domains referenced for the Data Type schemas no longer exist. Does this cause the XML readers to crap out?

If so, is this a vulnerability in the Gramps XML format too?

In 50 years, will our XML files be uninterpretable because the domain for the RELAX NG schema is has suffered linkrot or was clobbered?

Try digiKam or Darktable with the Lua Script for face recognition (in official lua repo i think)

There is a script for importing Picasa FR tags to digiKam

or you can use this script to import to digiKam directly from Picasa ini files in each folder…

There is also multiple Python Scripts for Face Recognition, but I do not know who is best or simplest to run…

And since those scripts is Python someone with the knowledge should be able to convert it to a “Gramps Script” or Gramplet…

Maybe it also can help doing the same from a digiKam database to Gramps…

I have revised my process a little for Family photo tagging.

By using the Test run results from @kku’s FilterParams Isotammi add-on tool, it becomes a bit faster to link immediate relatives.

After creating a custom filter using @Mattkmmr’s manually installed and tweaked from 5.1 version People Separated less than <N> degrees of <Home person> (GitHub repository and wiki doc)

After installing the add-on rule, create the following rule:

Or use @kku’s FilterParams Isotammi add-on tool to (clipboard) import the Custom Filter.

<filters>
  <object type="Person">
    <filter name="2 degrees of Home" function="and" comment="download ZIP of 5.1 rule from Matt Kemmer at https://gist.github.com/Mattkmmr/4d0369cbee38bdf14395cb2f4505fb8a">
      <rule class="DegreesOfSeparationHome" use_regex="False" use_case="False">
        <arg value="2"/>
        <arg value="1"/>
        <arg value="1"/>
      </rule>
    </filter>
  </object>
</filters>

In the Media category, make the group photo into the active media object, add the Photo Tagging gramplet to the side bar, then detach the gramplet. (By dragging the gramplet’s title tab up to the Gramps menu bar)

In the People category, Sort the Flat List view mode by Birth Date, set the Home Person to be someone known to be in the Family photo. Then apply the “2 degrees of Home” custom filter.

The filter will show:

  • the Home Person
  • their parents
  • their siblings and sibling-in-laws (only from their side, but not their spouse’s)
  • niblings (only from their side, but not their spouse’s)
  • Spouses
  • Children and spouses
  • grandchildren and spouses

Now you can select rectangle for the people in the Photo Tagging, then drag’n’drop the person from the Person view.

(Hint: when about to do a bunch of drag’n’dropsm it helps to hide the Navigator side bar. It is to easy to change categories if you accidentally drag over the side bar.)

Did some tweaks to the PhotoTagging gramplet and trying to decide what to added before releasing a beta.

Added:

  • add row Re-0rdering with controls for the selected row. (Toolbar, context menu, keybinding, drag’n’drop)
  • Row number (or Gramps ID) markers for each marquee (3rd state is to hide)
  • Age as of the date of the media object (highlight ages when media date outside the lifespan… including gestation and funeral/memorial)
  • Show/Hide Marquees
  • store the re-ordering as a custom Attribute for the media object
  • editing the marquee no longer moves that row to the bottom
  • stopped a few errors and warnings.


UK Royal Family; NPG by Thomas Patrick John Anson, 5th Earl of Lichfield, December 1971 Yes, some of the linking is wrong. I’m unfamiliar with the Extended Royal Family.

I’m trying to decide what else is needed before putting it out for beta testing.

Some possibilities are:

  • Caption generator (listing the people, probably as a Note)
  • Viewer of Normal and Large thumbnails for the selected person
  • Grouping of rows : Row 1, 2, 3, 4; A, B, C; Back, Top, Middle, Bottom; 1a, 3c, 4d
  • Name display format, date format override Configuration

(Due to a lack of space, Might have to go to a 2 line format for Name/Age. The XMP Region Name column is so infrequently visible that I didn’t even realize that it was there.)

One thing that I always do before putting photos out into the world is adding the citation to them, as text, either over an inconspicuous part of the background, or on a added border. I find this combats the proliferation of random images attached to people that no one knows where they came from. Maybe you could add a “citation tagging” to it?

Would not any provenance tracking be in Citations for the media object? And be displayed in Reports?

This tagging gramplet does not modify the file being referenced… not the bitmap nor the (embedded or external sidecar) metadata. Gramps just creates its own metadata catalog layer with the Media Objects.

Thank you so much for working on the great Gramplet. I used it for a while too. I think it’s fantastic. Since the clippings unfortunately can’t be exported as gedcom, it’s just decorative for me in Gramps. But that’s a different Story.

I did not see a strong reason to go to all the work of defining the marquee clippings either. It was neat that it could create thumbnails for the people. But as @Woody pointed out, I wanted something to help with persistent captioning of photos in my collection. And that could be output somehow.

So this 1.1.2 beta version has the ability to scrape the List and metadata labels into a “Caption” type Note. This creates a scratchpad as a jumpstart towards a proper caption. It can also be used as a worklist for finding (or creating) people to match the metadata.

This first version uses a custom attribute as a container for some of that info. But a sidecar (external or embedded) seems like a more standard opportunity to be explored. Then external programs could leverage the identification data too.

The beta also bundles some sample images with various Metadata format data for experimenting.
(in /PhotoTaggingGramplet/exif-orientation-vs-face-regions-main/images/)

This is definitely still in beta. (Testing in Gramps 5.2 and 6.0 on Fedorra) If it checks out, then a release will be submitted to the official Gramps addons-source repo.

Beta available for Addon Manager at:

  • https://raw.githubusercontent.com/emyoulation/CuratedGrampsPlugins/main/gramps52
  • https://raw.githubusercontent.com/emyoulation/CuratedGrampsPlugins/main/gramps60
  • https://raw.githubusercontent.com/emyoulation/CuratedGrampsPlugins/main/gramps61

Published to GitHub repo via the 1.9.6 version of Addon Pack and Ship Gramps utilities tool (generating .tgz and .json files) push to GitHub Desktop for uploading

Here’s where the gramplet stands today (v.1.2.1 available for beta testing. Please make a backup!)

Michal Maňas ( @snek01 ) suggested looking into DigiKam and metadata sidecars. For compatibility sake, we chose to experiment with a sidecar as an Attribute. (The format for the sidecar Attribute will evolve from beta test feedback. Right now it reads embedded Metadata or an external sidecar with the media object. But if there is also a Gramps sidecar attribute with conflicting info, Gramps wins. Eventually, there will be tools to export a sidecar or embed metadata. Those are risky actions and so they wait for the future.)

In the image below, the names for the face rectangles were clues provided in a companion metadata sidecar with the image. (A sidecar has the same filename with a .xmp extension. It contains XML data as plain text. There are many dialects of metadata. Besides sidecars, the gramplet reads some embedded metadata dialects.)

The Photo Tagging gramplet parsed that sidecar data and set the 3 placeholders, guessed that the names were in Given Surname format and separated that to show in the Name column with the current Name Display format.

In this example, none of the people were in my Tree database. So clicking the + button started a New Person with that Metadata clue name. Clicking the Magnifying Glass copied the name to the OS Clipboard so that I could Google without typing. (Eventually, some researching utilities will be wired into that button. That’s why it is a magnifying glass, not a “copy” icon.) After adding a birth and death information from the Google results for Lincoln, Abraham, saved the New Person dialog and the Person object was linked to that face rectangle marquee. (Linking in a similar fashion as adding Child or Partner to a Edit Family dialog.)

Now the rows showed names (in red) as placeholders for rows 1 and 3 but calculates the Age of the person at the date stored for the media object. If the Person failed a “Probably Alive” test for the data of the media object, they would have been flagged in red too.

The placeholder data is persistent. So you can switch between media objects without losing your “worklist”. Undocking the gramplet lets you use a 2nd monitor to park the Gramplet while you use other views, gramplets and tools to search for matches. Matches can be dragged to the row in the undocked Photo Tagging gramplet.

The Caption button scrapes the Media object name, date, list of Names and Metadata clues into a new (Caption type) Note.. hotlinking where it can. Or you can add a Caption note to Media object as a worklist for creating named “placeholder” marquees and rows.

Clicking on a marquee, a row, a hotlinked caption item will navigate the Active Person. All those Photo Tagging items are highlighted if they match the Active person.

When I add a media to someone, I put in the mediaRef a citation indicating what allowed me to say that the face in the photo is indeed that of the person.

Would it be among the right-hand columns to choose a citation to associate with the mediaRef of this media-person link?

And…

When you click on a name on the bottom left, does it open the person or the medium in the context of the person (media + MediaRef)? You will have understood that I would prefer the second option.

Currently, a click (or double-click) in the (lower left ) caption panel, is the same as a click on a row in the (right) list panel or within a face marquee rectangle of the (upper left) preview panel.

  • A click selects the same element in all 3 panels.
  • A double-click opens the Person Editor for the person (I have never prompted Claude to check whether it is possible to have that make the gallery tab active and the Thumbnail selected in that Person Editor. Would that be useful?)

A right-click in the List panel will show a list of other options. We can add to that list.

@cdhorn had a Configuration option in CardView that allowed the double-click action to be selectable from the context menu options. I suppose that’s doable.

Did not quite grok what you’re asking for the Media Ref.

There is no current interaction/linkage with any Citation. Not sure what you’re doing there or want.

That was a slight misuse of terms on my part.

I was referring to opening the media reference editor directly for the person in question, where you can find a citation tab:

I usually avoid the ref dialogs. But mostly because the navigation to the dialog and upwards to the referring object is so… arbitrarily argumentative.

But perhaps that could be a context menu choice? And if we add that double-click action selector, it could be made more accessible.

I use them extensively!

They offer real opportunities to describe the link between two objects—where it originates, why it exists, what additional information the connection provides or what is specific to it, and so on.

I regret that they aren’t well-represented in Gramps tools like filters or reports… or in the option to open that specific object. I really like CardView and a few other add-ons that do exactly that.

They are also poorly represented in the Gramps clipboard. They have no context menu. Context menus give clues about the potential uses of an object.

(I did a hack to add a context menu to clipboard “ref” objects. But it is a bit stunted.)