Gramps plugin concept: OCR + name index for large document collections

I have a database with about 14,000 individuals in Gramps.
I also have an archive of around 1,700 scanned documents (obituaries, PDFs, images), of which only part are already included in Gramps as sources.
I do not intend to import this entire collection wholesale into Gramps.

My idea is to develop a Gramps plugin that:

  • scans an external folder of documents,

  • applies OCR and name detection,

  • stores the results in a JSON index (document → names found),

  • and, when a person is selected in Gramps, allows checking whether that name (or variants) appears in that index.

The goal is not to replace Gramps’ source system, but to make it easier to locate documents relevant to a specific person within a large collection of scans.

I am posting here to ask whether this is viable within Gramps’ architecture or whether this is an unrealistic idea.

This sounds interesting! Having a way to cross-reference external documents … without cluttering the (non-hierarchical) Media Object category view… sound valuable.

There is a Media Manager feature (Add images not included in database) that scans a folders containing Media Objects and adds any that do not already exist as Media Objects. But nothing that helps correlates those media objects to possible matches in the Tree

Although the Merge Media addon will find references to the same media objects and merge them. (So this might be used to correlate files that have already been manually added from that particular folder.)

Gramps currently includes a built-in SoundEx gramplet and a Soundex match of People with the <name> filter rule. (I don’t see a corresponding SoundEx name field in the Gramps Data Model diagram. And there are no hits when searching for SoundEx in the Sphinx-based developer docs. And there is a gramps/gen/soundex.py)
But name matching might be more internationally compatible with Double Metaphone (There’s probably no analog for the commercial Metaphone3 in the open source realm.) Perplexity suggested the jellyfish.double_metaphone(name) (pip install jellyfish metaphone) for “for Gramps or Python/SQL fuzzy name matching” as a “Genealogy/Software Dev Fit”

Do you still look at efficient tools for that? Or have an idea for which one might be used on a Gramps plugin?

Yes, I am still looking for an addon or utility that can do this.

Furthermore, I have been working on my own project. I named it Fíos (threads in Galician) but it is only in alpha and I haven’t been able to dedicate time to it in the last few weeks.

You can find the README here:

But for the moment it is mostly a “must be” / design document and does not yet represent real functionality.

My current idea is:

  • a Tool for batch indexing (OCR + name extraction) into an internal SQLite DB, never modifying the original PENDING/ folder
  • a Person Gramplet that shows ranked candidates for the active person
  • a Media Gramplet to see the OCR text and re-index if needed

If you know of any existing tool that already does this kind of external-document → person matching, I would still be very interested.

It seems very ambitious. Great project.

In fact, I am just experimenting (pseudo debug stage) some OCR tools, most part for handwritten text recognition (HTR).

My current limitations are also on my local configuration (python 3.6 and related set of libs). i.e., I cannot use Tesseract 5.x and python 3.10+ ([1] - via google groups). Only get the python wrapper (pytesseract 1.7.2) for Tesseract 4.1.3… :frowning:

Sure, we can still run Tesseract from CLI:

$ tesseract -l fra A.JPG stdout --psm 6 -c tessedit_char_whitelist=0123456789
$ tesseract -l fra A.JPG stdout --psm 1 -c tessedit_char_whitelist=0123456789
$ tesseract -l fra A.JPG stdout --psm 1
$ tesseract -l fra A.JPG stdout --psm 6
etc.

So, I only started from scratch for running (rather testing) some of them via a basic python script (interactive CLI). As the script was growing, I also need to customize some tools like EasyOCR for a possible PyTorch 2+ support [2] and need to deal with pillow or bidi versions, even for a local TorchFreeEasyOCR fork [3].

Paddle OCR cannot be installed on my local config (as well as tesserocr, which uses python3.9+). And I did not re-test hOCR and the others set of tools (tested some years ago)

I found the HTML preview and CLI return from the script, more simple for a debug stage (thanks to vibe coding!). After some tweaking (and fixing…) on code sections and blocs, there is now, maybe a simple workflow for getting something more useful from these ocr engines under my local configuration (test).

Note, I am testing them for a large set of documents (corpus), which is most part of time images like indexes, with names and digits on columns pair (2, 4, 6, etc.). So, the design (columns) is oriented or very specific. This could also match (sometimes) on any typed sources without columns (e.g., I was able to get most characters from the technical manual of my fan machine !)

[..]
> « Les pales du ventilateur seront à 2,3 m au moins, au-dessus du * Localiser les câbles d’alimentation électrique
> plancher. dans le plafond. Avant d’accrocher le
> La circulation d’air des pales du ventilateur ne sera pas génée pardes | ventilateur ou tout autre fixation, mesurer
> murs où des piliers à moins de 0,6 m de l’extrémité des pales. les câbles d’alimentation pour s’assurer d’une
> + Le ventilateur sera installé directement sous une solive ou une longueur minimale de 150 mm dans la boîte
> entretoise qui soutiendra la boîte de sortie et le poids du ventilateur. de sortie pour la connexion avec le ventilateur. [..]

If need, you can see my last version of the script from:

sorry, comments and maybe queries are in french on the script

$ python3 script_ocr.py --help 

should give you a quick overview of my experimentations.
There is also an old (incomplete and not up-to-date) help documentation:

As wrote, I used mixed set of libs under python 3.6. This is not universal (and I do not plan to make it).

I was able to give a try on your Fios project (even under gramps 52)

Note, after my local tests, it seems that the tesseract API/ecosystem (tesserorc : Pillow-friendly, Python wrapper around tesseract-ocr API using Cython - pytesseract : a wrapper for Tesseract-OCR Engine or whatever python declinaison - as well as any python subprocess for tesseract from cli) seems to have many dependencies issues (or versions mixup).

Maybe for an “all in one” bundle (or any light packaging), and basic typed sources/media, TorchfreeEasyOCR can return expected translation for typed text with “few” (I guess more and less 300 MB by downloading a local[e] onnx model ~ 100 MB for latin and french). Sure, performances for complicated sources or any large project might be better with tesseract or PyTorch based tools (maybe like Transkribus, via PyLaia). I just think that for basic tasks (or few ressources or token via IA), it does the (local) job too.

Thank you. Your work too.

In fact, I am just experimenting (pseudo debug stage) some OCR tools, most part for handwritten text recognition (HTR).

I am experimenting too. I am working with obituaries in A4 format like this, so my work seems easier than yours with handwritten text. But this only an starting point. My intention is to be able to extract names from documents and to compare them with database in Gramps.

I am using AI as an assistant. I want to learn Python. Sometimes I use vibe coding, it will be useless to me if I can not understand the code. I used it too to discuss the requirements of the project and maybe the AI was too optimistic about the future.

I was able to give a try on your Fios project

Fios is not my first project for scanning obituaries. I had another private project that I have not published yet. So tesseract and pytesseract were legacy dependencies and the first options recommended by ChatGPT, when I used it

This is ocr_renamer. And sorry, I also code in my mom’s language. I my case, it is a mixture of Galician and Spanish. It is the vice of being born this way :stuck_out_tongue_winking_eye:

#!/usr/bin/env python3

import argparse
import subprocess
import os
import sys
import re
import curses
import glob
from curses import textpad

# Intenta importar PIL
try:
    from PIL import Image
except ImportError:
    print("""
    PIL non está instalado. Necesitarás instalalo para poder procesar as imaxes.
    Instálao en Ubuntu:
        sudo apt install python3-PIL
    """)
    sys.exit(1)

# Definición da función de reconocemento de imaxes en función do módulo dispoñible

def extraer_texto(ruta_imaxe, porcentaxe = 100, escaneo_vertical = False):
    global recordatorio

    imaxe = Image.open(ruta_imaxe)
    ancho, alto = imaxe.size

    # valorando se a imaxe é un recordatorio polo seu tamaño
    if ancho <= max_ancho and alto <= max_alto:
            recordatorio = ' recordatorio' 
    else :
        recordatorio = ''

    # calculando area a procesar para reducir:
    # - tempos de procesado
    # - xeración de opcións innecesarias

    if porcentaxe == 100 or porcentaxe == -100:
        area = False
    elif escaneo_vertical and porcentaxe > 0:
        area = (0, 0, int(ancho * porcentaxe / 100), alto)
    elif escaneo_vertical and porcentaxe < 0:
        area = (int(ancho * porcentaxe / 100), 0, ancho, alto)
    elif porcentaxe > 0:
        area = (0, 0, ancho, int(alto * porcentaxe / 100))
    elif porcentaxe < 0:
        area = (0, int(alto * porcentaxe / 100), ancho, alto)
    else:
        print('Valor de porcentaxe de imaxe non válido. Ignorado.\n')
        area = False
    if area:
        imaxe = imaxe.crop(area) 
    
    # Converter a imaxe a escala de grises
    # Mellora a lexibilidade do texto
    imaxe = imaxe.convert('L')

    # OCR
    # Selección da fonte de datos do motor 
    #'eng', 'spa', 'glg', 'glg+spa'
    idioma = lingua

    # selección do motor
    if OCR_ENGINE == 'tesserocr':
        texto = tesserocr.image_to_text(imaxe, lang=idioma)
    else:
        texto = pytesseract.image_to_string(imaxe, lang=idioma)
    return texto.strip()


def extraer_nome_e_apelidos(texto, ignorar_maiusculas = False):
# Lista de prefixos a ignorar (Don, Dona, D., Dª., etc.)
    prefixos_excluidos = ['Don', 'Dona', 'Doña', 'D\\.', 'Dª', 'Dª\\.', 'Da', 'Do', 'Sr', 'Sr.', 'Sra', 'Sra.']
    
    # Expresión regular para nomes compostos, evitando prefixos
    patron_nome = r"\b(?!(?:" + "|".join(prefixos_excluidos) + r")\b)([A-ZÁÉÍÓÚÑ][a-záéíóúñ]+(?: [A-ZÁÉÍÓÚÑ][a-záéíóúñ]+)+)"
    
    # Expresión regular para buscar un patrón de "Nome Apelido"
    # patron = r"([A-ZÁÉÍÓÚÑ][a-záéíóúñ]+(?: [A-ZÁÉÍÓÚÑ][a-záéíóúñ]+)+)"
    
    flags = re.IGNORECASE if ignorar_maiusculas else 0
    
    # Buscar todos os posibles nomes e apelidos no texto
    coincidencias = re.findall(patron_nome, texto, flags)

    # Normalizar: primeira letra en maiúscula, resto en minúscula
    coincidencias_normalizadas = [
        " ".join([p[0].upper() + p[1:].lower() if len(p) > 1 else p.upper()
                  for p in nome.split()])
        for nome in coincidencias
    ]

    # Detectar a posible aparición das palabras clave para un cabodano
    patron_cabodano = r"((?:\bprimeiro\b\s*)cabodano|(?:\bprimer\b\s*)aniversario)"
    cabodano = " cabodano" if re.search(patron_cabodano, texto, re.IGNORECASE) else ""

    # Devolvemos as coincidencias e se se detectou que é un cabodano
    return coincidencias_normalizadas, cabodano  


def mostrar_menu(stdscr):
    global recordatorio

    # Configurar codificación Unicode para evitar problemas con acentos
    curses.curs_set(1)
    stdscr.encoding = "utf-8"  # Configurar UTF-8 en `curses`

    texto_extraido = extraer_texto(camiño_completo, 50)
    if not texto_extraido:
        texto_extraido = extraer_texto(camiño_completo, -50)

    coincidencias, cabodano = extraer_nome_e_apelidos(texto_extraido)
    prefixo = 'ESQ '
    cadea = False
    
    while True:
        
        # Limpar a pantalla
        stdscr.clear()
        
        stdscr.addstr(f"Texto extraído de\n{camiño_completo}\n")
 
        # Mostrar menú
        stdscr.addstr("\nPulsa unha tecla e selecciona o nome de arquivo a usar:\n")
        opcions = []

        if not coincidencias:
            stdscr.addstr("\n\tSEN COINCIDENCIAS DETECTADAS\n\n")
        else:
            # Mostra as 3 primeiras coincidencias
            # pode haber máis, pero o nome do defunto soe estar arriba
            for i, coincidencia in enumerate(coincidencias[:3]):
                #coincidencia = coincidencia.title()
                stdscr.addstr(f"\t{i + 1}. '{prefixo}{coincidencia}{cabodano}{recordatorio}{extension}'\n")
                # Gardamos os códigos ASCII das opcións        
                opcions.append(i + ord('1'))
            stdscr.addstr("\n\tm. Escoller e editar a opción máis aproximada.\n")

        stdscr.addstr("\to. Abrir o arquivo no visor predeterminado.\n")
        stdscr.addstr("\tv. Escanear de novo o arquivo, primeiro a metade esquerda.\n")
        stdscr.addstr("\tr. Reintentar ignorando maiúsculas\n")
        stdscr.addstr("\tn. Omitir e continuar.\n")
        stdscr.addstr("\te. Saír\n")
        stdscr.refresh()

        # Esperar a pulsación dunha tecla
        tecla = stdscr.getch()

        if tecla == ord('m'):
            stdscr.addstr("Escolle a opción máis aproximada\n")
            stdscr.addstr("ou pulsa 'b' para voltar ao menú anterior:\n")
            stdscr.refresh()

            tecla = stdscr.getch()
            if tecla in opcions:
                index = tecla - ord('1')
                cadea = f'{prefixo}{coincidencias[index]}{cabodano}{recordatorio}{extension}'
                cadea = editar_texto(stdscr, cadea)  # Chamar á edición de texto
            elif tecla == ord('b'):
                continue
        
        elif tecla in opcions:
            index = tecla - ord('1')
            cadea = f'{prefixo}{coincidencias[index]}{cabodano}{recordatorio}{extension}'
            
        elif tecla == ord('r'):
            coincidencias, cabodano = extraer_nome_e_apelidos(texto_extraido, ignorar_maiusculas = True)

        elif tecla == ord('v'):
            texto_extraido = extraer_texto(camiño_completo, 50, escaneo_vertical = True)
            if not texto_extraido:
                texto_extraido = extraer_texto(camiño_completo, -50, escaneo_vertical = True)

            coincidencias, cabodano = extraer_nome_e_apelidos(texto_extraido)
            continue

        elif tecla == ord('o'):
            stdscr.addstr(f'Abrindo ruta {camiño_completo}\n')
            abrir_imaxe(camiño_completo)
            continue

        elif tecla == ord('n'):
            stdscr.clear()
            stdscr.addstr('Procesando imaxe...')
            return
        
        elif tecla == ord('e'):
            # Acción para saír
            sys.exit(0)
        else:
            # Mensaxe de erro se a tecla non é válida
            stdscr.clear()
            stdscr.addstr("Tecla non válida, por favor, selecciona unha opción válida.\n")
            stdscr.refresh()
            stdscr.getch()  # Esperar á entrada dunha tecla
        
        # validamos cadea en cada iteración
        if cadea:
            if renomear_arquivo(stdscr, cadea):
                break
        else:
            continue
    
def editar_texto(stdscr, cadea):
    # función para modificar o texto coa opción m
    stdscr.clear()
    stdscr.addstr("Modifica o nome a continuación (incluída a extensión) e preme ENTER:\n")
    stdscr.addstr("(Se queres cancelar, preme ESC)\n")
    stdscr.refresh()

    alto, ancho = 1, max(len(cadea) + 10, 70)
    win = curses.newwin(alto, ancho, 2, 2)
    win.addstr(0, 0, cadea)
    stdscr.refresh()

    box = textpad.Textbox(win)

    def validar_tecla(ch):
        if ch == 27:  # ESC
            raise KeyboardInterrupt
        return ch

    try:
        novo_texto = box.edit(validate=validar_tecla).strip()
        return novo_texto if novo_texto else cadea
    except KeyboardInterrupt:
        return False  # Se ESC, devolvemos o nome orixinal

def renomear_arquivo(stdscr, novo_nome):
    # Recibe o nome que se lle quere poñer e modifica o arquivo

    stdscr.clear()
    #novo_camiño = os.path.join(os.path.dirname(camiño_completo), novo_nome)
    
    # veriiica se existe un arquivo co mesmo nome
    contador = 1
    nome_base, ext = os.path.splitext(novo_nome)
    if os.path.exists(novo_nome):
        stdscr.addstr("\nO ARQUIVO XA EXISTE!\n\nModificouse a cadea.\n")
        while os.path.exists(novo_nome):
            novo_nome = f"{nome_base}_{contador}{ext}"
            #novo_camiño = os.path.join(camiño_completo, novo_nome)
            contador += 1
    
    stdscr.addstr(f"Vaise renomear {camiño_completo}\n")
    stdscr.addstr(f"por {novo_nome}\n")
    stdscr.addstr("\nPreme ENTER para confirmar a acción.\n")
    stdscr.addstr("(Se queres cancelar, preme ESC)\n")
    stdscr.refresh()

 
    while True:
        tecla = stdscr.getch()
        if tecla == 10: # tecla ENTER
            try:
                os.rename(camiño_completo, novo_nome)
                stdscr.addstr(f"\nFicheiro renomeado a: {novo_nome}\n")
                return True
            except Exception as e:
                stdscr.addstr(f"\nErro ao renomear o ficheiro: {e}\n")
        elif tecla == 27: # tecla ESC
            return False

def abrir_imaxe(ruta_imaxe):
    try:
        if sys.platform.startswith('linux'):
            subprocess.run(['xdg-open', ruta_imaxe], check=True)
        elif sys.platform.startswith('darwin'):  # macOS
            subprocess.run(['open', ruta_imaxe], check=True)
        elif sys.platform.startswith('win'):
            os.startfile(ruta_imaxe)  # Windows
        else:
            print("Sistema operativo non soportado para abrir imaxes.")
    except Exception as e:
        print(f"Erro ao abrir a imaxe: {e}")

def procesar_ruta(ruta):
    # python pasa como parámetro o nome do executable
    global camiño_completo, nome_arquivo, extension

             
    if len(ruta) > 0:
        for arquivo in ruta[0:]:
            print(arquivo)
            if os.path.isfile(arquivo):
                camiño_completo = arquivo
                nome_arquivo, extension = os.path.splitext(camiño_completo)
                procesar_arquivo()

    else:
        print(f"{ruta}\nA ruta non é válida.")

def procesar_arquivo():
    curses.wrapper(mostrar_menu)

###
camiño_completo = nome_arquivo =  extension = recordatorio = ''
escaneo_vertical = False
max_ancho = 2100
max_alto = 2000
lingua = 'spa+glg'

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Uso: python script.py <ruta_do_arquivo_ou_directorio>")
        sys.exit(1)

    # --- Aquí vai o argparse ---
    parser = argparse.ArgumentParser(
        description="Script para extraer texto de imaxes e renomear ficheiros baseado no contido OCR."
    )
    parser.add_argument("ruta", nargs="+", help="Ruta do ficheiro ou directorio a procesar.")
    parser.add_argument("-p", "--porcentaxe", type=int, default=100,
                        help="Porcentaxe da imaxe a analizar (positivo: parte superior, negativo: parte inferior).")
    parser.add_argument("-l", "--lingua", type=str, default="spa+glg",
                        help="Lingua usada para o motor OCR (ex.: eng, spa, glg, spa+glg). Debe estar instalada.")
    parser.add_argument("-v", "--ver", action="store_true",
                        help="Abrir a imaxe co visor predeterminado.")
    parser.add_argument("--engine", choices=["tesserocr", "pytesseract", "auto"],
                        default="auto", help="Escolle o motor OCR a empregar.")

    args = parser.parse_args()
    # --- Fin argparse ---

    # Selección do motor OCR segundo argumento
    if args.engine == "tesserocr":
        try:
            import tesserocr
            OCR_ENGINE = 'tesserocr'
        except ImportError:
            print("Erro: non está dispoñible tesserocr.")
            sys.exit(1)
    elif args.engine == "pytesseract":
        try:
            import pytesseract
            OCR_ENGINE = 'pytesseract'
        except ImportError:
            print("Erro: non está dispoñible pytesseract.")
            sys.exit(1)
    else:  # auto
        try:
            import tesserocr
            OCR_ENGINE = 'tesserocr'
        except ImportError:
            try:
                import pytesseract
                OCR_ENGINE = 'pytesseract'
            except ImportError:
                print("""
                Non se atopou un motor de recoñecemento.
                Instálao en Ubuntu con:
                    sudo apt install python3-tesserocr
                ou:
                    sudo apt install python3-pytesseract
                """)
                sys.exit(1)
    
    print(f"Usando motor OCR: {OCR_ENGINE}")

    # Gardar lingua seleccionada en variable global
    lingua = args.lingua

    # Pasar argumentos á función de procesado
    procesar_ruta(args.ruta)

I used it to scan obituary documents and suggest and confirm name changes for jpg files. It saves me time when I do batch scanning of obituaries from physical copies.

$ python3 ocr-renamer <path to file, use with wildcard>

ocr_renamer can also suggest and add tags to several types of obituaries:

ESQ José Deogracias Franco Vidal [cabodano] [recordatorio].jpg

Where “cabodano” is for anniversary obituaries and “recordatorio” is for small cards.

And here is why I need Fios and ocr_renamer. +1000 files only with obituaries.

Would be interesting to see a plugin that perhaps integrates with GitHub - paperless-ngx/paperless-ngx: A community-supported supercharged document management system: scan, index and archive all your documents · GitHub

Which is a service that does exactly what OP mentioned, OCR and all, specifically made for handling tons and tons of documents. Would be interesting to see Gramps work with it!

After some refactoring, I also made profiles handling more friendly.
So:

$ python3 script_ocr.py --list-profiles
✅ EasyOCR disponible (version 1.7.2)
✅ TorchFree OCR disponible (version 1.1.0)
⚠️ Fichier de dictionnaire introuvable : dictionnaire_noms.json
   Un dictionnaire vide sera utilisé.

======================================================================
           📋 PROFILS DISPONIBLES (utilisez --profile <nom>)           
======================================================================

🔹 document_imprime
   └─ Description: Document imprimé propre (PDF, livres)
   └─ Moteur OCR: pytesseract
   └─ OEM: 1 (LSTM only)
   └─ PSM: 6 (Bloc uniforme)
   └─ Langues: fra+eng

🔹 manuscrit_ancien
   └─ Description: Manuscrit ancien (bruit, fond non uniforme)
   └─ Moteur OCR: pytesseract
   └─ OEM: 1 (LSTM only)
   └─ PSM: 11 (Texte épars)
   └─ Langues: fra

🔹 tableau_colonnes
   └─ Description: Tableau avec colonnes (noms + folios)
   └─ Moteur OCR: pytesseract
   └─ OEM: 1 (LSTM only)
   └─ PSM: 6 (Bloc uniforme)
   └─ Langues: fra

🔹 numeros_purs
   └─ Description: Extraction de numéros uniquement (folios, dates)
   └─ Moteur OCR: pytesseract
   └─ OEM: 1 (LSTM only)
   └─ PSM: 8 (Mot unique)
   └─ Langues: fra

🔹 easyocr_manuscrit
   └─ Description: Manuscrit avec EasyOCR
   └─ Moteur OCR: easyocr

🔹 torchfree_tableau
   └─ Description: Tableau avec TorchFree OCR
   └─ Moteur OCR: torchfree

======================================================================
Exemple: python script.py --profile manuscrit_ancien --input image.jpg
======================================================================

Sure, I also need to add some extra checking around list of scenari. e.g.,

$ python3 script_ocr.py
✅ EasyOCR disponible (version 1.7.2)
✅ TorchFree OCR disponible (version 1.1.0)
⚠️ Fichier de dictionnaire introuvable : dictionnaire_noms.json
   Un dictionnaire vide sera utilisé.

📋 Sélectionnez un scénario (ou tapez le nom) :
# typed source
1. Document imprimé propre (PDF, livres) (document_imprime) 
# handwritten source or old sources
2. Manuscrit ancien (bruit, fond non uniforme) (manuscrit_ancien) 
# experimental tabular data checking
3. Tableau avec colonnes (noms + folios) (tableau_colonnes) 
# digits only
4. Extraction de numéros uniquement (folios, dates) (numeros_purs)
# EasyOCR (with PyTorch support) 
5. Manuscrit avec EasyOCR (easyocr_manuscrit) 
# light EasyOCR (without PyTorch support)
6. Tableau avec TorchFree OCR (torchfree_tableau) 

Choix (numéro ou nom) : 7
Traceback (most recent call last):
  File "script_ocr.py", line 527, in <module>
    args.profile = select_profile_interactively()
  File "script_ocr.py", line 449, in select_profile_interactively
    return list(DEFAULT_PROFILES.keys())[idx]
IndexError: list index out of range

As wrote, it is just my own way for testing some OCR tools available under my config.

For advanced work (and community tools), I should also have a look at:

and