What Fuzzy international name matching system should be added to Gramps?

[emyoulation]

Gramps is bundled with support for the phonetic indexing system: SoundEx. This fuzzy name matching system was adopted Census Bureau and the Social Security Administration in the 1930s. But is is not well-suited to non-English phonetics. And the implementation is the original 4-character “Russell” (NARA), not the the 6-digit Daitch–Mokotoff Soundex, a specialized refinement for Slavic and Yiddish surnames. (The 2 variants are compared here.)

If we wanted a more globally applicable phonetic ‘fuzzy’ name matching system, what should it be?

Here are a couple systems:

• Metaphone and Double Metaphone (Metaphone 3 is the professional version.) But are still western culture oriented.
• Beider-Morse (BMPM) was developed specifically for Genealogical Use.
• phonex for French culture oriented

Any others to consider?

[romjerome]

Maybe minor adjustments for French?

A local AI for Gramps?

C’est marrant, j’avais fait un patch relativement propre à l’époque ! --- finddupes.glade 2010-10-24 17:05:33.000000000 +0200 +++ finddupes.glade 2010-12-17 18:43:53.000000000 +0100 @@ -264,6 +264,25 @@ </packing> </child> <child> + <object class="GtkCheckButton" id="phonex"> + <property name="label" translatable="yes">Use phonex codes</property> + <property name="visible">True</property>…

[emyoulation]

From a superficial look, Phonex appears to be a French dialect of the 4-character “Russell” (NARA) Soundex. It does not seem to build on the 6-digit Daitch–Mokotoff Soundex base. So you get one or the other?

[romjerome]

By adjustments, I only thought on the replacement sections on code (oriented sounds into french locale), not really on a complete soundex alternative.

    # 1 remplacer les y par des i
    r = r.replace('Y','I')
    
    # voir 7    
    r = r.replace(u'É','Y')
    r = r.replace(u'È','Y')
    r = r.replace(u'Ê','Y')
                
    r = unicodedata.normalize('NFKD', 
                    unicode(strval.upper().strip())).encode('ASCII', 'ignore')
  
    # 2 supprimer les h qui ne sont pas précédées de c ou de s ou de p
    r = re.sub(r'([^P|C|S])H', r'\1', r)

    # 3 remplacement du ph par f
    r = r.replace(r'PH', r'F')
  
    # 4 remplacer les groupes de lettres suivantes :
    r = re.sub(r'G(AI?[N|M])',r'K\1', r)
  
    # 5 remplacer les occurrences suivantes, si elles sont suivies par 
    # une lettre a, e, i, o, ou u :
    r = re.sub(r'[A|E]I[N|M]([A|E|I|O|U])',r'YN\1', r)
    
    # 6 remplacement de groupes de 3 lettres (sons 'o', 'oua', 'ein') :
    r = r.replace('EAU','O')
    r = r.replace('OUA','2')
    r = r.replace('EIN','4')
    r = r.replace('AIN','4')
    r = r.replace('EIM','4')
    r = r.replace('AIM','4')
  
    # 7 remplacement du son É:
    # voir plus haut
    r = r.replace('AI','Y')
    r = r.replace('EI','Y')
    r = r.replace('ER','YR')
    r = r.replace('ESS','YS')
    r = r.replace('ET','YT')
    r = r.replace('EZ','YZ')

    # 8 remplacer les groupes de 2 lettres suivantes (son â..anâ.. 
    # et â..inâ..), sauf sâ..il sont suivi par une lettre a, e, i o, 
    # u ou un son 1 Ã 4 :
    r = re.sub(r'AN([^A|E|I|O|U|1|2|3|4])',r'1\1', r)
    r = re.sub(r'ON([^A|E|I|O|U|1|2|3|4])',r'1\1', r)
    r = re.sub(r'AM([^A|E|I|O|U|1|2|3|4])',r'1\1', r)
    r = re.sub(r'EN([^A|E|I|O|U|1|2|3|4])',r'1\1', r)
    r = re.sub(r'EM([^A|E|I|O|U|1|2|3|4])',r'1\1', r)
    r = re.sub(r'IN([^A|E|I|O|U|1|2|3|4])',r'4\1', r)

    # 9 remplacer les s par des z sâ..ils sont suivi et précédés des 
    # lettres a, e, i, o,u ou dâ..un son 1 Ã  4
    r = re.sub(r'([A|E|I|O|U|Y|1|2|3|4])S([A|E|I|O|U|Y|1|2|3|4])',r'\1Z\2',r)

    # 10 remplacer les groupes de 2 lettres suivants :
    r = r.replace('OE','E')
    r = r.replace('EU','E')
    r = r.replace('AU','O')
    r = r.replace('OI','2')
    r = r.replace('OY','2')
    r = r.replace('OU','3')  

    # 11 remplacer les groupes de lettres suivants
    r = r.replace('CH','5')
    r = r.replace('SCH','5')
    r = r.replace('SH','5')
    r = r.replace('SS','S')
    r = r.replace('SC','S')

    # 12 remplacer le c par un s s'il est suivi d'un e ou d'un i
    r = re.sub(r'C([E|I])',r'S\1',r)
  
    # 13 remplacer les lettres ou groupe de lettres suivants :
    r = r.replace('C','K')
    r = r.replace('Q','K')
    r = r.replace('QU','K')
    r = r.replace('GU','K')
    r = r.replace('GA','KA')
    r = r.replace('GO','KO')
    r = r.replace('GY','KY')

    # 14 remplacer les lettres suivante :
    r = r.replace('A','O')
    r = r.replace('D','T')
    r = r.replace('P','T')
    r = r.replace('J','G')
    r = r.replace('B','F')
    r = r.replace('V','F')
    r = r.replace('M','N')
 
    # 15 Supprimer les lettres dupliquées
    oldc='#'
    newr=''
    for c in r:
        if oldc != c:
            newr=newr+c
        oldc=c
    r = newr

    #16 Supprimer les terminaisons suivantes : t, x
    r = re.sub(r'(.*)[T|X]$',r'\1', r)

[Woody]

My old genealogy program offers the following phonetic search options:

Sondex
Extended Soundex
Kölner Phonetik
PHONEM
dMetaphone 1
dMetaphone 2
Daitch-Mokotoff
Phonet 1
Phonet 2

I never tested the differences.

[emyoulation]

I was originally became motivated to discuss the Soundex feature after having Claude add a Fuzzy Search option to Relationship Filter gramplet, an expansion of @dsblank’s rework of the Flat Filter gramplet by @bugbear.

A bit of searching only surfaced a few features with Soundex options:

• Find Possible Duplicate People (“Use soundex codes” option)
• Soundex match of People with the <name> filter rule
• Soundex gramplet

And the built-in Soundex gramplet was unusable on large trees. (It tries to add pop-up menu ALL known surnames. A menu with thousands of surnames causes the GUI to freeze.) And didn’t seem useful when it did work.

Reworked it to be a “Fuzzy Matching” gramplet that does not add as much burden but still gives more features. And has groundwork that hopefully will support adding other Phonetic encoding systems.

[romjerome]

@emyoulation

Phonex appears to be a French dialect of the 4-character “Russell” (NARA) Soundex.

True, for the pasted code, which looks like a NARA Soundex extended.

By looking at common use and alternative, we can quickly find some basic rules or an alternate return.

I do not know what is this algo. name:

Consonant mappings:
B,P,V,F → B
C,K,Q,G,J → C
S,Z,X → S
D,T → D
L → L
M,N → M
R → R

Special combinations:
PH → F, KN → N
GH → removed
WR → R

Examples:
STEPHEN → SDBM0
  S-T[D]-[e]-PH[F→B]-[e]-N[M]

ASHCRAFT → ASCRF0
  A-S[S]-H-C[C]-R[R]-A-F[B]-T[D]

KNIGHT → NCD0
  KN[N]-I-GH-T[D]

Accuracy benchmarks (Pfeifer et al. 1996, information retrieval studies on English surname recall):
• Soundex: ~85% recall, ~30% precision — too many collisions.
• NYSIIS: ~88% recall, ~45% precision — better consonant handling.
• Phonex: ~90% recall, ~55% precision — preserves more phonetic structure.
• Phonix: ~92% recall, ~60% precision — highest precision among classical algorithms.
• Double Metaphone: ~94% recall, ~70% precision — current gold standard for English.

When to choose Phonex specifically:
• You need a simpler algorithm than Double Metaphone but better than Soundex.
• You’re working on a legacy system that specifies Phonex (e.g., UK NHS patient matching historically used Phonex-family algorithms).
• Your dataset is dominated by English/Celtic/Welsh names (where Phonex was tuned and tested).

When NOT to use Phonex:
• Multilingual name matching (immigrant records, global customer data) → use Double Metaphone or Beider-Morse.
• Jewish/Eastern European surnames → use Daitch-Mokotoff.
• German-only data → use Kölner Phonetik (Cologne Phonetic).

Phonex | Phonetic Name Encoding Algorithm

It also matches fine (or better than NARA Soundex), for some “french names and sounds”.