AI API for Supertool

@kku, I have an idea that could significantly expand Supertool’s capabilities. Gramps does not have built-in artificial intelligence, likely due to privacy concerns. However, Supertool is designed mainly for developers, and anyone can use AI to meet their needs without limitations – solving any tasks they want.

Users can select data from the database, generate AI queries, receive recommendations, and even use the results to update database records. In my opinion, this is a very interesting and promising feature. And I believe Supertool is the perfect place for its implementation.

3 Likes

I don’t know much about AI. Maybe you can write a prototype.

1 Like

I think a regular Gramps addon could work. I do this for my day job. Let me see if I can put something together.

FYI, @DavidMStraub also has a prototype AI chatbot in gramps-web. But it was designed specifically to allow answering questions about your family tree. Which an addon could do too.

3 Likes

I already have a working simplest prototype. But I want prepare more useful script to demonstrate how we can use it

import requests

OPENAI_API_KEY = "sk-..."

url = "https://api.openai.com/v1/chat/completions"
headers = {
    "Authorization": f"Bearer {OPENAI_API_KEY}",
    "Content-Type": "application/json"
}
data = {
    "model": "gpt-4o-mini",
    "messages": [
        {"role": "system", "content": "Return only 'true' or 'false'."},
        {"role": "user", "content": "Does this description contain a page number at the end? Example: 'Document Page 12'"}
    ],
    "temperature": 0,
    "max_tokens": 5
}

response = requests.post(url, json=data, headers=headers)

try:
    print("API Response:", response.json()["choices"][0]["message"]["content"].strip())
except Exception as e:
    print("Error:", e)
1 Like

Yes, that is a start :slight_smile: I’m imaging a real addon that allows you to even use your own model without costing you anything.

5 Likes

Yep, see gramps-web-api/gramps_webapi/api/llm/__init__.py at master · gramps-project/gramps-web-api · GitHub

and please also have a look at this discussion:

2 Likes

Well, I have obtained positive results demonstrating the possibility of using artificial intelligence in conjunction with SuperTool. I pass the father’s first name and the middle names of his children. The AI’s task is to verify whether the children’s patronymics are correct. If they are not, it suggests specific corrections.

Solving this problem using traditional conditional logic is quite difficult because Ukrainian names have many forms, endings, and exceptions to the rules. This is exactly the kind of task that AI is suited for.

In the terminal, I demonstrated how it works for a single person. I have not tested the script on the entire database. This is a demonstration script that requires improvements and additional validations.

Supertool script:

[Gramps SuperTool script file]
version=1

[title]
Check Patronymic Validity

[description]
This script checks whether the patronymic of children matches their father's name.

[category]
People

[initial_statements]
import json
import sys
sys.path.append("/home/my/Documents")
from openai_client import OpenAIConfig, OpenAIClient

# OpenAI API Configuration with optional training data
config = OpenAIConfig(
    api_key="ADD KEY HERE",
    training_examples=[
        {
            "role": "user",
            "content": json.dumps({
                "father": {"gramps_id": "I7204", "name": "Порфирій Савович"},
                "children": [
                    {"gramps_id": "I0861", "name": "Антон Порфирович"},  # ✅ Correct
                    {"gramps_id": "I1118", "name": "Макар Іванович"},  # ❌ Incorrect
                    {"gramps_id": "I2345", "name": "Марія Порфиевна"},  # ❌ Misspelled
                    {"gramps_id": "I9876", "name": "Оксана"}  # ❌ Missing patronymic
                ]
            }, ensure_ascii=False)
        },
        {
            "role": "assistant",
            "content": json.dumps({
                "I0861": {"status": "correct"},
                "I1118": {
                    "status": "incorrect",
                    "explanation": {"current": "Іванович", "suggested": "Порфирович"}
                },
                "I2345": {
                    "status": "incorrect",
                    "explanation": {"current": "Порфиевна", "suggested": "Порфирівна"}
                },
                "I9876": {
                    "status": "incorrect",
                    "explanation": {"current": "No patronymic", "suggested": "Порфирівна"}
                }
            }, ensure_ascii=False)
        }
    ]
)

openai_client = OpenAIClient(config)

def check_patronymic_validity():
    """
    Prepares and sends a request to OpenAI API to validate children's patronymics.
    """
    father_firstname = firstname

    data = {
        "father": {
            "gramps_id": gramps_id,
            "name": father_firstname
        },
        "children": []
    }

    for child in children:
        data["children"].append({
            "gramps_id": child.gramps_id,
            "name": child.firstname
        })

    user_messages = [
        {
            "role": "user",
            "content": json.dumps(data, ensure_ascii=False)
        }
    ]

    # Use assistant training if needed
    result = openai_client.send_request(
        user_messages,
        system_prompt="Identify the father's first name and ignore his patronymic. "
                      "For each child, extract only the patronymic and ignore the first name. "
                      "Compare them using proper Ukrainian patronymic formation rules. "
                      "If incorrect, return 'status': 'incorrect' with 'explanation': "
                      "{\"current\": \"<child's current patronymic>\", \"suggested\": \"<corrected patronymic>\"}. "
                      "If correct, return only 'status': 'correct'.",
        include_training=True  # Toggle training data on/off
    )

    print("Request:", json.dumps(data, indent=4, ensure_ascii=False))
    print("Response:", json.dumps(result, indent=4, ensure_ascii=False))

[statements]
if gender == "M":
    check_patronymic_validity()

[scope]
selected

[unwind_lists]
False

[commit_changes]
False

[summary_only]
True

OpenAI API Client

# openai_client.py
import requests
import json

class OpenAIConfig:
    """Configuration class for OpenAI API."""

    def __init__(
        self,
        api_key: str,
        model: str = "gpt-4o-mini",
        temperature: float = 0,
        max_tokens: int = 100,
        api_url: str = "https://api.openai.com/v1/chat/completions",
        system_prompt: str = "",
        training_examples: list = None
    ):
        """
        Initializes the OpenAI configuration with default or custom parameters.

        :param api_key: OpenAI API key.
        :param model: AI model to use (default: "gpt-4o").
        :param temperature: Sampling temperature (default: 0.0).
        :param max_tokens: Maximum response tokens (default: 100).
        :param api_url: OpenAI API endpoint (default: GPT-4 endpoint).
        :param system_prompt: Default system prompt (can be overridden per request).
        :param training_examples: Optional assistant training data (default: None).
        """
        self.api_key = api_key
        self.model = model
        self.temperature = temperature
        self.max_tokens = max_tokens
        self.api_url = api_url
        self.system_prompt = system_prompt
        self.training_examples = training_examples or []  # If None, use an empty list


class OpenAIClient:
    """Client for interacting with OpenAI API."""

    def __init__(self, config: OpenAIConfig):
        """
        Initializes the OpenAI client.

        :param config: OpenAIConfig instance containing API settings.
        """
        self.config = config

    def send_request(self, user_messages: list, system_prompt: str = None, include_training: bool = True):
        """
        Sends a chat completion request to OpenAI API.

        :param user_messages: List of messages in OpenAI format (e.g., [{"role": "user", "content": "Your input"}]).
        :param system_prompt: Custom system prompt (if None, default from config is used).
        :param include_training: If True, include assistant training examples.
        :return: Parsed JSON response from OpenAI.
        """
        messages = [{"role": "system", "content": system_prompt or self.config.system_prompt}]
        
        # Add training examples if enabled
        if include_training and self.config.training_examples:
            messages.extend(self.config.training_examples)

        # Add user messages
        messages.extend(user_messages)

        request_data = {
            "model": self.config.model,
            "messages": messages,
            "temperature": self.config.temperature,
            "max_tokens": self.config.max_tokens
        }

        headers = {
            "Authorization": f"Bearer {self.config.api_key}",
            "Content-Type": "application/json"
        }

        try:
            response = requests.post(self.config.api_url, json=request_data, headers=headers)
            response_json = response.json()

            if "choices" in response_json and response_json["choices"]:
                return json.loads(response_json["choices"][0]["message"]["content"].strip())

        except Exception as e:
            print("API Request Error:", e)
            return None

Terminal results:

3 Likes

Interesting! Something similar could be used for Irish names:

“For about two centuries (from the late 1700s through to the early to mid-1900s) the Irish favoured a precise convention for naming their children that can suggest what names to look for in a previous generation. All that’s needed is for one sibling in a family to have used this pattern with accuracy (even if one’s own direct ancestor deviated a little).”

https://irelandxo.com/ireland-xo/news/irish-naming-conventions-and-baptism-traditions

1 Like

yeah, this is similar ukrainian middlenames

1 Like

But I think the example by @Urchello is dealing with surname data derived from given names of parents. And there are some “origin” etymology that could be filled in.

The Irish was inheritance pattern of given name to given name. I suppose you could add eponym and namesake associations.

1 Like

I believe I have many duplicate people with the same names and surnames.
Some of them are actually different people, while others are true duplicates. Identifying which ones are duplicates requires a very complex analysis. To make accurate decisions, it is necessary to examine not only the people themselves but also their partners, children, parents, godparents, dates, attributes, events.

I believe that if AI is given such data samples, it could assist with this kind of analysis. Even if it cannot fully resolve the issue due to some missing facts, AI could still make well-founded assumptions, eliminate clearly incorrect options, and significantly reduce the workload.

This is how I plan to use artificial intelligence in conjunction with SuperTool in the coming years. I believe that this problem is not unique to me but is also encountered by other researchers.

Gramps could be the first genealogic app which makes similar things.

2 Likes

Perplexity is fairly good at refining and fleshing out such concepts. Here is a discussion about “disambiguation of identically named persons” after several rounds. It probably needs to be expanded with discussions

perplexity responds:

When discussing genealogy research and distinguishing between individuals with the same birth name, several terms and concepts are used interchangeably with “disambiguation”. These include:

  1. Deconfliction: Resolving conflicts between records that appear to represent the same person.

  2. Name resolution: Determining which records belong to which individual when multiple people share the same name.

  3. Identity resolution: Establishing a person’s unique identity beyond just their name.

  4. Deduplication: Removing duplicate records from databases or family trees.

  5. Name differentiation: Distinguishing between individuals with identical or similar names.

  6. Record linkage: Connecting different records that belong to the same individual, even if names vary slightly.

  7. Entity resolution: Determining whether multiple records refer to the same real-world entity.

  8. Same Name Rule Out methodology: Systematically excluding individuals who share a name but cannot be the person of interest.

  9. Unique identification: Assigning or recognizing unique identifiers for individuals to prevent confusion.

  10. Name disambiguation: The primary term, often used interchangeably with the others mentioned.

  11. Namesake and eponym identification: A crucial aspect of genealogical research, especially when dealing with individuals who share the same name. This process involves several strategies:

    a. Contextual Analysis: Examining historical and familial context to distinguish between individuals.
    b. Temporal Differentiation: Using birth, marriage, and death dates to distinguish between namesakes across generations.
    c. Geographical Distinction: Considering locations associated with each individual.
    d. Suffix Usage: Employing suffixes like I, II, III to distinguish between individuals across generations.
    e. Occupation and Social Status: Using information about profession or social standing to differentiate.
    f. Family Relationships: Analyzing family structure and relationships to identify distinct individuals.
    g. Nickname or Middle Name Identification: Looking for variations in how a person’s name is recorded.
    h. DNA Evidence: Using genetic testing to differentiate between individuals who share the same name but are not closely related.

These terms and strategies are used in various contexts within genealogical research to address the challenges of namesakes, unrelated individuals with identical names, and duplicate records resulting from data entry errors or import issues. By employing these methods, genealogists can more accurately identify and distinguish between individuals, ensuring the integrity of their family history research.

2 Likes

Small example:

Yes, this is huge job and AI can make a big part. Not all of course

agreed. And I noticed something it overlooked in the discussion (and equally large dataset that is good AI fodder): Points of correlation

Perplexity responds

Yes, “points of correlation” suggests another important aspect in genealogical research and disambiguation. This concept refers to the specific elements or data points that align or connect across multiple sources, helping to establish relationships or identify individuals[7].

Points of correlation can include:

  1. Consistent demographic information (e.g., age, birthplace, occupation)
  2. Recurring family members or associates
  3. Matching event dates (birth, marriage, death)
  4. Similar residence locations
  5. Consistent naming patterns

By identifying and analyzing these points of correlation, researchers can build a stronger case for linking records to the same individual or family, even when dealing with common names or limited information[1][5]. This approach is particularly useful in resolving ambiguities and distinguishing between individuals with similar names or characteristics.

Citations:
[1] Correlation: When and How « Genealogy Certification: My Personal Journal
[2] Timelines for Analysis & Correlation « Genealogy Certification: My Personal Journal
[3] https://books.byui.edu/fhgen_110_textbook_/chapter_3_genealogical_analysis_and_conclusions
[4] 3 Ways to Advance Your Research with Correlation - Legacy Family Tree Webinars
[5] Eight Key Types of Evidence in Genealogical Research - Kentucky Genealogical Society
[6] https://www.ngsgenealogy.org/wp-content/uploads/Complimentary-NGS-Monthly-Articles/NGS-Monthly-Johnson-Analysis-Correlation-Apr2015.pdf
[7] https://cms-b-assets.familysearch.org/4b/71/a2f6de0e0aa0b1347419c72020d7/course-handout.pdf
[8] https://www.youtube.com/watch?v=Vn7rrAP9xIM

I should have started on this a long time ago :slight_smile:

Here is the start of a GrampsChat. This should also be a good way for exploring a chatbot for gramps-web as well. There is no reason they need to be two different things.

Currently, I have it hardcoded for openai, but through litellm, it should be easy to configure for any AI model, including your own.

Note that if you don’t restart your conversation often, your context messages will build up and that costs $$$$.

It doesn’t currently load any any of your local database, but it could. I’ll package this up soon and make an addon. (It’s pretty ugly right now, and is really just a Gtk wrapper around a chatbot conversation. But let me know if this would be useful to you.)

output

5 Likes