Talk to your tree!

[DavidMStraub]

Already in August, I implemented a new AI chat feature in Gramps Web but it took me a while to sort out a few outstanding things and write documentation, now I can finally start advertising it. Gramps Web will soon be the first open source genealogy system with an AI assistant!

When will it be released?

Gramps Web API v2.5.0 will be released in the next few days, followed directly Gramps Web (frontend) v24.10.0.

How does it work?

The chat assistant is based on retrieval augmented generation, a.k.a. RAG. You can check the Wikipedia article if you are interested in details, but the principle is fairly simple:

1. Every Gramps Web object is converted to a textual string
2. Every object string is encoded into a numerical vector using an embedding model
3. A chat prompt is converted to a numerical vector using the same embedding model
4. A vector search index finds the Gramps objects that are most semantically similar to the prompt, using cosine similarity
5. The retrieved Gramps objects are sent to a large language model (LLM), which produces an answer to the prompt given the provided information.

What this means is that no model training is needed on Gramps data. Steps 1.-4. always happen fully locally. Only step 5 is where information is potentially sent to a third party like OpenAI’s API, but this is totally up to the person managing the Gramps Web instance. The implementation fully supports running an open source LLM locally, provided sufficient computing power is available.

The vector search engine is provided by Sifts that I had written originally to replace whoosh in Gramps Web.

What can it do?

At the moment, the assistant only retrieves a limited number of Gramps objects based on the question. That means its good at retrieval of information about individuals, events, or places. How good it is depends a lot on what you have in your database. Please play around and share your findings!

What can’t it do?

A lot of things! Since the system is currently only retrieving Gramps objects by semantic similarity, it will fail to answer even simple genealogical questions like “which of John Doe’s great grandparents were born in Southern Europe?”, unless the answer is contained in a textual note, since it cannot connect people across Gramps objects (children/parents work because they are part of the same family object).

Can it speak my language?

Yes, most likely! It depends on the embedding model and LLM you use, but many of them support a large number of languages.

How can I use it?

The new Gramps Web version will be released soon, you can already use it in the development version now. The documentation for setting it up is here:

• Setting up AI chat - Gramps Web
• AI chat - Gramps Web

How can I help?

Please share things that work and that don’t work - both for configuration (different models) and usage (different prompts, different databases).

If you want to help developing the chatbot, you can have a look at the relevant Python module.

Will it get better in the future?

Most likely yes.

One obvious way to improve the assistant (contributions welcome!) is to include tool calling. This would allow the assistant to tap into Gramps’ powerful genealogical and filtering capabilities, e.g. executing a relationship calculation.

I want to know more.

I am planning a series of blog posts to delve into the details of implementation & usage, I will share the links here.

Do we really need this?

I don’t know, but it was instructive & fun implementing it.

EDIT: By the way, chat & semantic search are disabled by default in Gramps Web. Once enabled, access to the chat feature can be limited to specific user groups (e.g., only owners). Of course, the LLM will never see information that a particular user cannot see, e.g. users with guest permissions will not get answers containing information on private records.

[Urchello]

Great! The main thing is to start!
My greetings!

The capabilities of Gramps Web (and the desktop part) would be significantly broader in this area if artificial intelligence were integrated directly into the core, allowing it to directly access the database with its own queries. However, this would require finding ways to restrict its access. It might be necessary to add a settings page with permissions for artificial intelligence. For instance, it should not be able to access data marked as private. The AI would be able to make queries to the database that are limited by privacy and other permissions, receive responses, and process them. This would then provide us with unlimited opportunities to further develop the project.

[dsblank]

Interesting… I wondered what all of the Machine Learning libraries required in the tests was about.

I have a few concerns, namely around security/privacy (what is sent offsite), and around making sure that people understand what LLMs can and can’t do.

[DavidMStraub]

Yes, those are valid concerns and we have to make sure everything is sufficiently documented to allow admins/users to make informed decisions.

Concerning data being sent offsite, there are two scenarios:

1. A third-party API is used, e.g. OpenAI (but there are various others using the same API,
2. The LLM is run locally, e.g. using Ollama.

In the second case, no data ever leaves the server. In the first case, the semantic search happens on the server and results in an ordered list of potentially relevant Gramps objects. A prompt is then sent to the API that has the content:

You are an assistant for answering questions about a user’s family history. Use the following pieces of context retrieved from a genealogical database to answer the question. If you don’t know the answer, just say that you don’t know. Use three sentences maximum and keep the answer concise. In your answer, preserve relative Markdown links.

Context: <Textual form of relevant Gramps objects>

<User prompt>

The context does not contain all the data in the database, but can contain data of a few dozen objects.

As for private data, it depends on the permissions of the user: if the user is authorized to access private records, the context will contain private data as well.

What should be stressed is that these APIs are paid services which need to comply with data privacy laws, so they are not allowed to use the data to train AI, so it’s not like the free version of ChatGPT, for instance.

But of course it’s up to every administrator to decide whether they trust a given provider and the option to run an LLM locally has been included in the design from the beginning. Since the LLM does not have to do a lot of reasoning, a “small” local LLM should do a pretty decent job.

If administrator’s don’t trust their users, they can also enable AI chat but only for themselves.

[DavidMStraub]

Concerning what LLMs can and can’t do, it might help to add a disclaimer and a link to the documentation in the chat window.

What makes it a bit less critical is that, using the above system prompt, the LLM usually doesn’t hallucinate, so if you ask it for the weather or whether you’re related to George Washington, it will simply say “I don’t know”.

[dsblank]

Both of these responses are great and need to be in the docs and in the gramps-web UI I think. (the UI could use some links or dialogs in a number of places, and this is one).

I do look forward to trying it. ML and LLMs are my day job: www.comet.com and I did my PhD in neural networks.

[dsblank]

@DavidMStraub

“which of John Doe’s great grandparents were born in Southern Europe?”

BTW, for practical Gramps use, I am more interested in questions like this. Although I don’t think AI is the right approach for these questions, I do think a nice query language should help (and be fast).

[DavidMStraub]

@dsblank

Although I don’t think AI is the right approach for these questions, I do think a nice query language should help (and be fast).

The query language is the right approach, the perfect approach would be training AI on the query language, so we can ask those questions without knowing the query language

[dsblank]

Maybe AI : I actually explored learning family tree relationships in my PhD dissertation 25 years ago: Learning to See Analogies chapter 5, section 3. This was based on work done by Geoff Hinton, co-winner of the Turing Award (2018) and now the Nobel Prize (Physics, 2024). I found some interesting things, but the limits still exist today.

I love the work on LLMs today, but I don’t trust their output.

[codefarmer]

Agree with both @dsblank in asking why we need AI for this, and at the same time, with @DavidMStraub, about not needing to learn a (structured) query language to get answers to simple queries.

A natural language query makes it so easy to get quick answers for one-off questions and exploration. Sure, a query language and the ability to build and save queries is useful when the same query needs to be run over and over again, but most of the time I don’t really need that.

Another way this could unlock more value is if it could search everything in the family tree, in comparison to a query language which might not have that attribute as part of the formal query language.

Trust is an issue, so I am looking forward to try this out and validate the answers. In fact, I set up GrampsWeb only for this capability!

[emyoulation]

@DavidMStraub

How can I use it?

The new Gramps Web version will be released soon, you can already use it in the development version now. The documentation for setting it up is here:

• Setting up AI chat - Gramps Web
• AI chat - Gramps Web

The UI documentation is unclear. In the “How to ask a question” section, the information seems to be about how to compose a well-written question. I was hoping for more basic info… about where to access the AI Chat in the user interface.

[emyoulation]

@DavidMStraub

When will it be released?

Gramps Web API v2.5.0 will be released in the next few days, followed directly Gramps Web (frontend) v24.10.0.

@DavidMStraub

EDIT: By the way, chat & semantic search are disabled by default in Gramps Web. Once enabled, access to the chat feature can be limited to specific user groups (e.g., only owners). Of course, the LLM will never see information that a particular user cannot see, e.g. users with guest permissions will not get answers containing information on private records.

When will the AI Chat be rolled out into the Gramps Web Demo?

[emyoulation]

@DavidMStraub

A prompt is then sent to the API that has the content:

You are an assistant for answering questions about a user’s family history. Use the following pieces of context retrieved from a genealogical database to answer the question. If you don’t know the answer, just say that you don’t know. Use three sentences maximum and keep the answer concise. In your answer, preserve relative Markdown links.

Context: <Textual form of relevant Gramps objects>

<User prompt>

Can the AI be trained to an alternative to the “just say that you don’t know”? Perhaps it could be an expert in the Gramps Web source javascript code and UI, then suggest how users could discover that information via the Gramps Web UI. Or suggest how the question could be re-written to suggest that workflow (for a layman) as a follow-up prompt.

[dsblank]

I think the initial idea of @DavidMStraub is a very nice “proof of concept” (POC) but to do this task (“talk to your tree”) in a more general way is a problem that requires some experimentation.

I’d recommend that we spin this off into a separate project for experimentation and development. For example, can we provide a LLM context that is not as limiting as this initial POC? Generating Python code, and running it, could allow the system to create an interaction like this:

Show me the great-grandparents of John Smith, and the countries they were born in

… Generating code…
… Running code…

|--------------------------|---------------------------|
| Sara Jones               |  Hungary                  |
| John Jones               |  Hungary                  |
| Robert Smith             |  Virginia                 |
|--------------------------|---------------------------|

It could even operate stand-alone: upload your tree (GEDCOM), and then talk to it. Sounds like a viable product

[DavidMStraub]

@dsblank

I think the initial idea of @DavidMStraub is a very nice “proof of concept” (POC) but to do this task (“talk to your tree”) in a more general way is a problem that requires some experimentation.

I don’t disagree! Getting the infrastructure and UI in place was a chunk of work, but now that this is in place, there is a lot of opportunity to improve the actual quality of responses. I also want to write a blog post about how anyone who is interested can play around with this. It’s actually quite straightforward, just a couple of lines in a Jupyter notebook and anyone can play around with different system prompts, models, or APIs.

[DavidMStraub]

@emyoulation

When will the AI Chat be rolled out into the Gramps Web Demo?

I don’t think it’s realistic add AI chat to the demo, because it either needs a local LLM which is very expensive to run with potent enough hardware (GPUs cost hundreds of $ a month) or needs an API key, which costs per request and would likely require some work to prevent somebody from flooding it with nonsense creating a large bill.

Apart from that, the current example tree does not contain enough meaningful information.

[hartenthaler]

Do you plan to integrate AI into the SaS GrampsHub? The user can provide an API key and all the costs belong to him (and he has control over the usage).

[DavidMStraub]

My plan is to roll out AI chat on Grampshub soon (perhaps in a week or so if no ciritcal bugs are reported) and try if it’s sustainable to cover the AI API costs by the existing monthly subscriptions. I am quite optimistic this will work because the cost per question is quite low (and partly because the AI chat is not yet very powerful ). If the costs are unexpectedly large (e.g. individual users asking a huge number of questions), probably the easiest solution is to limit the number of question per tree and month to a reasonable number.

[emyoulation]

@DavidMStraub

I don’t think it’s realistic add AI chat to the demo, because it either needs a local LLM which is very expensive to run with potent enough hardware (GPUs cost hundreds of $ a month) or needs an API key, which costs per request and would likely require some work to prevent somebody from flooding it with nonsense creating a large bill.

Can you create a throttle for the AI chat? So a DoS attack can be automatically aborted?

And for the Demo, perhaps the AI could start with an estimate of cost of processing the request. And have a processing “budget” per user level per day? Even if budget has been exhausted, the demo could still show the interface.

@DavidMStraub

probably the easiest solution is to limit the number of question per tree and month to a reasonable number.

Given the known complexity (but perhaps too many people, given the preset limit levels for GitHub) of the Example.gramps tree, the cost estimating might be more doable? Here’s an Organic layout diagram of the GEDCOM in yEd

[DavidMStraub]

Theoretically that’s doable sure, I don’t think it’s worth it right now, but once we have a better example.gramps we can look at it again.