Gramps attracts users who like having complete control over their data.
Is there a recommendation for a set of tools to run an LLM locally? (Probably on desktops with a GPU.)
What do you mean by the word “tools”?
I was seeing comments on the AI and Genealogy Facebook group about NotebookLM. Those comments that made it look like a tool/accessory application that was able keep its training documents off the net. But their promo pages did not give the same impression.
The reason I wrote “Tool” rather than “application” is because they don’t seem productive in of themselves.
@dsblank
While trying to test a local LLM, I found that the call to complete() in chatbot.py needs to pass on the base URL like so:
base_url=GRAMPS_AI_MODEL_URL,
I posted this on GitHub, but I am not sure whether comments on a merged commit trigger notifications or not.
2 Likes
Thanks for pointing this out; fixed!
1 Like
Maybe. How to know the current active version?
$ python3
Python 3.6.9 (default, Mar 10 2023, 16:46:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import litellm
>>> help(litellm)
pip list | grep litellm
[corrected a typo]
Also:
$ pip show litellm
1 Like
$ pip list | grep litellm
litellm 0.1.202
$ pip show litellm
Name: litellm
Version: 0.1.202
Summary: Library to easily interface with LLM API providers
Home-page:
Author: Ishaan Jaffer
Author-email:
License:
Location: .local/lib/python3.6/site-packages
Requires: cohere, openai
Required-by:
Ok, the installed version is a very old one and the active python3 version is also too old for running gramps 6.x. Anyway, I thought I was able to test it…
Hi @dsblank !
This is really amazing stuff.
Have played a bit with the code, and wanted to run the code in a python virtual environment.
As I was also curious about the interaction that is going on with the remote AI Model, and I also wanted to run it with Gemini, have shamelessly copied the code and altered it slightly, to include debug information.
If others are interested to interact with Gemini and the debug info that goes on when the script interacts with the remote AI model, the source is here: https://github.com/MelleKoning/genealogychatbot
2 Likes
I am curious if it would be possible to set up chatbots in parallel (or series) and feed them the same prompt?
Some direct comparison of output (possibly with some performance and resource usage metrics) would be informative.
You don’t have to run them in parallel to get results to compare, but yes you can.
That is exactly what my company’s open source project “opik” was built for. See:
2 Likes
Great that you find it useful! I plan on continued development of this code (and the associated gramplet) so I’ll be working some of this functionality into the code.
It is hard to find an open LLM that can handle tools. And some models require custom tool handling. I wonder if the Gramps community would be interested in funding a trusted server powerful enough to do such queries on their private data (and keeping it private, at least to the trusted server). Exploration in progress…
We have funds available for community initiatives such as this.
Interesting! I can create a proposal and cost estimates. I think the project would require:
- A server in the cloud running a powerful enough LLM (need GPUs)
- Ability to safely copy your tree to the server
- Ability to encode communication between local machines and the server
I am a bit of a broken record but… could this server be trained on the Gramps wiki and addons PDFs too? So that it would be expert on Gramps as well as genealogical tree information?
And also on the example.gamps tree … so that we ask about what features do not have representative data.
Having the whole personal database uploaded to the cloud sounds a bit risky for privacy reasons, that is, excluding living persons and all.
The nice thing about running the chatbot locally is that locally the right tools are selected to query the database. Extending these tools seems the challenge.
That way, only the relevant information is send over to the remote chat-bot.
I have not yet tried, but it should be possible to run these models locally (with ollama), whereby a model does support tooling. As all knowledge comes from the local database, it is about if the LLM can handle the questions to understand what tools should be called.
I just added the “get_child_in” tool (same github repository as before), and noticed that the LLM needs to be informed a bit about the grams structure to understand what the data fields of gramps mean. Some fields like “handle” and “ref” are a little bit interchangeable depending on what data was asked, like between getting a person or getting data for a whole family.
Have a look what I added for the get_child_in (families) - and the example chat. I am also surprised that some questions are too hard for the model, like “who is the brother of the home person” - would say with the amount of tools it would be able to work that out, but that is not always the case.
can LLM be trained on the Gramps wiki
That’s a different question from interacting with a local gramps database. Training an LLM takes huge amounts of processing time, and the context windows of current LLMs were small until Gemini hit the 1 million token barrier.
I usually just use the famous search machines to find answers to Gramps questions 
As an alternative, I am trying to play with a
Few recents models (with tools), but should run locally too.
It really depends on exactly what model you are using. What model (and size) exactly were you using? Do you mind sharing your machine details (RAM, GPUs)?