SuperTool Script: Auto-Generating Citations with Media Linking

I want to share another SuperTool script with the Gramps community. I don’t think this script will be particularly useful in its current form, but some of its elements and the general idea might be helpful to someone.

First, I’d like to say a few words about my workflow, which I have fine-tuned for myself. I constantly modify, adapt, and improve it. But at this moment, it looks like this:

  1. First, I review an archival case and identify all the records that interest me. Typically, one case contains 10-50 records that I need to add to my Gramps database.
  2. I record the found cases in a Google spreadsheet. I’m not working alone; there are several of us, so this is the best place for collaboration.
  3. Recently, I optimized my Google spreadsheet. Now, it automatically generates correct document names for me. For example: "744-1-520_metrik_birth_{place}_1918-01-18_p249". All my documents have similar naming conventions. Maybe some people don’t follow such a structure, but I decided it’s better to have a consistent naming system.
  4. I use the generated names to rename all downloaded documents that interest me. Most of these come from FamilySearch.
  5. I add all these documents to Gramps and assign each of them a FamilySearch URL attribute.
  6. The next step is to create citations. I used to do this manually. Now, the SuperTool script (included below) handles this for me. For this script to work, I pass it a JSON file generated from the same Google spreadsheet that created the file names. This table also contains dates, page numbers, and record numbers, so it can generate a JSON file accordingly.
  7. After adding the JSON file, I select the media I just added and run the script.

As a result, I get all the citations I need. Each iteration of this process saves me about 10-20 minutes, depending on the number of records found. Right now, my database contains 812 archival cases that I have already reviewed. If I had used this approach from the beginning, the total time savings would have been around 135-270 hours. But better late than never!

I’m sharing this script as is, exactly as I use it for my tasks. :rocket:

[Gramps SuperTool script file]
version=1

[title]
Import citations based on media file descriptions

[description]
WARNING: Despite the setting of the [commit_changes] section, citations will still be created. Do not rely on this option to prevent changes.

This script automates the creation of citations for newly added media documents that the user selects. Only the selected media objects will be considered for citation creation. Unselected media will not be processed.

A citation will only be created if the media file's description (desc) exactly matches a filename entry in the provided JSON dataset. The script associates the media file with a citation referencing a predefined source, ensuring that document-based citations are generated efficiently. Additionally, the citation will include the relevant page number and record reference if available.

[category]
Media

[initial_statements]
import json
from gramps.gen.lib import Citation, MediaRef, Date

# Predefined source Gramps ID for citations
source_gramps_id = "S0408"
# Confidence level to be assigned to each citation
confidence_level = 3
# Counter to track the number of citations created
counter = 0

# JSON dataset containing references to media file descriptions
json_data = '''
{
  "data": [
    {
      "page": "249-250",
      "record": "8",
      "filename": "744-1-520_metrik_birth_1918-01-18_p249-250",
      "date": "1918-01-18"
    },
    {
      "page": "253",
      "record": "12",
      "filename": "744-1-520_metrik_birth_1918-02-03_1918-02-07_p253",
      "date": "1918-02-03"
    },
    {
      "page": "253",
      "record": "13",
      "filename": "744-1-520_metrik_birth_1918-02-03_1918-02-07_p253",
      "date": "1918-02-07"
    },
    {
      "page": "255",
      "record": "19",
      "filename": "744-1-520_metrik_birth_1918-02-16_p255",
      "date": "1918-02-16"
    },
    {
      "page": "256",
      "record": "20",
      "filename": "744-1-520_metrik_birth_1918-02-18_1918-02-19_p256",
      "date": "1918-02-18"
    },
    {
      "page": "256",
      "record": "21",
      "filename": "744-1-520_metrik_birth_1918-02-18_1918-02-19_p256",
      "date": "1918-02-19"
    },
    {
      "page": "257",
      "record": "23",
      "filename": "744-1-520_metrik_birth_1918-02-22_p257",
      "date": "1918-02-22"
    },
    {
      "page": "258",
      "record": "27",
      "filename": "744-1-520_metrik_birth_1918-03-09_p258",
      "date": "1918-03-09"
    },
    {
      "page": "261",
      "record": "27",
      "filename": "744-1-520_metrik_birth_1918-03-28_p261",
      "date": "1918-03-28"
    },
    {
      "page": "263",
      "record": "40",
      "filename": "744-1-520_metrik_birth_1918-04-08_p263",
      "date": "1918-04-08"
    },
    {
      "page": "264",
      "record": "29",
      "filename": "744-1-520_metrik_birth_1918-04-08_p264",
      "date": "1918-04-08"
    },
    {
      "page": "266",
      "record": "44",
      "filename": "744-1-520_metrik_birth_1918-04-17_1918-04-24_p266",
      "date": "1918-04-17"
    },
    {
      "page": "266",
      "record": "45",
      "filename": "744-1-520_metrik_birth_1918-04-17_1918-04-24_p266",
      "date": "1918-04-22"
    },
    {
      "page": "266",
      "record": "47",
      "filename": "744-1-520_metrik_birth_1918-04-17_1918-04-24_p266",
      "date": "1918-04-24"
    },
    {
      "page": "268",
      "record": "36",
      "filename": "744-1-520_metrik_birth_1918-05-12_p268",
      "date": "1918-05-12"
    },
    {
      "page": "270",
      "record": "55",
      "filename": "744-1-520_metrik_birth_1918-05-23_p270",
      "date": "1918-05-23"
    },
    {
      "page": "271",
      "record": "59",
      "filename": "744-1-520_metrik_birth_1918-05-31_p271",
      "date": "1918-05-31"
    },
    {
      "page": "274",
      "record": "47",
      "filename": "744-1-520_metrik_birth_1918-06-25_p274",
      "date": "1918-06-25"
    },
    {
      "page": "275",
      "record": "50",
      "filename": "744-1-520_metrik_birth_1918-06-30_p275",
      "date": "1918-06-30"
    },
    {
      "page": "276",
      "record": "54",
      "filename": "744-1-520_metrik_birth_1918-07-01_p276",
      "date": "1918-07-01"
    },
    {
      "page": "279",
      "record": "74",
      "filename": "744-1-520_metrik_birth_1918-07-29_p279",
      "date": "1918-07-29"
    },
    {
      "page": "283",
      "record": "81",
      "filename": "744-1-520_metrik_birth_1918-08-15_1918-08-21_p283",
      "date": "1918-08-15"
    },
    {
      "page": "283",
      "record": "82",
      "filename": "744-1-520_metrik_birth_1918-08-15_1918-08-21_p283",
      "date": "1918-08-21"
    },
    {
      "page": "285",
      "record": "88",
      "filename": "744-1-520_metrik_birth_1918-09-05_p285",
      "date": "1918-09-05"
    },
    {
      "page": "286",
      "record": "68",
      "filename": "744-1-520_metrik_birth_1918-09-06_p286",
      "date": "1918-09-06"
    },
    {
      "page": "287",
      "record": "70",
      "filename": "744-1-520_metrik_birth_1918-09-08_1918-09-14_p287",
      "date": "1918-09-08"
    },
    {
      "page": "287",
      "record": "93",
      "filename": "744-1-520_metrik_birth_1918-09-08_1918-09-14_p287",
      "date": "1918-09-14"
    },
    {
      "page": "288",
      "record": "96",
      "filename": "744-1-520_metrik_birth_1918-09-17_p288",
      "date": "1918-09-17"
    },
    {
      "page": "289",
      "record": "98",
      "filename": "744-1-520_metrik_birth_1918-09-20_p289",
      "date": "1918-09-20"
    },
    {
      "page": "290",
      "record": "74",
      "filename": "744-1-520_metrik_birth_1918-09-25_1918-09-26_p290",
      "date": "1918-09-25"
    },
    {
      "page": "290",
      "record": "75",
      "filename": "744-1-520_metrik_birth_1918-09-25_1918-09-26_p290",
      "date": "1918-09-26"
    },
    {
      "page": "290",
      "record": "100",
      "filename": "744-1-520_metrik_birth_1918-09-25_1918-09-26_p290",
      "date": "1918-09-26"
    },
    {
      "page": "291",
      "record": "103",
      "filename": "744-1-520_metrik_birth_1918-09-27_p291",
      "date": "1918-09-27"
    },
    {
      "page": "292",
      "record": "106",
      "filename": "744-1-520_metrik_birth_1918-10-02_p292",
      "date": "1918-10-02"
    },
    {
      "page": "293",
      "record": "77",
      "filename": "744-1-520_metrik_birth_1918-10-06_p293",
      "date": "1918-10-06"
    },
    {
      "page": "294",
      "record": "80",
      "filename": "744-1-520_metrik_birth_1918-10-07_1918-10-08_p294",
      "date": "1918-10-08"
    },
    {
      "page": "294",
      "record": "111",
      "filename": "744-1-520_metrik_birth_1918-10-07_1918-10-08_p294",
      "date": "1918-10-07"
    },
    {
      "page": "296",
      "record": "84",
      "filename": "744-1-520_metrik_birth_1918-10-14_1918-10-15_p296",
      "date": "1918-10-15"
    },
    {
      "page": "296",
      "record": "113",
      "filename": "744-1-520_metrik_birth_1918-10-14_1918-10-15_p296",
      "date": "1918-10-14"
    },
    {
      "page": "297",
      "record": "116",
      "filename": "744-1-520_metrik_birth_1918-10-18_p297",
      "date": "1918-10-18"
    },
    {
      "page": "298",
      "record": "119",
      "filename": "744-1-520_metrik_birth_1918-10-24_p298",
      "date": "1918-10-24"
    },
    {
      "page": "303",
      "record": "130",
      "filename": "744-1-520_metrik_birth_1918-11-09_p303",
      "date": "1918-11-09"
    },
    {
      "page": "304",
      "record": "98",
      "filename": "744-1-520_metrik_birth_1918-11-13_p304",
      "date": "1918-11-13"
    },
    {
      "page": "305",
      "record": "100",
      "filename": "744-1-520_metrik_birth_1918-11-15_p305",
      "date": "1918-11-15"
    },
    {
      "page": "311",
      "record": "112",
      "filename": "744-1-520_metrik_birth_1918-12-09_p311",
      "date": "1918-12-09"
    },
    {
      "page": "348",
      "record": "8",
      "filename": "744-1-520_metrik_marriage_1918-01-21_1918-01-28_p348",
      "date": "1918-01-26"
    },
    {
      "page": "348",
      "record": "9",
      "filename": "744-1-520_metrik_marriage_1918-01-21_1918-01-28_p348",
      "date": "1918-01-28"
    },
    {
      "page": "354",
      "record": "37",
      "filename": "744-1-520_metrik_marriage_1918-02-11_p354",
      "date": "1918-02-11"
    },
    {
      "page": "354",
      "record": "38",
      "filename": "744-1-520_metrik_marriage_1918-02-11_p354",
      "date": "1918-02-11"
    },
    {
      "page": "359",
      "record": "62",
      "filename": "744-1-520_metrik_marriage_1918-05-20_p359",
      "date": "1918-05-20"
    },
    {
      "page": "362",
      "record": "81",
      "filename": "744-1-520_metrik_marriage_1918-06-13_1918-09-24_p362",
      "date": "1918-09-24"
    },
    {
      "page": "367",
      "record": "3",
      "filename": "744-1-520_metrik_death_1918-01-10_p367",
      "date": "1918-01-10"
    },
    {
      "page": "368",
      "record": "4",
      "filename": "744-1-520_metrik_death_1918-01-19_1918-01-24_p368",
      "date": "1918-01-19"
    },
    {
      "page": "368",
      "record": "5",
      "filename": "744-1-520_metrik_death_1918-01-19_1918-01-24_p368",
      "date": "1918-01-24"
    },
    {
      "page": "371",
      "record": "10",
      "filename": "744-1-520_metrik_death_1918-03-21_p371",
      "date": "1918-03-21"
    },
    {
      "page": "374",
      "record": "15",
      "filename": "744-1-520_metrik_death_1918-05-29_p374",
      "date": "1918-05-29"
    },
    {
      "page": "379",
      "record": "35",
      "filename": "744-1-520_metrik_death_1918-08-14_p379",
      "date": "1918-08-14"
    },
    {
      "page": "382",
      "record": "30",
      "filename": "744-1-520_metrik_death_1918-09-18_p382",
      "date": "1918-09-18"
    },
    {
      "page": "386",
      "record": "39",
      "filename": "744-1-520_metrik_death_1918-10-09_p386",
      "date": "1918-10-09"
    },
    {
      "page": "389",
      "record": "55",
      "filename": "744-1-520_metrik_death_1918-10-20_p389",
      "date": "1918-10-20"
    },
    {
      "page": "390",
      "record": "59",
      "filename": "744-1-520_metrik_death_1918-10-22_1918-10-23_p390",
      "date": "1918-10-22"
    },
    {
      "page": "390",
      "record": "60",
      "filename": "744-1-520_metrik_death_1918-10-22_1918-10-23_p390",
      "date": "1918-10-23"
    },
    {
      "page": "396",
      "record": "80",
      "filename": "744-1-520_metrik_death_1918-12-11_p396",
      "date": "1918-12-11"
    }
  ]
}
'''

# Load the JSON data into a Python dictionary
data_list = json.loads(json_data)["data"]

def create_citation_from_media():
    """
    Creates a citation for a media object if its description (desc) matches a filename in the JSON dataset.
    """
    media_desc = obj.get_description()  # Retrieve the media object's description

    # Fetch the source object using the predefined Gramps ID
    source = db.get_source_from_gramps_id(source_gramps_id)
    
    if not source:
        print(f"❌ Source with ID {source_gramps_id} not found!")
        return
    
    # Get the handle of the source object to reference it in the citation
    source_handle = source.handle
    global counter  # Reference the global counter for successful citations
    
    # Iterate over the JSON entries to find a matching filename
    for entry in data_list:
        filename = entry["filename"]
        page = entry["page"]
        record = entry.get("record")  # Optional field
        date_str = entry.get("date")  # Optional field

        # Check if the media description matches the filename in JSON
        if media_desc == filename:
            # Create a reference to the media object
            media_ref = MediaRef()
            media_ref.ref = handle  # Link the media object to the citation

            # Create a new citation object
            citation = Citation()
            citation.set_reference_handle(source_handle)  # Attach the source
            
            # Format the page reference, including record number if available
            citation.set_page(f"p. {page}" if not record or record == "null" else f"p. {page}, rec. {record}")
            
            # Associate the citation with the media object
            citation.set_media_list([media_ref])
            citation.set_confidence_level(confidence_level)  # Assign confidence level
            
            citation_date = None  # Ensure the date variable is initialized
            if date_str:
                try:
                    # Parse the date string (YYYY-MM-DD)
                    year, month, day = map(int, date_str.split("-"))
                    citation_date = Date(year, month, day)  # Create a Gramps date object
                    
                    if citation_date:
                        citation.set_date_object(citation_date)
                except ValueError as e:
                    print(f"⚠️ Error parsing date '{date_str}': {e}")  # Handle invalid dates
            
            # Add the citation to the database and retrieve its handle
            citation_handle = db.add_citation(citation, trans)

            # Verify that the citation was successfully added
            if not citation_handle:
                print(f"❌ Error: Citation did not receive a handle! {entry}")
            else:
                # Commit the changes and increment the counter
                counter += 1
                print(f"✅ {counter}) Citation created for media: {media_desc}, handle: {citation_handle}")

[statements]
create_citation_from_media()

[scope]
selected

[unwind_lists]
False

[commit_changes]
True

[summary_only]
False

I want to add a few words of my thoughts.
I would really like to see all of us moving toward saving human time. It doesn’t matter how much time we have, but I’m sure we won’t be bored if processors take over at least the non-intellectual work for us. As we know, 90-95% of all work is non-intellectual. So why not delegate at least some of it to electronics? Instead, we could focus our efforts on something truly important, where our involvement is irreplaceable.

I could see this being evolved into a valuable tool for attributing citations to a session’s worth of created or edited data.

It is likely that all changes within a certain timeframe will be from a single reference document.

Since Gramps does not have Feature to auto-cite new (creation timestamp) or edited (last change timestamp) primary (or secondary?) objects with a “current reference” Citation, a retro-active (selectable?) mass citing tool would be a big time-saver.

1 Like