Media compression tool and media format change tool

Ideas
,
1

Hello,

My save file is becoming big because of all the media I add to it.

The medias are birth, mariage and death certificates saves as either .png or .jpg.

I have 4517 media for 8929 Mo.

The idea would be tools/plugins :

  • to convert files format inside of gramps (to lighter format)
  • to compress image inside of gramps
  • to trim image inside of gramps (again to reduce file size)

It would save a lot of time instead of doing it manually.

I have no idea if it’s even possible but I would like to know what you think of it.

Thank you

I don’t know what to to with this : (Please create and include the associated Feature Request number.)

2

You can’t compress an already compressed JPG or PNG much further using standard compression tools. Saving your database to the Gramps .gpkg (Gramps Package) archive format will compress the XML data itself, but it won’t reduce the file size of your images. In fact, due to file headers and archive metadata, packing already compressed JPGs and PNGs inside a zip container can sometimes even increase the total file size slightly.

However, trying to “micro-manage” and shrink your media files to save a few megabytes inside a database app is fundamentally the wrong approach for genealogy. Here is why:

  1. The 300 DPI Rule (Why Resolution is King)

In genealogy, most of our images are historical documents, certificates, and old church books containing handwriting (like gothic script or faded ink). If you reduce the pixel dimensions or drop the resolution (e.g., from 300 DPI down to 150 DPI), you are actually throwing away 75% of the pixel data! You might think it looks fine on a small screen, but the moment you need to zoom in to decipher a single faded name or date, the image will turn into unreadable mud. Once that data is stripped away, it’s gone forever.

  1. The Golden Standard for Archiving

If you want your research to survive the next 50 to 100 years, you should stick to open, lossless, and standardized archive formats. Personally, I never recommend WebP or other modern web-delivery formats for archival purposes. The only formats that truly matter for long-term preservation are:

  • TIFF / DNG (including Linear DNG): For high-resolution, lossless image and camera scans.
  • PDF (ideally PDF/A): For multi-page documents.
  • SVG: For graphic charts, coats of arms, and family trees (since it is XML/text-based and infinitely scalable).
  • Markdown (.md) / Plain Text: For all your notes, ensuring they can be read by any computer centuries from now.
  • PNG: As an absolute fallback for raster images if storage constraints force a compromise, since it is at least lossless.
  • Or any type of LaTeX documents.
  1. Storage is Dirt Cheap – Don’t Destroy Your Data

Instead of ruining the readability of your historical documents, solve the problem at the hardware level. Storage is incredibly affordable now.

The most sensible workflow is to export only your raw tree data as a tiny, lightweight .gramps file (without media). Keep your heavy, high-quality media folder entirely separate on your hard drive.

Then, buy two external 4TB or 8TB hard drives. Use one as your active working disk, and use a simple automation tool like Robocopy in Windows (or a basic bash script in Linux) to regularly mirror and backup your media folder to the secondary drive. It is incredibly simple, perfectly safe, and ensures your descendants will actually be able to read the documents you worked so hard to collect.

A Warning About Long-Term ZIP Storage:
Relying on big monolithic archive files like .gpkg (which is just a renamed ZIP) or standard .zip /.7z files for long-term storage is highly risky.
In compressed data streams, every single bit is critical because all data redundancy has been stripped away.
If a single bit flips on your hard drive (due to cosmic radiation, drive wear, or bit-rot) inside a compressed archive, the decompression algorithm will lose synchronization.
If that flip happens in the central index directory of the ZIP file, the entire archive becomes permanently corrupt and unreadable. You risk losing your entire media collection and database in a split second."

So, if you are creating long-term backups of your Gramps data, the best way to do it is to copy your media files to an external hard drive in their raw, open folder structure, and make a data-only export in XML that you store on the same disk. To be 100% sure, you should also save the installation file of the specific version of Gramps that you currently use. Normally, the latest version of Gramps will be enough, but having the exact executable saved on your backup disk ensures that your data can always be reconstructed, even decades from now.

Note: This text was written in Norwegian and translated to English and edited by my instructions for better readability by Google AI.

3

StoltHD, This was excellent advice. The only thing to add would be to examine future-proofing alternatives should the storage medium become out-dated or obsolete. Example, ZIP drives were once considered a standard archival medium.

4

Thanks a lot for the explanation and taking the time to respond.

I will stop saving as one .gpkg archive and begin saving tree data and media separately.

I am in France and most of my media is download from online departmental archive as .jpg

Do you have a recommendation on how to effectively convert them as .dng ?

5

The management of Media objects has always been a challenge.

Strictly specking, Gramps does not manage Media files… except for Backup and Import (Restore). It just keeps an index with thumbnails and metadata.

But you’re right. A few new Media Object management tools could offer some new capabilities and flexibiity. But we would need some new developer volunteers to buy into such a project. (Hopefully, without grabbing the attention of team members currently making so much progress in all the other areas.)

6

Gramps is not a photo editing program. There are good programs to manipulate and otherwise edit photos. Gramps already does what it needs to access these raw images and make them available to the user’s database.

7

When you download JPG or PNG files from online archives, there is no need to convert them. Just download them in the highest available quality. Once a JPG or PNG has already been created by the archive, converting it to TIFF or DNG will have no practical benefit for the image quality; it will only result in a much larger file size.

However, you should use TIFF or DNG when you scan or photograph images and documents yourself:

  • Standard Scanner Software: If your scanner software doesn’t support DNG, use 16-bit color TIFF—even if the original document is just black and white. Always scan at a minimum of 300 DPI. For clean, typewritten text, you can test 150 DPI, but make sure to lock the physical size dimensions to match the original document.
  • Using VueScan: If you use VueScan, I highly recommend exporting to DNG, or a dual setup of DNG + JPG (with JPG quality set to 80). You can then use the lightweight JPG files for daily work and publishing inside Gramps, while keeping the original DNG/TIFF files safely stored in your external archive folder. Keep in mind that Gramps does not natively render thumbnails for DNG files anyway.

PDF is for documents: Use PDF only for pure text documents, multi-page records, or finished books in the PDF/A format for long-term archiving. Make sure to use PDF version 1.7 or higher. Earlier PDF versions had a hard physical dimension limit of 200 inches (5.08 meters) in width or height. If you exported a massive family tree with thousands of people, it would quickly crash into this limit and get cropped. Version 1.7 utilizes a scaling factor called /UserUnit, which expands the maximum page dimension to several kilometers. This allows you to safely save gigantic chart reports that span several meters and contain tens of thousands of pixels in full resolution.

If you simply want to batch-convert or organize your existing file collection safely on the outside of Gramps, you can use these free utilities:

  • XnView MP / XnConvert: Excellent for a graphical user interface (GUI) approach. It handles over 500 image formats and lets you automate batch processes safely with a few clicks.
  • ImageMagick: The ultimate choice if you prefer the command line or want to script the entire workflow.

Note: This text was written in Norwegian and translated to English and edited by my instructions for better readability by Google AI.

8

You make a very good point, but we need to separate proprietary hardware formats (like the old Iomega ZIP drives) from standard hard drives and SSDs.

To be safe, you should never let a backup stay on a single drive for more than 5 to 10 years without replacing the hardware. This is true even for standard magnetic hard drives (HDDs), as mechanical parts seize up and lubricants degrade over time. Because we constantly migrate our archive folders to newer, larger drives every few years, format obsolescence is rarely an issue for standard external drives or SSDs today.

However, for optical media like CDs, DVDs, and Blu-ray discs, degradation is a massive problem. Standard burned discs often degrade within 5–10 years due to dye rot. Even specialized formats like the M-DISC (which uses a stone-like layer meant to last 1,000 years) require specific optical hardware in the form of newer BR-read/write devices.

My current hardware solution for future-proofing is to keep an external USB Blu-ray/DVD/CD Read/Write drive tucked away safely on the shelf. This ensures that even if future computers lack optical drives, I can always plug it in via a standard USB port to retrieve old archival data.

To be honest, I am both paranoid and a total tech hoarder, which helps a lot with future-proofing! For instance, in my home lab, I still keep an old Iwill Dual Xeon server (with a 16-port Areca PCI-X hardware RAID controller) ready for older enterprise storage.

Even crazier, I have an ancient ASUS motherboard with dual physical Socket 7 slots running genuine Pentium CPUs still up and running natively on Windows 2000.

I actually have a zip drive laying around with 5 disks… so a workstation with a parallell port in the back is a must… haha…

If you have the right legacy hardware on your shelf and a proper data migration routine, your genealogy data will never become obsolete.

Note: This text was written in Norwegian and translated to English and edited by my instructions for better readability by Google AI.

9

Are we talking at cross purposes?

I am talking about file management, not file editing. (Although more features for managing metadata… both inline and sidecar… would have value for automating the tracking and conservation of provenance data.)

There is a fundamental need to start partitioning the backup archives after surpassing a certain volume of Media files. (To limit the loss if an archive becomes corrupted and to create incremental backup efficiencies.) And value in automating the file storage organization instead of manually managing file naming and optimizing folder hierarchy.

Which is a subject that would need discussion.

Is meaningful file naming more important? There is a tension between human-readable filesystem organization and metadata-driven / database-driven organization. Which school of thought would the Gramps community prefer?

  • Filesystem-centric organization: meaning is carried in filenames and folder paths, so a person can browse the tree and infer what a file is.
  • Metadata-centric organization: filenames stay simple and unique, while descriptive information lives in metadata, tags, or a database that supports search and filtering.
  • Hybrid organization: a compromise where filenames and folders provide coarse structure, and metadata provides the detailed retrieval layer.
10

There is no real reason to zip down archive files anymore. In fact, you should never ever zip down archives for long-term storage.
Most modern media files are already compressed, so zipping them saves virtually no space. More importantly, if a zip archive gets corrupted, you risk losing everything.

This is also true if you split or partition a zip file into smaller parts; if just one part gets corrupted or goes missing, the entire archive becomes unrecoverable without very niche, specialized software tools and a whole lot of patience.

11

That was not what I am saying. The .gpkg is a backup archiving format with a lossless compression stored with a directory structure.

The current format includes a (doubly compressed) XML .gramps file and the Media objects.
I suggest that these would make more sense as two separate (but daisy-chained) files. And as Media Objects are added and the Tree data revised, a more efficient incremental backup could save a complete replacement .gramps archive, keep the daisy-chained media archive and add a delta file media file archive to the daisy-chain.

One the other hand, a FULL backup would take the daisy-chain back down to 2 items: a fresh .gramps archive and a freshly compressed set of Media objects.

Of course, incremental backup schemes are inherently complex. So so such ideas need discussion before a conclusion/consensus is reached.

12

Yes, that is exactly what you were saying. In your earlier post, you explicitly wrote: ‘There is a fundamental need to start partitioning the backup archives… to limit the loss if an archive becomes corrupted and to create incremental backup efficiencies.’

Splitting or partitioning zip-archives (or the Linux equivalents) is a relic from the floppy disk and tape backup era. Wrapping this outdated practice in fancy terms like ‘daisy-chained media archives’ and ‘delta files’ does not change the underlying structural flaws.

You don’t use ‘daisy-chains’ of files for incremental tracking. In actual systems engineering, you use an SQL database index to track and map file states and changes, keeping the underlying storage flat and independent.

Creating an application-layer dependency chain where one archive file relies on the integrity of another is a terrible practice for long-term preservation. If just one part or link in that file-chain suffers from bit-rot, the entire chain after it becomes permanently unrecoverable. Sane archiving relies on raw, open, flat file structures.

Let’s please drop this theoretical backup discussion and leave it out of this thread.

13

Monolithic backups in an age of possible terrabyte size collections of Media object resurrects the issue. Only instead of segmentation for small capacity floppy disk, CD-R, tape, or Zip drive; the scale is more immense.

14

After reading your responses I am glad I don’t have to convert my original .jpg files.

With this new information I realise converting inside gamps wasn’t really my problem anyway.

It can be done easily with other software. Add the new file in the same folder with the same file name but different extension.

The problem would have comes from manually changing each media extension in gramps.

So a plugin or tool to modify the link to the file with the new correct extension would be enough.

image
image1920×1080 315 KB

for example : selecting the files - right clic - change extension to and choose from there.

It would probably need an error message if the file with the new extension doesn’t exist.

I say this in case someone find this idea still usefull.

Thanks again for your responses