Need some JSON parsing tools

I am looking for some Tools that would import a JSON into a spreadsheet. (I am using LibreCalc. Excel apparently supports direct import and parsing of JSON into tables but LibreOffice does not. The old {prior to 5.2} Text versions could easily import as CSV.)

In particular, directly inspecting the various 5.2 and 6.0 gramps-project/addons JSONs would allow using them as a checklist for addons.

A lot of the plug-ins need improvements in the descriptions (such as mentioning Gramplets that are category specific or tool/report subcategories) or are missing key data… like the help_url, prerequisites, or author/maintainer data.

And localized addons JSONs identify terms/descriptions that are still in English.

I also need this kind of spreadsheet checklist to review my installed 5.2 addons compared to the 6.0 addons list. (Where I pipe a list of folders in my .gramps/gramps52/plugins folder to text file to populate another page in the spreadsheet.) Comparing would help identify which “undistributed addons” will need to be manually re-installed when Gramps switches over to 6.0

Likewise, reviewing the addons-xx.json files in the external projects (such as Isotammi/@kku 5.2 listings, GlopGlop/@grocanar 5.2 listings, JMichault/@jmichault 5.2 listings, @ztlxltl 5.2 listings) would help diagnose make.py and translations problems.

But I need a less laborious way to peruse the JSON data. (And one that doesn’t pass through the unanticipated blinders in the Gramps plug-in registration system.)

You can try to convert json to csv with python3 and pandas module:

import sys
from pathlib import Path
import pandas as pd

input_path = Path(sys.argv[1])
output_path = input_path.with_suffix('.csv')

df = pd.read_json(input_path)
df.to_csv(output_path.open(mode='w'), encoding='utf-8', index=False, header=True)

This script takes the path to the JSON file as the first argument and creates the CSV in the same folder.

1 Like

@emyoulation
What do you mean by privacy settings of my the repo? There are no hidden directories. You can find the listings directory here on the dist branch.

Thanks @jmichault ! I should be able to adapt that Python code to read the various language JSONs from all those GitHub repositories and compile a consolidated CSV.

And (sometime later) tackle writing a report to write addons_en (JSON and CSV) for the core plug-ins. That would let us review those built-ins for registrations shortfalls. We’ve missed the deadline for adding help_url and better descriptions to the 6.0 release. But the tweaked JSON could be used as a local “Project” with the Addon Manager for Built-In plug-ins. It wouldn’t have Install/Update capabilities, but it could be searched and have Wiki help buttons.

I was not able to browse or search for the listings folder due to what is ‘public’ in the GitHub settings.

Thanks for the direct link. Tweaked the original posting.

I just got a tips about this tool, never tried it myself:

Else, you have Oxygen.

There is also some addons for both VSC and Notepad++

This one is an open source command line tool:


For those that can navigate Python, you can use the python json library and panda to convert and write a json file…
Just don’t ask me how…


The two first cost a lot of money…


Oooh, but haven’t you used OpenRefine earlier?

You can easily import different formats and files into the same project and consolidate all the data into a single CSV file.

1 Like

a standard Gramps install doesn’t have pandas. So ran with your sample code and worked out a converter that work in the Python Evaluation gramplet and uses the Gtk file chooser.

Key .json files are the addons_xx.json files… but more and more Gramps files are switching to .json format. So such a tool will gain value in the future. I suppose the next step is to wrap it in a Gramplet shell. (Since it is likely that a series of conversions would performed… the addons JSON in english, french, german, et cetera… then Tools would be tedious. Learned that from the tedium of doing a series of WebConnection lookups from a context menu. Those lookups would be a lot more convenient as a Gramplet.)

I won’t be able to contribute it as an addon since Perplexity AI did most of the heavy lifting. (Although it took many iterations to get to code only using things that come with Gramps… like Gtk toolkit and importing only from already installed modules.)

import sys
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk

class JsonToCsvConverter(Gtk.Window):
    def __init__(self):
        Gtk.Window.__init__(self, title="JSON to CSV Converter")
        self.set_border_width(10)

        vbox = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6)
        self.add(vbox)

        self.input_button = Gtk.Button(label="Select Input JSON")
        self.input_button.connect("clicked", self.on_input_clicked)
        vbox.pack_start(self.input_button, True, True, 0)

        self.output_button = Gtk.Button(label="Select Output CSV")
        self.output_button.connect("clicked", self.on_output_clicked)
        vbox.pack_start(self.output_button, True, True, 0)

        self.convert_button = Gtk.Button(label="Convert")
        self.convert_button.connect("clicked", self.on_convert_clicked)
        vbox.pack_start(self.convert_button, True, True, 0)

        self.input_file = None
        self.output_file = None

    def on_input_clicked(self, widget):
        dialog = Gtk.FileChooserDialog(
            title="Select Input JSON File",
            parent=self,
            action=Gtk.FileChooserAction.OPEN
        )
        dialog.add_buttons(
            Gtk.STOCK_CANCEL, Gtk.ResponseType.CANCEL,
            Gtk.STOCK_OPEN, Gtk.ResponseType.OK
        )
        dialog.set_current_folder(sys.path[0])
        response = dialog.run()
        if response == Gtk.ResponseType.OK:
            self.input_file = dialog.get_filename()
            self.input_button.set_label(f"Input: {self.input_file.split('/')[-1]}")
        dialog.destroy()

    def on_output_clicked(self, widget):
        dialog = Gtk.FileChooserDialog(
            title="Select Output CSV File",
            parent=self,
            action=Gtk.FileChooserAction.SAVE
        )
        dialog.add_buttons(
            Gtk.STOCK_CANCEL, Gtk.ResponseType.CANCEL,
            Gtk.STOCK_SAVE, Gtk.ResponseType.OK
        )
        dialog.set_current_folder(sys.path[0])
        dialog.set_current_name("output.csv")
        response = dialog.run()
        if response == Gtk.ResponseType.OK:
            self.output_file = dialog.get_filename()
            if not self.output_file.endswith('.csv'):
                self.output_file += '.csv'
            self.output_button.set_label(f"Output: {self.output_file.split('/')[-1]}")
        dialog.destroy()

    def on_convert_clicked(self, widget):
        if not self.input_file or not self.output_file:
            dialog = Gtk.MessageDialog(
                transient_for=self,
                flags=0,
                message_type=Gtk.MessageType.ERROR,
                buttons=Gtk.ButtonsType.OK,
                text="Please select both input and output files."
            )
            dialog.run()
            dialog.destroy()
            return

        import json
        import csv
        from collections import OrderedDict

        try:
            with open(self.input_file, 'r') as json_file:
                data = json.load(json_file)
            
            fieldnames = set()
            for item in data:
                fieldnames.update(item.keys())
            fieldnames = list(fieldnames)

            with open(self.output_file, 'w', newline='') as csv_file:
                writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
                writer.writeheader()
                for row in data:
                    writer.writerow(OrderedDict((k, row.get(k, '')) for k in fieldnames))
            
            dialog = Gtk.MessageDialog(
                transient_for=self,
                flags=0,
                message_type=Gtk.MessageType.INFO,
                buttons=Gtk.ButtonsType.OK,
                text="Conversion completed successfully."
            )
            dialog.run()
            dialog.destroy()
        except Exception as e:
            dialog = Gtk.MessageDialog(
                transient_for=self,
                flags=0,
                message_type=Gtk.MessageType.ERROR,
                buttons=Gtk.ButtonsType.OK,
                text=f"An error occurred: {str(e)}\nError type: {type(e)}"
            )
            dialog.run()
            dialog.destroy()

win = JsonToCsvConverter()
win.connect("destroy", Gtk.main_quit)
win.show_all()
Gtk.main()

An update. This experiment had some good results.

It allowed comparing the 154 registered addon plug-ins in the new Gramps 6.0 addons-source repository to discover that only 31 would be able to take advantage of the new extension of Help that @Nick-Hall added in the 6.0 beta.

@dsblank offered to bring the registrations up to snuff if provided a list of the appropriate documentation to be linked.

With the CSV list in-hand (generated by the above script) and few hours of cross-referencing, Doug had a list and did the updates over the next couple days. Then @GaryGriffin validated the changes, applied the updates… et voilà!

Bottom line, starting with Gramps 6.0 (beta), users will have Wiki buttons for all the 154 Addon Manager listings. So they can quickly read docs before deciding whether to install. In the Plugin Manager enhanced, they will be able to look up docs for the plugins that are built-in as well as Addon.


And, rather than use the Tools and Reports menus that are sometime too terse to hint which does what, users can launch the Report Selection and Tool Selection dialogs from the toolbar and use those enhanced dialogs to see the extended description and a button to quickly find the docs. (The Report Selection dialog is very helpful when you want to step through the Reports. It keeps track of your progression.)

1 Like