Setup of VmBox Mint Gramps Sandbox

Development
1

I am trying to setup a Gramps development sandbox in a VMBox environment, running Mint 20.3 and using the community edition of PyCharm. My Host is Win 10 64-bit.
At this point, I have cloned the current version of Gramps from Github and have (successfully, I think) ‘setup.py build’
Now when I try to launch Gramps.py in my testing root, I get one message

/home/arnold/PycharmProjects/gramps/venv/bin/python /home/arnold/PycharmProjects/gramps/Gramps.py 
Gtk-Message: 14:06:07.785: Failed to load module "xapp-gtk3-module"

although Gramps still starts but complains about “GExiv2 not loaded” and “OsmGpsMap module not loaded”
From previous experience & new searches on the 'net, I have installed the modules recommended in the VMBox, exited and restarted PyCharm, but the errors still pop up.
Since the metadata for images are of particular interest I cannot ignore these errors.
Any pointers or explanations will be greatly appreciated

2

Most missing modules can easily be installed with the Synaptic package manager. If you use that, you can search for gexiv2 and omsgpsmap, and it will show you a couple of options, from which you need the ones that start with gir-1.2.

You can also use the terminal for this, by typing sudo apt install gir-1.2-gexi and then using the Tab key for command completion. Same for gir-1.2-osmgps.

3

Thank you, Enno
I am sure I had been doing this sometime ago, but could not find the reference in my old mail account.
This gives me a clean start when I run Gramps from the command line in my Mint box, but I still have to sort out how to make it all work when I try to run Gramps inside the virtual environment with PyCharm and possibly some other issues down the road for some of the other packages.

I had been looking at the Github readme which spells out which packages are needed, but for a newb like myself, knowing which packages are essential is only half the battle, especially within a development environment, such as Eric or PyCharm.

4

I have never worked with PyCharm, but I have used other JetBrains products in a professional setting, sometimes running stand-alone, but most times as plug-ins in Visual Studio. And that was all on Windows.

Because of this, I also don’t know much about that xapp-gtk3, but most times Google should be your friend, for cases like this, or a search on this forum. I often see that our wiki is getting outdated, and I am too lazy to work on that. Some experimentation when I set up Gramps on a new platform is often enough, and I have a Gramps 5.1.6 built from source on the machine where I am typing this, which is a PC running LMDE 5. I migratied from LMDE 2, and like de Debian Edition better than the standard ubuntu based Mint, largely because it’s faster. I have Linux Mint 21 MATE on my laptop though.

Have you considered upgrading to Mint 21? And would you consider Visual Studio Code instead of PyCharm. I use Visual Studio Code a lot for other tricks, like loading a Gramps XML file into a C# program, with which I can do lots of interesting things.

You can find more info about Visual Studio Code and Python here:

5

Created a new VMBox for Mint 21, but the current version lacks the ability to adjust the virtual screen size, at least with my hardware, which makes it very unattractive because it limits the display to a rather small ‘screen’ and makes it hard to work with.
So I installed VSCode on the older Mint 20.3, but, even though I have used VSCode under Windows it does not seem all that much more intuitive than PyCharm to get it to set up with all of the proper modules installed.

As it is, I was planning to keep that specific VMBox for Gramps development only, so I suppose I could do with a plain global Python environment, but even that has a seemingly steep learning curve :frowning:

6

OK, I understand, and that learning curve looks like something that you have to deal with yourself.

Is there anything else that you need help for?

7

Thank you, yes, I am used to climbing learning curves, but always appreciate it when someone shows me some hand grips and foot supports :slight_smile:

As it is, right now, I have a couple of puzzling questions:

Since my plain Gramps installation finds the GExiv2 and OsmGps modules because they are installed via apt, why can Gramps, running within PyCharm (or VSCode) not find them?
and what do I need to do so it can find them?

Do I need to adjust some path variables, move/copy files? & which ones?

As I am mostly a C++ & WIndows guy, Linux & Python are still a bit of a ‘magic’ kingdom

8

One possible cause might be the way that you installed Gramps. How did you do that?

I’m asking, because I know that a Gramps installed with the software manager, or by opening a .deb file downloaded from our GitHub site uses other paths than a Gramps that you build and install from source.

I grew up with Unix, in the mid 80’s, so I’m used to the command line, and I have a Gramps config, in which I have all sources in a folder called /home/enno/gramps, which I created by cloning my own Gramps repo, which is on my GitHub account named ennoborg. This repo has one patch that I made to make the Consanguinity Gramplet work, and is otherwise the same as the main repo.

When I’m in working with the command line inside /home/enno/gramps, I can build Gramps with “python3 setup.py build”, and run it with “python3 Gramps.py”, and after installing all missing libraries, with synaptic, or apt, this all goes without warnings.

Once this worked OK, I installed it with “sudo python3 setup.py install”, and verified that changing to /home/enno and then typing “gramps” worked without warnings too. And I have no special tricks there.

Today, I also tried opening that /home/enno/gramps folder with Visual Studio Code, and loading the Gramps.py file in the editor. Visual Studion then asked to install the Python extensions, and once I had those installed, I could run and debug it in the same way as I run and debug C# programs, all without warnings. And when stepping into the main procedure, and using F10 to step further, I could see that Gramps is building its own paths to find everything, and also noticed that by pressing F5, I could run it with my main tree in the way I’m used to.

When I run “env”, I see no special variables.

This is on my LMDE 5 setup, which is Debian based, but it should work pretty much the same on the ubuntu based Mint 20 or 21. I’m not using a VM, except to run Windows inside Mint, but that too should not make a difference.

9

Once you pointed it out, it seems quite reasonable that installing a genealogy ‘working’ copy via a deb file will differ from cloning the source directly from Github.

While I do have a Github account, I am assuming that, at least at the very start, I would be able to just work with a local clone of the 5.1.x branch?

I am not all that unfamiliar with the command line as I grew up with DOS and worked for some time with Solaris. Still, it seems that working with a GUI one has more clues visible for the available options and it does seem easier to navigate than a command line :slight_smile:

Working with MSVC has spoiled me and got me away from all of the makefile intricacies.

My main interest here is trying to make image metadata at least ‘visible’ within Gramps.

As part of that, I have built up some utilities to work with that data in C++ and have made a feeble start to port some of those to wxPython - all of that is happening with VS Code, but in a Windows environment.
One reason for moving to (wx)Python was because I wanted to move to a more portable environment for my utilities
A second one was to see if there might be a way for me to help Gramps to deal with that data better.

10

If you want, you can indeed work with a local clone of the main repo. If you do that, you won’t be able to send a pull request, but you can always attach a .patch file to an issue in Mantis, so that an authorized developer can evaluate that.

Can you explain what you mean with that image metadata? Do you want to make it visible with a Gramplet? Or maybe add metadata to Gramps attributes?

I am assuming that the utilitues that you want to build are meant to work independent of Gramps itself. Is that right?

11

We’re assuming that you’ve explored the Image Metadata Gramplet add-on.

Even if it doesn’t do all that you want, it is good sample code to build upon.

And I like using the added features in @cdhorn 's experimental LinkedView modes in combination with such metadata add-ons.

12

Certainly, until I am more familiar and confident in my skills related to Gramps & Python, I want to keep things local.

As for metadata, I realized some time ago that I needed to make some of the relevant data - persons (shown or described/listed) in an image, dates, locations etc - part of the image and that is when I discovered image metadata.

Once I started working with Gramps, I wanted/needed to at least inspect and verify the data associated with some of the records and files (in the DB) and with that saved in the metadata of the associated media.

Initially, I was somewhat disappointed in how little of the data was actually displayed/available within Gramps, but my unfamiliarity with both Python & Gramps left me little choice but to live with things as they were.

Currently, there are 2 gramplets which at least partially try to address this issue, but neither really meets my needs and both need more work, IMNSHO :slight_smile:

Still, I needed to at least be able to inspect the data present, and eventually to also be able to edit and add my own data as needed.

Being a C++ wxWidgets fan and only having the then much better debugger/IDE of MSVC, I stuck with Windows for quite some time. Now I would prefer to move away from it. Meanwhile, tools and languages also evolved and I now feel much more inclined to move to wx(Python), but porting all of the features of the current Windows utilities will take much more time.

My current thoughts on Gramps and metadata are that Gramps ought to be able to at the very least show all data present in the media files.

I don’t think it would be too good of an idea to expect Gramps to also modify/edit the metadata. That would be too far outside the scope of the program and would only unnecessarily complicate things.

IMO, all editing etc of metadata ought to be done outside of Gramps, as long as Gramps will recognize that the image file has changed and is able to at least make the user aware of this fact.

I realize that this may not be as easy and simple as it may seem, because, at present, I have no idea how much of the image files are part of the checksum or other hash used to compare/verify image files in the Media verify tool.

But, at least, Gramps ought to be able to simply refresh the metadata when it detects a change in a media file.

There are a number of other issues, but this is getting long enough, so I’ll save those for another time :slight_smile:

13

I’m going to disagree with you on the subject of not editing metadata within Gramps.

Primarily because I want to be able to use the existing data in the trees to tag photos with people places & events… without redundantly transcribing data.

For instance, the LinkedView feature allows me to calculate the age of the people against that of the photo. I’ve already be able to determine that some people were mis-identified in some holiday photos and corrected those.

I do wish the photo Tagging Gramplet allowed to tag other objects that People. Certainly Places & Events would be appropriate. And linking the Photographer would often be nice.

14

Yes I have, but …

One of the issues is that, from what I understand right now, is that it uses and depends on a library derived from the exiv2 project and how up-to-date either one of those is, I am not sure right now.

There have been major changes at exiv2 and I have not kept up with that project.

As for the linked view add on?, I remember you mentioning it some time ago and I do recall having a quick once over at the time.

But my unfamiliarity with all things Gramps and Python made easy for me to leave that to one side until I get a better handle on a lot more things in this context…:slight_smile:

15

At the back of my mind I sort of have to agree with you, but practically speaking I don’t see any one having the time and expertise to add something like it to Gramps.

Sure, the convenience of doing it all in one app looks attractive, but one of the biggest problems, aside from making Gramps even bigger, is that there is, to my knowledge, no standard setting out which tag should hold which data.

It seems there have been some starts at some sort of standard, but nothing has matured, AFAIK

There is: https://www.facebook.com/groups/www.fhmwg.org but it has been so quiet, too quiet, for my liking.

One of my favorite libraries at this stage is Exiftool. My Windows app interfaces to it, simply because, currently it is the most up-to-date and actively developed library in the metadata domain.
But, on last count, that library list some 27 000+ tags

This means that if one would want to have Gramps edit any metadata, most folks would expect that their favorite field/tag is supported.

Hence Gramps would need a way to make these choices user selectable. Granted many tags are most unlikely to be of interest, but even so, that leaves a big pile.

Add to that, that none of the currently defined tags were designed with genealogy needs in mind and most of the ones which might seem useful, such as the ones meant for time, dates, location, generally have several variations on the exact meaning.

There is another issue at the bottom of all of this and I have not seen it discussed anywhere, though I have not really looked very hard.

It is this:

Each image file is composed of several sections. Some of these contain the actual image data - which is encoded in different ways, depending on both the file type/extension and other specific options, such as compression type and ratio, specially for jpg images.

The metadata typically resides in one or more separate chunks. When apps edit the metadata, they are free to move these chunks about in the final image file written to disk. What is worse, not all available utilities offered for editing the metadata follow all of the rules set out by the standards.

Based on my current understanding of the media verify tool, it computes some sort of checksum over the whole file. This means that every time you modify even one byte in the metadata, that checksum will change.

How Gramps and each user will handle this aspect, we’ll have to see.Should Gramps compute several checksums? one for the actual image data and a separate one for metadata?

My current preference would be to make Gramps display all metadata in an image and allow the user to inspect it, copy it to another app via the clipboard for updating and ensure Gramps reloads the image quickly so that the user can verify the result.

That is what I am aiming for, but of course, those are simply intentions right now, though I am hoping that I will be able to progress along the way well enough to make it useful with time.

Partly because I do not want to be bound by choices made already in the currently available plugins, I intend to start from scratch and, FWIW, hope to be able to use Exiftool as the backend

That is where I am at today :slight_smile:

16

When I started using Gramps, somewhere around 2010, the only way to make hacking easy was to create a multiboot setup, to run ubuntu next to Windows, and although I have moved to Mint now, this is still the way I work. Today however, things may be a bit easier, because you can install a Windows subsystem for Linux, so that you can run bash in an easy way, without relying on MinGW.

But even with WSL available, I still prefer to work with Linux, because it is way more consistent than Windows, is much easier to update, and has all the commands that I learned in the mid 80’s. It’s also way more transparent to me, as a developer. I must add that since about 2000, I have worked with Windows clients running Eclipse to have an IDE that worked well with the languages that I used then, which were C++ and TCL.

Are you planning to move to Linux too, and port all C++ applications that you made to it, using wxWindows? I am quite curious about the tools that you made, as far as they are genealogical, because I’m always interested in expanding the ecosystem here.

One problem with C++ though is, that it’s not easy to access the Gramps database with that, because it contains pickled Python objects, so I can imagine that you’d move to wxPython for that, if you like the wxWidgets application model.

Learning that model would be nice for me too, because I find Microsoft’s WPF quite a hassle, and it doesn’t run on Linux anyway.

Do you have any sources on the web that I can experiment with?

17

Years ago & at work, I had to/got a chance to work with MSVC & C++. Back then, and even now, I have not found a C++ IDE & debugger which matches MSVC and it has gotten better since I started.

But now, I have realized that I want/need to move along and into the Linux world. Even in the past I had been occasionally wanting to make some utility or other available for my son who lives in the Linux world and it was always a bit of a tedious chore. As I am, by now, relatively proficient with wxWIdgets, I decided to move to the more OS-agnostic world of wxPython etc.

But, that means a new learning curve and so, right now, very little of my C++ stuff has really made it to wxPython. In fact, I have just now taken up an old(er) utility to make some of the intended improvements and so it goes back and forth.

To boot, in trying to move up to Mint 21, I have caused myself a bit of trouble and it has taken me more than a day to sort of recover. Fortunately, my backup system is relatively robust and I think I have lost very little of the things which were in ‘development’ and hence in a state of flux.

None of my wxPython stuff has matured enough to make it to the 'net, mainly, because I am also trying to move along with my genealogy work and specifically my attempts to make the metadata viewable within Gramps.

A somewhat older version of the main app for this is available at http://wxutils.com/wxeti.php and it is the one I am currently (and have for the past few month) working on to add a few more features and fix some more of the warts I want to remove.

You can have a look at it and let me know what you think. If it is of interest, I will make an effort to package the more recent version(s) as they become stable enough.

Naturally, that utility currently only runs under Windows and needs a copy of the Exiftool utility available. Hopefully, the readme and help files give enough information to get started. If not, feel free to let me know and I’ll see what needs to be fixed or added.

18

OK, thanks. I tried both tools, and saw a lot of interesting data in some old pictures.

For wxWidgets however, it looks like it’s better for me to find soms example sources on the web myself. I’m the kind of person that learns best by example, and I really like to have something that works on Linux and Windows that does not depend on the standard output that I now see in my VS code debug window.

19

Not sure which tools you are referring to. Yes, Exiftool is only a command line tool.

My wxETI should display the data in some of the tool windows tabs, particularly in the Metadata tabs.

My goal is to port wxETI to wxPython and I have started on this, but just the bare minimum, more or less a proof of concept at present.

Once I got that far, I got sidetracked into trying to get a test version of Gramps going - which is where this thread started :slight_smile:

In the process, I realized I needed more features in wxETI, which I had planned on, but put off till I really needed them – which is now.

But to port all of the features I have already working to wxPython, would take me too far off testing metadata in Gramps because I am also learning (wx).Python at the same time.

My plan is to make wxETI the utility I spoke of which allows me to see all the data. The current version is intended to also write new data to the supported image files, but that feature has had the least testing done, so be careful if you try that. And the version on the web, is several versions behind my current working copy.

20

OK, I understand. I asked this primarily, because I was thinking about a toolkit to add a GUI to my own C# applications. And wxWidgets would be totally new to me, so it would need to be very good to invest time in, knowing that Gramps is based on GTK, and it is also possible to use that with C#.

Do you have that test version going now?