@Nick-Hall has there ever been any discussion of possibly splitting the Gramps code out into a separate set of repositories and packages such as gramps-core, gramps-cli and gramps-ui maybe to formally separate the layers?
I am somewhat unsure of the amount of work it would take. I see Gtk references in gen/constfunc.py. The gen/utils/image.py and gen/utils/thumbnails.py would clearly need to be relocated. There is the matter of gen/plug/_gramplet.py as well.
I’d think maybe plugins would also get separated out based on whether they were core or ui related, and I know some of the importers and exporters have some Gtk dependencies related to options handling and would need some refactoring to properly separate concerns.
I am asking about this because I am wondering if it might be beneficial in the long run.
If things were in separate packages then the release cadence need not be in lock step between them. Maybe the core package could follow something closer to a continuous delivery model, or at least a more frequent release model, like @DavidMStraub does with the WebAPI. So if fixes or new features were added to the core code they potentially could be made available for use much sooner.
I also think something like that might help with new features that are larger in scope. I think you had stated that @prculley Place Enhancements PR is very large and would have been easier to handle if it had been done in smaller peices. But if the Gramps code was split and the one large PR was two smaller more focused ones maybe that would have helped make them easier to review and merge?
I love this idea and would be willing to help implement it in case.
I also think it would make it easier for new developers because of the possibility to have a higher release cadence for the core library and to have features added in core before adding them in the UI.
In any case, I think it would be great to automate the release & packaging process as much as possible to reduce the workload for release packagers, which I feel is also a factor contributing to long release cycles (Gramps 5.1.5 was released one year ago today).
By the way - I think now would be a also good moment to decide because the current build system will cease to work in Python 3.12, to be released later this year, so some work is needed anyway.
I find the suggestion appealing. I tried to switch Gramps to Qt vs. GTK+ and I was really hampered by the fact that Gramps starts without knowing if it is run CLI or GUI. It decides after scanning the command line and launches the appropriate part. But this makes initialisation tricky because various libraries expect to be initialised at the very beginning of app.
For the time being, I cancelled my Qt conversion to focus on SQL.
I think that two separate start scripts would be better. This would allow for a more homogeneous and straightforward startup.
But, many CLI functions are invoked by the GUI. Perhaps this is a sign that these functions which have been store in cli/ directory belong in fact to the core.
Also, from reading the code, I find that it is based on two implicit assumptions:
data is stored in key-value pairs as imposed by the BSDDB initial choice, the value being a binary unspecified unstructured block
widgets are GTK+
This makes architecture modifications very difficult because you quickly bump into one or both tacit assumptions.
The top level Gramps package is divided into four sub-modules.
gen
cli
gui
plugins
The gen module was designed to contain the data model and genealogical functionality. It was intended that this module could be imported and used outside of Gramps. For this reason, our programming guidelines specify that code in this module should not import code from other modules.
The cli modules contains code to open a database and run Gramps from the command line. It should not import Gtk.
The gui module can obviously import Gtk and anything else. We do try to limit the dependencies though.
The plugins module contains various plugin types. Dependencies will depend on the type. A database plugin should obviously not contain gui code, whilst a gramplet plugin may well do. Separating these out seems like a good idea.
Some functionality like the image utilities and thumbnailers are required for reports run from the command line. However a command line installation shouldn’t depend on Gtk. I actually created the new thumbnailer plugin type to make this easier.
We need to assess the impact this would have for our packagers. We can also discuss release schedules.
I wrote a proof of concept Qt front-end for Gramps some years ago. I got the main window and list views working, but nothing else. Writing a complete new front-end would take a lot of effort.
Do you have a project in mind that has achieved this kind of independence between the backend and multiple frontends?
I’ve had some slight involvement in the MythTV project for nearly 15 years. Myth is a personal video recording system that was designed from early on to have a backend (that handles scheduling, recording and serving videos) and one or more frontends (that handle all user interaction with the videos). But building a comprehensive API that third party frontends can use to provide the full functionality of the native frontend has been near impossible.
Essentially, every subtle element of business logic that the application needs must be expressed in the API functions. That makes the frontends quite constricted in what they may do. If they want to do something different, usually the API needs to be expanded. Which can lead to a confusing sprawl of API functions that make it that much harder to learn. Plus the problem that the backend must either then support multiple API versions or all frontends must update to the latest API every time the backend is updated.
The bottom line with Myth is that one developer was particularly motivated to develop an Android front end. He pretty much rewrote the backend API while creating the Android frontend. (There is now Version 2 API that is substantially different from the original.) AIUI, the Android frontend is not ‘feature-complete’ and likely never will be. Some other developers have dabbled with other frontends but I don’t think any of them are on par with the Android version.
For Gramps, I don’t see how the complexity of an API between backend and frontend(s) is a clean win. It would take an enormous investment of time to design and create the API layer and then time to maintain and enhance it in the future. I don’t see benefits approaching anywhere near that scale. But if there is a counter-example of a project that has benefited enormously from a similar split, let’s hear about it.
That is good design though, the use cases aka business logic should be abstracted out from the UI layer anyway.
But key point you are making taken, in the existing codebase any significant changes to the core libraries or object model would still require the UI and WebAPI be updated and kept in lock step. You could release bug fixes and add new functionality that does not change the existing call interfaces and object model, but perhaps it is not worth the effort at this point.