Web based Gramps

Excellent!

I suggest that I create a new repository and that we use your existing code as a starting point.

In addition to the views that you already retrieve in dictionaries, I’d like to have the ability to get, patch and delete individual objects in json format.

Then we can think about adding more complex functionality such as using filters and running reports.

2 Likes

Hi Jaran

From what I understand you are talking about a data format (RDF) that has been standardized across some research tools. I think Gramps data could be formatted in a number of standard formats, however I think this thread is not trying to solve that specific problem.

A web based Gramps would decouple the client (visual) from the backend database via web based REST APIs.

This would allow multiple users/ clients to access the database.

Your RDF formatted data could be implemented as an API or by the client which just converts Gramp formatted data into any other format.

@StoltHD thanks for your input and ideas, but I suggest to move them to a new thread as they are a bit off-topic.

Back to the original discussion, @Nick-Hall :

I suggest that I create a new repository and that we use your existing code as a starting point.

Yes, perfect!

In addition to the views that you already retrieve in dictionaries, I’d like to have the ability to get, patch and delete individual objects in json format.

Yes, fully agree.

Then we can think about adding more complex functionality such as using filters and running reports.

Indeed. Basically, everything that can be done in the CLI is also straightforward to do in a web backend.

To start off, I would have a couple of architectural questions/suggestions.

  • Authentication. This is quite app-dependent so I would suggest to have a completely unauthenticated backend. A given app could then decorate all or some of the view functions afterwards with login_required; or somebody might decide to just put it behind HTTP basic auth (e.g. when used only on the local network).
  • Database locking. Defininitely this has to be restricted to DBAPI (I don’t know what the current plans are for deprecating BSDDB); but for a read-only backend, I would suggest to ignore any locking. This would allow opening the DB from multiple threads/workers, as is required for a scalable web application.
  • DB write access. I suspect that this is not a goal initially? But actually I think it’s not impossible. It would require locking the DB for writes.
  • Deployment. At the moment, I use my own Docker image with Gramps installed in Ubuntu as a basis for the web app. However, for most of the funcionality the pure-Python gramps.gen and gramps.cli modules would be enough. It would be much more portable/user-friendly to have a PyPI-installable package. Would you be opposed to publishing a “core Gramps lib”-like package to PyPI? This wouldn’t have to be used in the Desktop app.

I’m sure more questions will come up with time…

I still don’t understand why you want to reinvent the wheel

If you think this is reinventing the wheel, quite frankly I believe you are misunderstanding. But of course ideas are always welcome, so I suggest you implement a working example of what you have in mind as a demonstration. This is what I did.

That could be very handy for people (like me) who want to try using R for some data analysis and presentation. There is an R package called reticulate that allows it to interface with Python, and it could be nice to access some of the Gramps logic used in filters etc. rather than having to reinvent it.

Also, for those of us who don’t really know what RESTful means except that it sounds rather pleasant, what are the similarities and differences (in terms of user experience) in this web-based approach as compared to the rollApp version of Gramps that was mentioned in this earlier thread?

I ask so that I can understand enough to provide meaningful feedback from a user perspective.

Thanks.

Have you considered the possibility of keeping the WebApp a multi-user ‘Read Only’ database tool but adding a Feedback form that sends free-form edit suggestions to the webmaster?

Something akin to FindAGrave’s “suggest other corrections” feature in their “Suggest edits” feature?

It could start out as a simple eMail with the suggestion and reference to the webpage that spawned the Feedback form.


Down the line, the suggestion might be encased in an XML wrapper to make an importable ToDo/Feedback Note. The wrapper could link the new Note to the Person’s GrampsID with a Citation under your Website as a Source and the spawning URL as the page.

So you’d have tracking for where the information came from and could convert the Note from a ToDo/Feedback type when the information had been hand-parsed into the master database.

Hi George-

The rollApp approach (from my limited view of it) seems to add another layer of complexity, basically trying to redirect the desktop gui to a web interface. More of a tack-on blob to get existing linux apps to show on the web.

Building a RESTful API (which is basically a set of ideas on how to interface to any resource on the web), would provide a clean interface to our back-end Gramps database. Through these APIs, one could build a modern web app or mobile app calling the same back-end APIs. It also provides a way to safely deliver the data in the database without having each user directly connect to the database.

It’s more of a design change from a monolith desktop app to more of a micro-service approach, where the data may be stored securely in one place, and the client/app is being run in another place (tablet, phone, laptop, web, etc).

Another thread talking about GUI design would fit nicely into this project. I’m new to open source, but from my perspective this whole endeavor would require some project management so it all heads in the direction that 80% of us want it to head in.

I’m definitely not a UX designer, but from having played with David’s POC (proof of concept), I did realize that GUI design (and JavaScript implementation) is very important.

To me, this would be broken into maybe 3 parts - the web GUI (design), the API design, and the back-end (core) GRAMPS database. We’d want to be able to leverage as much from the existing core Gramps as possible, but I think most of it is GUI/front-end work, that would need to be totally re-done. (and thus, we’d want to really think about what a new UX design might look like).

Jeff

If our databases became accessible from the web, most definitely the owner would be able to set it as read-only. As far as being able to update a database from multiple users…I know most genealogists cringe at this idea, but if we could develop in a way that tracks changes, and easy revert, or like GIT where people can submit changes, for review. I think there is a lot of opportunity around this. But, we’d have to really design a good back-end database control system (probably some open source code already exists that handles this well).

1 Like

I’m not quite sure I understand the proposed architecture. I saw the web API as a separate layer to the web app. As such the API should probably include authentication. An app could of course decide to use a single login hidden from the users.

We will also need a place to store user preferences such as date format.

The BSDDB backend will be removed in the next major release (v5.2). All old databases will be automatically upgraded to SQLite. So it should be safe to ignore locking for read-only access.

Yes, write access will be a future goal. I can see multi-user access being possible in the future (v5.3?) if we switch the raw object format to json.

I have no objection to publishing a core package on PyPI. In fact, part of the reason for the file organization in GEPS 008 was to allow the core libraries to be used in other projects outside of Gramps.

Probably not doable in the first release, but something to keep in mind… It seems that many users want to share some or all of their research on the web, some publicly, some only with family. Currently people achieve that, for example, by generating and distributing the narrated web report using filters and/or masking living people. It would be nice if the new web-based Gramps could support some kind of row- and column-level, role-based security to control what each group of the ultimate end users can see and do. (I am just talking about authorization; authentication would be a separate matter.)

It would be nice if the new web-based Gramps could support some kind of row- and column-level, role-based security to control what each group of the ultimate end users can see and do. (I am just talking about authorization; authentication would be a separate matter.)

Yes indeed. There are two separete issues here:

  1. How role-based permissions ares handled in Gramps itself
  2. How the web backend handles the authorization.

Concerning 1., if I’m not mistaken, there is no role-based permission mechanism in Gramps at the moment other than private records. So for the backend it would basically just be serving to two groups of users: the ones allowed to view private records and the ones not allowed.

@Nick-Hall, thanks for your comments. Concerning the PyPI package, I can start a proposal (via pull request) how to set this up in the main Gramps repository so it won’t be necessary to have another repository just for the PyPI package.

Once you have set up a repo for the REST API, wew could continue discussing technical details on Github.

I have created a new repository called “web-api”.

Of course we’d use Python to develop the web-api, but is FLASK the framework we want to use? I know that David used FLASK. I just didn’t know if there were other frameworks we should consider. I’m more familiar with JAVA based frameworks such as Spring/Boot.

I would definitely suggest to use Flask since it’s lightweight but powerful, actively developed, and has many useful extensions (in particular one for building REST APIs). It also has a much shallower learning curve than Django, which I think would be overkill for this.

2 Likes

Hi all, after re-reading this thread, I noticed that I had missed a few questions/comments partly directed at me, apologies.

@GeorgeWilmes

That could be very handy for people (like me) who want to try using R for some data analysis and presentation.

For that I don’t think an API is necessary. In fact I use Jupyter (with Python, but you mentioned you can also call the Python code from R) directly with the Gramps Python library to have fun with data analysis & visualization. This will become even easier once Gramps becomes pip-installable, which will allow doing this in a virtual environment.

@emyoulation

Have you considered the possibility of keeping the WebApp a multi-user ‘Read Only’ database tool but adding a Feedback form that sends free-form edit suggestions to the webmaster?

That’s a great idea for allowing contributions from relatives but not trusting them with making direct edits (I see a lot of motivation for that)! I think this is “only” an issue of permissions and UI: it can be handled within Gramps as a note with a certain tag (TODO or something like that), so one would only have to make sure that a certain user only has API permissions to add new notes. The rest is “just” the UI implementation.

@jeff

To me, this would be broken into maybe 3 parts - the web GUI (design), the API design, and the back-end (core) GRAMPS database. We’d want to be able to leverage as much from the existing core Gramps as possible, but I think most of it is GUI/front-end work, that would need to be totally re-done.

Fully agree. The backend should be almost completely covered by gramps.gen, but one could make minor additions that make the use for the API easier without affecting the desktop app (such as this one). Concerning the frontend, I am working on one, but the great thing about a common API would be that there can be multiple different approaches/apps. Whether an “official” one will emerge I am not sure. At the moment it seems a bit too ambitious to me.

If our databases became accessible from the web, most definitely the owner would be able to set it as read-only. As far as being able to update a database from multiple users…I know most genealogists cringe at this idea, but if we could develop in a way that tracks changes, and easy revert, or like GIT where people can submit changes, for review.

If I think about my own use case, I would definitely want to keep it read-only for most users, perhaps allowing the “feedback as note” suggested by @emyoulation above. But I do see a use case also for allowing full edits - for instance, by myself when I’m sitting in an archive and don’t have my laptop with me. Or maybe for one or two select genealogists in my wider family whom I trust.

Having versioning with possible rollbacks is certainly intersting, but to do this in full generality seems too ambitious to me as it would probably require major changes to the database model, with no benefit for the deskstop app. More realistically, one could enable automated XML exports on every edit. (One could even think about making a pretty visualization of a diff between two Gramps XMLs, maybe somebody has already done that?)

1 Like

It is overly ambitious but … how about thinking beyond comparing 2 XMLs towards some histogram features?

(The Import Merge Tool basically does a DIFF and then offers accept/decline. It’d be kinda nice if there was a way to save just the DIFF so options for dealing with it via automation … or manually in piecemeal fashion … could evolve.)

The auto backup does timed iterations of our trees. Diagramming the change patterns would tell us a lot about our productivity. The history in page edits in the wiki, the diffs the GitHub produces; both of these widen our perspective of the data. And even the calendar views of Windows Restore Points & Wayback Machine page capture help us understand how well we track ‘change’.

There is already a report that does a diff compare Database Differences Report addon. In fact the import merge tool uses some of the same underlying code. But of course using this report in some sort of automation would be difficult at best. As usual, choosing a standard for such a purpose is the real problem.

1 Like

I do that a lot, and was one of the main reasons for me asking for an interchangeable graph file format…
Something that noone saw any point doing, it was an useless suggestion.
I also use Graph software to find connections over multiple nodes bewteen people that you normally don’t see any connections between.

Graph visualization and network graphs that show more than “just the family” is a great tool for finding new clues or even find connections that can’t be seen in a Table View or in a “Family Tree”.

Another thing is that if users shall be able to use this type of features, it must also be available on Windows and MAC, and it must be without the demand of having a degree in Python development for “normal” users… Ask “Normal” users about Docker, and they don’t have a clue what you talk about.

A lot of developers seems to “forget” or deny that there is people using Gramps that do not use and never have used Linux, and that many users do not have any knowledge about Python programming/scripting.

1 Like

Oh, you resuscitated an old thread which has partially been made obsolete by the developments on the Gramps web API :slight_smile:

Concerning write access to the Gramps web API, there is an open issue here:

My dream for managing synchronisation of a web-based database with multi-user write access and my local Gramps databse would be modifying @prculley’s Import Merge Tool for that purpose. It’s really powerful and I think it already contains all the necessary logic. After all, this is not too different from how commercial web-based genealogy services with Desktop apps work. There is an issue to discuss this here:

Another thing is that if users shall be able to use this type of features, it must also be available on Windows and MAC, and it must be without the demand of having an degree in Python development for “normal” users… Ask “Normal” users about Docker, and they don’t have a clue what you talk about.
A lot of developers seems to “forget” or deny that there is people using Gramps that do not use and never have used Linux, and that many users do not have any knowledge about Python programming/scripting.

I’m not sure what you’re referring to here. I agree of course that Gramps should be usable on Windows and Mac. I disagree though that a web app backend should run on Windows and Mac. If a user does not know Linux or Docker, they should not be setting up a web app. Why? Because it’s easy to screw up big time when you don’t know what you’re doing. For instance, set up the app without SSL and it will be trivial to spoof the access tokens, gain full control and delete stuff (or publish private stuff).

That doesn’t mean that less technically proficient users should be ignored. On the contrary! I started the whole thing also to share my research with people who I’m not sure are technically proficient enough to install Gramps :wink: But IMO the way to make this (the Gramps Web API and everything based on it) more accessible is not by making it run on Windows, but by offering a hosted service.

1 Like