There is definitely potential here, but lots of work left.
I have been successful with getting David’s version onto an AWS EC2.
If other people were interested in developing a Web-based gramps, we could try to pull our resources together, and try to have more of a common back-log (if possible).
I would expect that if a web based gramps were developed, it could be hosted anywhere.
It looks like it has not gotten much attention lately, but https://github.com/GenealogyCollective/gprime is a program that is designed to import Gramps XML, and allow web based access to a db running on a server.
Doug Blank was the first developer to start work on a web application for Gramps. It was called Gramps Connect and originally used Django and then later Tornado as a python web framework. This was forked to become gPrime. It is compatible with v5.0, so should still work with current Gramps XML exports.
Brylie Christopher Oxley more recently revived interest in a web application which we called Gramps Online. He has Django experience and was investigating using a Neo4j graph database backend. This project is still in a prototype phase.
I am aware of a couple of other developers who have written web applications for Gramps. It would be good to co-ordinate efforts. As a practical suggestion, perhaps we could design a RESTful interface that could be used by all web applications?
David Straub implemented a restful API using the Flask framework. And then called the gramps API to access the database. That seemed like a good starting place.
Alot of the work would be the frontend javascript code to create a nice user experience.
But like you said, a robust restful API to the backend would be able to serve any number of frontend implementations.
Hi @Nick-Hall and @jeff! I found this old thread by accident - I wasn’t notified by it - but I love the idea! Indeed I am still working on the web app (with the presumptuous title Gramps web app: https://github.com/DavidMStraub/gramps-webapp). In fact, having a frontend-agnostic REST API backend was one of the main ideas. Right now, I mostly use endpoints where I fetch huge dictionaries with all people, all places, …etc., and there are separate endpoints for media and thumbnails.
I would be happy to collaborate on an app-independent REST API backend.
I think some of those tools you are referencing are for slicing and dicing and visualizing data, which might be useful. But I think the more important aspect is a way to centralize the database with ability to access the data from multiple places. That’s where a core API interface would help.
I suggest that I create a new repository and that we use your existing code as a starting point.
In addition to the views that you already retrieve in dictionaries, I’d like to have the ability to get, patch and delete individual objects in json format.
Then we can think about adding more complex functionality such as using filters and running reports.
From what I understand you are talking about a data format (RDF) that has been standardized across some research tools. I think Gramps data could be formatted in a number of standard formats, however I think this thread is not trying to solve that specific problem.
A web based Gramps would decouple the client (visual) from the backend database via web based REST APIs.
This would allow multiple users/ clients to access the database.
Your RDF formatted data could be implemented as an API or by the client which just converts Gramp formatted data into any other format.
I suggest that I create a new repository and that we use your existing code as a starting point.
Yes, perfect!
In addition to the views that you already retrieve in dictionaries, I’d like to have the ability to get, patch and delete individual objects in json format.
Yes, fully agree.
Then we can think about adding more complex functionality such as using filters and running reports.
Indeed. Basically, everything that can be done in the CLI is also straightforward to do in a web backend.
To start off, I would have a couple of architectural questions/suggestions.
Authentication. This is quite app-dependent so I would suggest to have a completely unauthenticated backend. A given app could then decorate all or some of the view functions afterwards with login_required; or somebody might decide to just put it behind HTTP basic auth (e.g. when used only on the local network).
Database locking. Defininitely this has to be restricted to DBAPI (I don’t know what the current plans are for deprecating BSDDB); but for a read-only backend, I would suggest to ignore any locking. This would allow opening the DB from multiple threads/workers, as is required for a scalable web application.
DB write access. I suspect that this is not a goal initially? But actually I think it’s not impossible. It would require locking the DB for writes.
Deployment. At the moment, I use my own Docker image with Gramps installed in Ubuntu as a basis for the web app. However, for most of the funcionality the pure-Python gramps.gen and gramps.cli modules would be enough. It would be much more portable/user-friendly to have a PyPI-installable package. Would you be opposed to publishing a “core Gramps lib”-like package to PyPI? This wouldn’t have to be used in the Desktop app.
If you think this is reinventing the wheel, quite frankly I believe you are misunderstanding. But of course ideas are always welcome, so I suggest you implement a working example of what you have in mind as a demonstration. This is what I did.
That could be very handy for people (like me) who want to try using R for some data analysis and presentation. There is an R package called reticulate that allows it to interface with Python, and it could be nice to access some of the Gramps logic used in filters etc. rather than having to reinvent it.
Also, for those of us who don’t really know what RESTful means except that it sounds rather pleasant, what are the similarities and differences (in terms of user experience) in this web-based approach as compared to the rollApp version of Gramps that was mentioned in this earlier thread?
I ask so that I can understand enough to provide meaningful feedback from a user perspective.
Have you considered the possibility of keeping the WebApp a multi-user ‘Read Only’ database tool but adding a Feedback form that sends free-form edit suggestions to the webmaster?
Something akin to FindAGrave’s “suggest other corrections” feature in their “Suggest edits” feature?
Down the line, the suggestion might be encased in an XML wrapper to make an importable ToDo/Feedback Note. The wrapper could link the new Note to the Person’s GrampsID with a Citation under your Website as a Source and the spawning URL as the page.
So you’d have tracking for where the information came from and could convert the Note from a ToDo/Feedback type when the information had been hand-parsed into the master database.
The rollApp approach (from my limited view of it) seems to add another layer of complexity, basically trying to redirect the desktop gui to a web interface. More of a tack-on blob to get existing linux apps to show on the web.
Building a RESTful API (which is basically a set of ideas on how to interface to any resource on the web), would provide a clean interface to our back-end Gramps database. Through these APIs, one could build a modern web app or mobile app calling the same back-end APIs. It also provides a way to safely deliver the data in the database without having each user directly connect to the database.
It’s more of a design change from a monolith desktop app to more of a micro-service approach, where the data may be stored securely in one place, and the client/app is being run in another place (tablet, phone, laptop, web, etc).
Another thread talking about GUI design would fit nicely into this project. I’m new to open source, but from my perspective this whole endeavor would require some project management so it all heads in the direction that 80% of us want it to head in.
I’m definitely not a UX designer, but from having played with David’s POC (proof of concept), I did realize that GUI design (and JavaScript implementation) is very important.
To me, this would be broken into maybe 3 parts - the web GUI (design), the API design, and the back-end (core) GRAMPS database. We’d want to be able to leverage as much from the existing core Gramps as possible, but I think most of it is GUI/front-end work, that would need to be totally re-done. (and thus, we’d want to really think about what a new UX design might look like).
If our databases became accessible from the web, most definitely the owner would be able to set it as read-only. As far as being able to update a database from multiple users…I know most genealogists cringe at this idea, but if we could develop in a way that tracks changes, and easy revert, or like GIT where people can submit changes, for review. I think there is a lot of opportunity around this. But, we’d have to really design a good back-end database control system (probably some open source code already exists that handles this well).
I’m not quite sure I understand the proposed architecture. I saw the web API as a separate layer to the web app. As such the API should probably include authentication. An app could of course decide to use a single login hidden from the users.
We will also need a place to store user preferences such as date format.
The BSDDB backend will be removed in the next major release (v5.2). All old databases will be automatically upgraded to SQLite. So it should be safe to ignore locking for read-only access.
Yes, write access will be a future goal. I can see multi-user access being possible in the future (v5.3?) if we switch the raw object format to json.
I have no objection to publishing a core package on PyPI. In fact, part of the reason for the file organization in GEPS 008 was to allow the core libraries to be used in other projects outside of Gramps.
Probably not doable in the first release, but something to keep in mind… It seems that many users want to share some or all of their research on the web, some publicly, some only with family. Currently people achieve that, for example, by generating and distributing the narrated web report using filters and/or masking living people. It would be nice if the new web-based Gramps could support some kind of row- and column-level, role-based security to control what each group of the ultimate end users can see and do. (I am just talking about authorization; authentication would be a separate matter.)
It would be nice if the new web-based Gramps could support some kind of row- and column-level, role-based security to control what each group of the ultimate end users can see and do. (I am just talking about authorization; authentication would be a separate matter.)
Yes indeed. There are two separete issues here:
How role-based permissions ares handled in Gramps itself
How the web backend handles the authorization.
Concerning 1., if I’m not mistaken, there is no role-based permission mechanism in Gramps at the moment other than private records. So for the backend it would basically just be serving to two groups of users: the ones allowed to view private records and the ones not allowed.
@Nick-Hall, thanks for your comments. Concerning the PyPI package, I can start a proposal (via pull request) how to set this up in the main Gramps repository so it won’t be necessary to have another repository just for the PyPI package.
Once you have set up a repo for the REST API, wew could continue discussing technical details on Github.
