I would love to see Gramps sources and citations be enhanced to something similar to what is used in Family Historian 7 or Rootsmagic. What is used now looks like what was used with PAF before it was discontinued.
Can you add screenshot to see what youâll like to get ?
Agree that the system of Citation support needs another, much deeper look. there isnât enough structure to auto-generate high-quality inline references, footnotes, endnotes & bibliographies in standard formats.
Moreover, the collection & organizing of supporting digital documents as Media Objects is heavily weighted toward picture collections. And it is completely without document management capabilities.
Letting it evolve naturally has huge potentially to paint Gramps into a corner. So the implementation needs some serious design work.
How about a bridging implementation? Interface the APIs of Gramps and Zotero Research Management assistant. It could be very superficial⊠Let Gramps query individually for a pre-formatted citation or for adding a source to/from their collection.
Zotero already has years of development on these issues⊠that our project canât afford to duplicate. Particularly since such a commitment of resources would only result in a fraction of their present features⊠while they more further ahead.
It is another free and open source project for Linux, Windows (including PortableApps) & macOS. (Dunno about BSD) But it is JavaScript rather than Python.
4 Likes
Does anyone know how far along GEPS 18 was before it was abandoned, as I thought that was intended to address many of the issues around citations?
I agree Interfacing with external solutions that are more specialized certainly makes sense for more advanced researchers.
At the same time Gramps should probably provide better support for those things than it does. Novice and even intermediate level researchers may not want to learn and keep data in several tools.
At least those are my thoughts.
Went into storage and found a book I purchased back in 2000 call Family History Documentation Guidelines for PAF written by the Silicon Valley PAF Users Group. It has examples of different sources and citations that looks like the layout that Gramps is using. I searched all the documentation and Wiki for Gramps examples but mostly found blank pages. See if this guide helps out.
2 Likes
Family history documentation guidelines
| Author: | Silicon Valley PAF Users Group. |
|---|---|
| Publisher: | San Jose, California : Silicon Valley PAF Users Group, ©2000-2003. |
| Edition/Format: |
 Print book : English : 2nd ed View all editions and formats
|
| Subjects | * Personal Ancestral File (Computer program) |
2 Likes
I learned a lot from reading this book 20 years ago. Itâs still in my library next to the EE pad!
The problem with GEPS 18 was that it was based on EE, and even with a light version of that, you get hundreds of citation elements and templates, and thatâs almost impossible to translate to the number of languages that we need to support in Gramps, which is about 40.
Also EE by itself was never meant to be implemented in software in the way it was done in RootsMagic, which has more than a thousand templates. Thatâs not a problem for them, because they donât make translations anyway, but for us, itâs a problem.
As far as Iâm concerned, Zotero makes the same mistake as EE, meaning that for each type of source, you have a different set of variables, which effectively makes it a moving target. Thatâs why you can better rely on CSL, where types and variables are independent:
https://docs.citationstyles.org/en/stable/specification.html#appendix-iv-variables
One aspect thatâs missed in all models mentioned here is, that they donât understand the hierarchical nature of many sources that we work with. That leads to weird constructs where you need seperate variables for similar elements like the name/title of an article, which appeared in a magazine, that you found on a DVD. Thatâs sort of a three layer citation, where each layer may have an author, title, publication date, etc.
In this case, one can argue whether all those layers are easy enough for aunt Martha, so maybe CSL is still the best choice. Any EE like solution is likely to fail because of the reasons mentioned above. See also:
5 Likes
Very valid point on the proliferation of templates and related issues.
Iâve not looked at CSL, will try to at some point. I know someone opened a feature request asking for free form citation fields not that long ago. That solution at first glance appears to provide the most flexibility.
It would allow reports to differentiate between the one to use for a bibliography vs footnote and so forth. It would allow anyone to enter citations manually using the format / style they prefer. But most importantly it would not prevent anyone from developing or using Gramplets that might provide a template driven interface for entering them, or for interfacing with an external system like Zotero to populate them either.
On the flip side that probably doesnât play well with Gedcom exports for those that might still need or want them. Iâm sure there are other considerations that are not coming to mind at the moment as well.
1 Like
Zotero are based on CSL
1 Like
That may be so, but to me thatâs meaningless, because they are not really compatible. If you read
https://www.zotero.org/support/kb/item_types_and_fields
you can see that Zotero has item types that do not exits in CSL and vice versa, so I donât think we can use it. Thatâs because itâs not a real standard, which is supported by a proper community. Also, like EE it is a moving target, meaning that it will keep us busy. It is also very American, like EE, where Gramps is quite international.
Iâm Norwegian and have used Zotero for 3 years now in combination with Gramps and a lot of other research tools, without any problem. And actually the most limited software of all of the Open Source Free software I use, is Gramps, specially when it comes to interchangeable data formats outside lineage-linked genealogy, i.e. the formats is useless for any kind of analyzes or research work outside of Gramps without a lot of reformatting work either using Excel or a lot of python code.
Yes, there is still some limitations in Zotero, but as Zotero, CSL is as you call it a âmoving targetâ, whatever you mean with that, and at least, Zotero is supported either by native code or by a plugin, by a lot of other software, including most LaTeX clients, R Studio, Atom, VSC, Obsidian, Zettlr etc. etc.
Next version of Zotero have improvements, most likely more than Gramps will have in itâs next edition, if Gramps 5.2 ever get released, even though its, open source as wellâŠ
Zotero have a lot better source, Bibliography and Citation/Reference possibility than Gramps.
And if you donât like Zotero, use Jabref, or any other BiblTeX client, or buy Citava.
And it is very strange that most international Research Labs and Universities recommend Zotero as an alternative to Mendeley if it is so useless as you tare trying to make it.
There will always be special cases where any software come to short.
I canât use Gramps to most of my projects because of limitations in interchangeable export formats, so for me Gramps has become a locked in software, just as any of the other genealogy software.
Only thing Gramps is âokayâ for is registration and storage of data I donât need to analyze or do research on. So itâs great for archiving purposes, to archive finished data, but itâs more or less useless for research and analyzes of any data that is not directly lineage-linked, and for lineage-linked research, it is a lot of other software that is easier to use, and have more features than Gramps, EVEN other open source software projects.
I have also stopped using Gramps because of limitations in both Event functionality, since all real life events is a serie of main and sub event, and there is no way to register that in Gramps, and because of the lack of hierarchical registration of sources, like its in Zotero with Collections
So as I say, all software has limitations, but when users ask for functionality, developers, even in open source software project should listen to them, if not, the project will die.
I donât think Gramps need a field based source system, but it needs a lot more functionality to support external 3rd party software via interchangeable formats. Without it, Gramps is just âYet another mid range genealogy softwareâ that is extremely hard for users to get started with.
And for references, a gramplet that read citations or bibliography from a csl-json or bibtext file and add it to the pub. info and Volume/Page fields in Gramps, would be of great help for a lot of users, and unique in regards of genealogy software.
AND if it support csl-json/biblatex files as i.e. zettlr and OBsidian does, it will also support any other reference manager of choice that can export to those formats, including Pandoc citeproc.
CSL JSON, GraphML, full CSV export of every table, would be a beginning,
Gramps is a software for lineage-linked genealogy, obviously it canât compete with software which is spezialized in other areas like all the software you mentioned to use for your research.
Exporting gramps data into other formats and do that in a meaningful way for other 3rd party software is not a trivial task. Just exporting gramps data from a table into another format does not solve anything, since the 3rd party software needs also to know how to interpret the data to be useful.
1 Like
Jaran, I understand what youâre saying, but you are missing the point that standards like CSL and GEDCOM change quite slowly, and are well documented, while projects like Evidence Explained and all sorts of software for note taking (like Evernote) and reference managent (like Mendeley, and Zotero) do not. Evidence Explained will be expanded whenever Elizabeth Shown Mills feels that itâs needed, which may simply mean that a new version is needed, because her income depends on it. And in fact, she once wrote in another email group, that it was never meant to be implemented in software literally, as it was done in RootsMagic.
And even though Zotero is open source, that tool will also be expanded/improved whenever its authors or users feel a need to do so, which means that, like with EE, following Zotero is just plain stupid. And I use the term stupid, because it would mean that weâd give up our independence, and it would mean that our developers, who are all volunteers, would be obliged to make changes, and translations in about 40 languages, whenever such software is changed. And thatâs not very likely to work.
Like EE, Zotero also suffers from a multitude of object types, where each type has its own template, and that is also very stupid, because any smart person can imagine that when they think about a type, they will always look at the object type as it exist in their country, where the same object type may have different attributes in my country, and even has different sets of attributes when accessed in different archives, or on different sites.
This is why I think that we must concentrate on the attributes themselves, or variables, or whatever they are called. and where we can make an informed decision to adopth the variables of CSL version 1.01, just like we must make an informed decision about GEDCOM 7.0.
It would be nice to have converters for other tools, like EverNote, or OneNote, and a couple of reference managers, but I will never surrender to any cult, and any tool with a large following tends to turn into a cult, including Gramps itself.
What we really need to find is a common ground that works for most of us. And IMO, the citation model of GEDCOM is inadequate, and the same goes for any model that moves too fast, or is too hard for users to understand, and thatâs a problem for GEDCOM too, with its repositories, sources, and citations, all in different objects, where citations are completely different from what most normal people think they are.
And as long as GEDCOM and GedcomX donât really solve this, CSL variables may be part of a solution.
No, you are missing the point, Zotero is based on CSL and even though they have some limitations and some extra fields, the work on it, and the extra fields are user requested.
In addition, you can add ANY CSL or custom key you like in Zotero by using the Extra field.
And you also clearly donât understand that gedcom and CSL are two widely different type of standards.
In addition, ALL Gedcom standards are lossy, they are extremely limited and should have been deprecated 2 decades ago.
You also need to read before you comment, because no one has ever said âfollow Zoteroâ, what we talk about is supporting interchangeability with software that is heavily used in research enviroment.
So before you write a new comment, try to read about what we actually discuss here, itâs support for INTERCHANGEABLE FORMATS, in this case, CLS JSON and BibliaTeX.
By having a Gramplet that can read and/or write those formats, you can have source and reference interchangeability with a lot of other open source software, and you are open to use what ever reference manager you like, if you like you can use a plain text editor and create your references manually.
And it has neither never been said or written that whatâs in Gramps today should be removed, itâs a discussion about expanding and make Gramps more feature rich and user friendly for people that is NOT Linux and Python developers!
And you talk about stupidity? do you really think that 70-80% of all researchers, professors and siencetists would have used software like Zotero if it was as useless as you try to make it?
And no, templates are not a bad idea, as long as the system is robust and customisable.
There will never be a âGedcom Xâ or âGedcom 7â, and if it should come, it will still be so limited, that it canât be used for anything but LDS lineage-linked collecting of data.
You talk about Zotero and EE being limited, and then you use gedcom as an better example?
And tell med, whats the different by focusing on âattibutesâ than supporting read/write from a cls json file?, the only thing a plugin like that will do is add the Key Name and the Key Value to Gramps attributes.
And last, the Attribute editiong functionality in Gramps is anything but user friendly, so if the project team actually want the project to survive, i.e. get new users and python developers, they need to focus on two things, user friendly interface, and research environment that actually utilize this type of software and functionality, and that are willing to use time on the project.
And to get interest by that type of environment, Gramps need to have interchangeable formats that actually can be used in other research and analyzing tools and software.
It doesnât help at all to hide reference information in Gramps Attributes, when it canât be used to anything but Gramps, if that is done, Gramps is just yet another LOCKED IN SYSTEM.
just an digression, but it is actually easier to get data out of a Rootmagic or Legacy database in an interchangeable format, that it is to get same type of data out of Gramps, AND THAT INCLUDE sources and references.
Why reinvent the wheel?
And absolute last, CSL variables also need a name, and a âtemplateâ , even if that "template is user created. else you would need to define the same key name in Gramps Attributes Editor every single time you add a source⊠that is anything but user friendly, and the only thing that will do, is to make sure that users donât register sources and citations.
I for one knows that If I had to manually add key names and key values for any 20-40 variables for everyone of the approx. 5000-10000 sources I have and 3-4 times that in citations and references, I would have dropped adding bot sources and citations, or actually I would have start using another software.
The day I discovered that I actually could copy a Bibliography string from Zotero to any text field in Gramps, made registering sources a way easier, I think I shortened my source and citation registration with 10 minutes for each new source, and 2-3 min for each additional Citation, and when you shall add 2-300 newpapers with 2-3 citations for each, that is a lot of time.
Another thing is that the most used Reference Manager today have really good web clippers, so you get a nearly complete Reference/Citation/Bibliography string from them âright out of the boxâ.
And again its about using already well established interchangeable formats like csl json and biblatex, not to create an interface to zotero, zotero is just another tool that can import/export that format.
Next time, please read ones more what I actually write, please? I talk about support for read/write of interchangeable formats like csl json and biblatex, and there is already python libraries out there that do most of the job.
Those formats is also used by Pandoc Citeproc.
I have not once suggested that Gramps should in any way directly interface with a Zotero database or any other reference managers database, I have ALWAYS talked about interchangeable standard formats, and to be sure, I mention two of them again, like CSL JSON or BiblaTeX, where the later is somewhat limited.
As long as the open standards are supported, there will be interchangeability without any problems.
i.e. CSL JSON is a defined standard format, as is graphML or any other network graph formats that is open source, at the same time they are nowhere near to be as limited as gedcom.
There are no problem at all to make an complete export of a Gramps Database to a graphML that is readable in i.e. Cytoscape or Gephi,
I transform Gramps XML in Excel without changing any key name. I import all the objects with attributes to tables using Power Query, then I export those tables with a vba script. not a single value is lost.
And if I can do that in Excel, itâs surely doable directly from Gramps.
and yes, Gramps is a lineage-linked software, but why is it so dangerous for some people if it evolve to something more and more serious researches want?
why stay with the status of âYet another genealogy programâ with a -5 in user-friendliness, when it is just a little effort that is needed to take it up to a full research platform?
Itâs all about the willingness to actually support it
Itâs actually only 3-4 small changes thatâs needed to make Gramps a tool for both Event Based and Document Based research in addition to lineage-linked research.
those are
- Places as subjects of Events
- Main and Sub-Events (hierarchical registration of Events)
- Better Document function(Source and Citation handling), including hierarchical registration of sources. Like Collections in Zotero, just to use an example.
- And to make it a full research platform, all that is needed is to add support for 3 full database export formats that all is used on most other platforms, i.e. one already defined Web data interchangeable format like OWL or JSON-LD, one network graph format like graphML, and a full export to CSV.
And for you âpointâ about the other software need to be able to interpret the data, as long as the export confirm to the standard, itâs not a problem, itâs only a problem when software developers start creating their own hybrids of the format, something that is not needed to be able to export any and all data keys in Gramps.
You are totally missing the point.
If Gramps support a few interchangeable formats, it is no need to try to compete with other software, all that is needed to use any of them is to export the database in the format needed, i.e. you can export exactly the same data from Gramps to a graphML file, that you can in Gramps XML, only difference is the structure of the file.
Itâs kind of simple actually, any objects in Gramps is a Node, any connection or relation in Gramps is an egde, it doesnât matter how many or what the name for them is, any connection has two point, the source and the target, any edge or node can have as many attributes you want them to have, each and everyone with a custom name if needed.
Any other software that support that format, will read the file without any problem and draw a graph with nodes and the nodes attributes in a node tables, and then draw edges with the edges attributes in a edge table.
I really donât understand why you and a lot other think itâs so bad to extend the usage of Gramps to be a real research platform??? I really do not understand that mentality at all.
And I can not understand why anyone think lineage-linked research is anything but limited, doesnât people here do anything but register as many individs as possible, without any historical research around those individs???
Stepping back a bit hereâŠ
I think that since Kari Kujansuu (thank you @kku !!!) released the SuperTool add-on in March 2021, Gramps can now claim to have a viable complete export via CSV.
It will take some serious exploring of the documentation & interface⊠there may need to be some enhancement requested⊠but the export functionality is now there.
(One item of curiosity needing particular experimentation are the texts in the Note objects. Excel has a 32k limit on Cell contents. Are there similar limits in Gramps or CSV that would affect interchange? Perhaps there are data-length limit differences that would require segmentation & joining options?)
Complete CSV Import is a horse of a different color.
As long as it is not in the Gramps Gramplets Repository, I donât think of it as a Gramps feature.
When I write I only write of things that is easily accessible in Gramps, without manually and stuff. And I canât see that the excellent tools of Kari is included in the Plugin Manager.
And for most users it will be a problem to change to another plugin repo, with questions like, do I need to change back to be able to update the rest of the Gramplets, How will I get updated Gramplets now? etc. etc.
Maybe Gramps should support a comma separated list of ReposâŠ
But it really is proof that it is possible if the will to do so is present!
Thatâs an objection I would expect from the wider community. (And it is one with justifiable concern!) But youâve always been incredibly proactive seeking tools that work in combination⊠with little fear of about obscurity or market share.
It is inconvenient that we currently have to scan the standard Gramps Add-ons AND then toggle the Isotammi Configuration dashboard gramplet to scan for their add-ons. You are right⊠things would be SO much simpler if the âWhere to checkâ iterated through a CSV string of locations. (But although that should be a relatively minor enhancement to the Core code, i would not expect it. The Plug-In Manager Enhanced disagreement makes it unlikely anything less than a major rewrite will be approved. Of the 2 people interested, one hasnât the time and the other is frustrated. So that section of code is probably stuck in limbo.)
And it is disturbing that there is sufficient objection to Taapeliâs alternative approaches in the Isotammi project. There is enough push-back that their choice to maintain separation in repositories seems well-advised. And additionally worrisome that, despite the huge strides Isotammi is making in systematically merging crowd-based genealogy research via Gramps, no developer has actively assisted in getting the Isotammi Configuration add-on made available through the Plug-in Manager interface. (The only feature for users outside the Isotammi project is switching between repositories. But their âAPI keyâ setting, necessary for privacy restrictions to the Finnish genealogy, seems to be highly controversial.)
The add-on concept was one of the things that impressed me initially about Gramps. Not as an innovation, since plug-ins have been around for a long time. But the idea of giving alternative approaches a prominent forum⊠where a fair pubic hearing was offered to challengers of institutionalized guidelines. (The Plugin Manager versus Plugin Manager Enhanced being a prime example of conservative dogma and dissident innovator co-existing⊠maybe not without resentment⊠but at least peaceably.) This constant challenge is needed to avoid stagnation.
Personally, I can think of at least a dozen reasons why someone might want to maintain a separate Gramps add-on repository. (Not the least of which occurs when someone creates a Gramps-based genealogy community ⊠where the members need to herded towards standardizing on the same subset of tested tools. Just as Isotammi has done.)
1 Like