Gramps and recording and comparing DNA-matches

Hi,

(the first two of paragraphs is background, feel free to skip them)

I just downloaded Gramps and am liking what I see so far. I was (and still am) a “The Master Genealogist” (TMG) user, who hasn’t yet moved on since development stopped on that, and haven’t updated my database much for a few years.

I have however gotten into working with autosomal DNA tests, and it’s time to apply that to updating my genealogy database, so I need a new tool. I also want to make my approach much more source-centric and I was considering writing something myself, but it’s been years since I did much serious programming and I never made a large application from scratch, so I went looking for what already exists in open source and found Gramps.

I’ve searched a bit though, and I haven’t found much about how Gramps and Gramps users include DNA match results. So I’m wondering if that is because I’m bad at searching, or if it just isn’t out there. How do Gramps users include the results from DNA-testing in their database?

Ideally I’d like something like the functionality of Genome Mate Pro, for those that are familiar with that application, built into a genealogy program, and keep track of my tree, and imported DNA-matches and their ancestor trees, in one place.

3 Likes

Gramps doesn’t have any DNA specific features at this time. So if someone has a good understanding of what sort of features are needed, we would love to hear from you.

If I had to store something about DNA at this time, I would probably add it to a Persons Attributes. That way it would stay attached to the person. This would be good if the data was reasonably concise, not so good if it was a full report.

The next step would be to write a tool or report that could do reasonable things with the DNA attribute, like look for matches in other people.

Both of these things can be done without changing Gramps itself, as we have an addon ability that allows easy addition of many types of features. This would be the way to go for experimental features like this, in my opinion. It can be developed and tested on your own machine, even while Gramps is being updated by the developers. And it can be shared easily and quickly, one it gets to the point where it pretty much works.

3 Likes

I wrote an experimental gramplet which used DNA markers stored in attributes. There is some information in feature request #8919 (Add ability to record Genetic information eg: Haplogroup).

2 Likes

@Nick-Hall @prculley I’m currently working on two new filter rules “persons sharing yDNA” and “persons sharing mtDNA” to filter relatives sharing the same DNA/haplogroups.

I’m also interested to help adding new fetaures for genetic genealogy to gramps. Some ideas:

  1. I don’t think that Person Attributes alone are enough to store DNA data e.g. you can add the haplogroups and a few DNA mutations, but more than that ant it gets messy. A new genetic/medical tab in the person editor might be a better choice. That would also allow more than a key/value pair. The raw data files of the companies have usually 4 columns:
rsid        chromosome   position     genotype
rs1234       1            12345        GG
  1. The files could be added similar to other media. If a user provides the rsid of what they are interested in, gramps could scan through the external raw file and add the rest of the data without the user having to add it. Note: The position on the cromosome depends on reference genome and can be different between different files. The genotype can also be the opposite strand e.g. CC instead of GG or AA instead of TT.
  2. For reports the comparison between two files should also be possible to add as well as extracting how many cM of DNA is shared. The tool Lineage might be interesting to have look into.
  3. From the size in shared cM possible relationships between two persons can be calculated. (see here)
  4. Genogram reports have already been asked several times as a feature request.
2 Likes

Great answers!

I don’t think storing a whole raw DNA file is particularly useful at this point, although it might be in the future. The matching algorithms are quite large beasts and if one have access to the raw files for multiple people, it is likely better to just upload them.

Storing y-dna and mtdna haplogroups is likely useful. I don’t see it as too complicated, considering it’s genuinely a property of each individual. A challenge might be that the designations keep changing, at least that’s my impression from observing discussions about such tests, I haven’t sprung for one myself.

What I personally am looking for is the ability to store data on autosomal matches. For each match, which is a pair of individuals, match data consists of one or more segments with information about chromosome, start and end point and length in centiMorgans.

As for using this data a starting point would being able to search and compare such matches visually per chromosome. When I have a new match with unknown shared ancestry it’s convenient to be able to see if I already have hypotheses about the line for the segments in questions, across companies.

The next step up from that would be automating such comparison.

Now there are lots of tools offered by the testing companies themselves, but they of course do not work directly for matches from different companies. There are also several third party tools, but they require duplicating a lot of effort. I have primarily used GenomeMatePro for instance to store and compare match data and not to fully document the relationships discovered, because the built in tools for storing the ancestry of the matches is not user friendly and requires a lot of duplicated effort both within the software and with my genealogy program.

3 Likes

Yes, the haplogroups evolve over time, but what don’t change are an individual’s haplotypes, and perhaps those are what should be stored as personal attributes, or as attributes related to “DNA test” events. Of course it would also be useful to store the haplogroups too, realizing they could change.

I share your feelings about Genome Mate Pro and have tried to think of ways to store some information about autosomal segments. Mostly, I’m interested in replicating its Segment Match feature, to document which segments of my DNA I inherited from which ancestors, and secondarily which cousins share them.

For storing information about matches, I have thought of using a “DNA match” event to be shared by both parties, with an event attribute for each matching segment. The format of the attribute’s value would be something that could be parsed for purposes of comparison to segments from other DNA match events. For example, suppose I match a person on chromosome 2 from position 120000000 to 140000000. I could store this value as “2-120000000-140000000” or perhaps with some other delimiter. Now, if I want to know whether that overlaps with a segment from another match, such as “2-130000000-150000000”, then I somehow need to parse the values first. It would be nice to have filters that could do this.

So, to generalize the problem, I would be interested to learn whether there are precedents among Gramps users for storing other multi-part values within a single attribute, and using them in comparisons.

3 Likes

Hmm as short term solutions there are several ways which would work for DNA matches:

  1. shared custom “DNA match” events with match data as attribute for each segment
  2. shared notes with custom type “DNA match” and match data
  3. Assosiations with match data in the value field for each segment (would require to be added to both persons)

Personally I think #2 might be the best way, but they all would require users to add all match data the same way e.g. “Chr-Start-End-cM”. Each line of the note could also be a seperate matching segment for the two people.

2 Likes

An individual’s actual DNA doesn’t change over time, but their haplotype sort of do, for three reasons:

Most people take a test covering a limited set of markers, giving them a high level, not-very-specific haplotype. Later they may chose to take a more specific test.

The nomenclature for haplotype is not fixed and has changed in the time Y-DNA tests have been available.

The grouping of haplotypes is based on the sample of Y-DNA available today. As more people test, some groupings are found to be mistaken and renamed.

3 Likes

Yes, I agree.

I was thinking that for Y DNA, for example, one could have a separate attribute for each marker, storing as a value the number for short tandem repeats for that marker:

Key=“DYS393”

Value=“13”

Key=“DYS390”

Value=“23”

etc.

As more detailed testing is done, more attributes could be added. Having the attributes stored as part of a DNA test event could also be useful if somehow a person got slightly different results from different tests due to some error in the lab.

In that way, what is known about the person is stored as attributes of the person. I’m less sure about where to store what is assumed about the person, namely their assumed haplogroup.

Regardless of what is stored where, personally I need to become more familiar with how to use attributes in Gramps, especially for comparisons or calculations, and at what point some programming would be required in order to do anything useful with them.

2 Likes

This is somehow how Legacy familytree store the DNA results…

They have a “DNA” button that open a “DNA-Form”, something similar to the form gramplets, where you can add all your DNA results as data-pairs, but you need to type it in manually (from the CSV or other types of tabular formats you get from the test provider…
Legacy has a predefined list of tests that you can select (something like Types in Gramps?), the you get a list with the definitions, and types in the values…
If you have a Y12 and a Y65 and a Y100-something, you need to type inn all the values, and you register it on the test date, in addition it has a dedicated field for the haplogroup (but this is fixed, and if you change it you change it for all the test, its not a list of triplets with a field name, value and date (as it should be)…

I’m not a developer, so I dont know how much job it wil be to add a form for DNA, but I know that since Gramps use Python, it will have a advantage from many other software, because of the number of Pyton libraries for both analyzing and calculation and visualize DNA that’s already made for Python.

Nearly every DNA or Humanities research lab in the Worrld use Python or R for analyzes… and in addition you have great visualizations libraries like the NetworkX (but it is under BSD, dont know if that is compatible with Gramps license).
I’m sure it would be easy to find multiple libraries both for analyzing, calculating and visualising that are compatible with the Gramps licensing regime, and that can be used by the Gramps UI libraries (But that would be up to the developers to decide)

I think It would be best if it could be a seperate module, like the name editor, where you could add multiple test result with the name and type and date of the tests taken, this module could of course store its data as either or both personal attribute pairs/triples (if next version of Gramps will provide attributes with dates) and as an test event for the person, since an event would give some additional fields for information, and a DNA test is actually a Event in a persons life (something that person has done one or multiple times in her/his life)…

I would prefere that the data was added as pair/triplets/quadruplets (four-tuples), instead of in a value separated list or a list of pairs in one attribute field… I think it would be easier for the human eye to read and understand, but of course in reports and other visualization of the DNA data, it has to be merged to something understandable i.e. a list with the metadata of the test as a description…

Many of the libraries and research software i.e. cytoscape and other more specific medical research software read either csv ((Node, Edge) or Network Lists)), the Gramps CSV export actually only need a little rescaping to be imported to Cytoascape and be useful, it should be possible to add fields so that it could be used for DNA purpose, usually a graph visualiization tool like Gephi, Palladio, Cytoscape or any other, use a two list approach, one list (csv file) for nodes and one list (another csv) for edges (relations for those not in the graph world), and in addition to the basic, the list can hold additional values like dates, periods, tags, weights, directions and other attributes that can be used in calculations of different graph networks or for analyzing the data it self…

This is some of the Python libraries I have found for DNA, I have not tested any of them, just collected them for later “use”, if they can be used in Gramps, I do not know…

Andrews, J. (2019). A minimal desktop app for easy and convenient gene annotation.: J-andrews7/Genotify [JavaScript]. Hentet fra https://github.com/j-andrews7/Genotify (Original work published 2017)

Frampton, M. (2018). seqfam: A package primarily designed for analysing next generation sequencing DNA data from families with pedigree information in order to identify rare variants that are potentially causal of a d… [Python]. Hentet fra https://github.com/mframpton/seqfam (Original work published 2017)

Johns, L. R. (2019). Lorarjohns/DNA_pandas_selenium [Jupyter Notebook]. Hentet fra https://github.com/lorarjohns/DNA_pandas_selenium (Original work published 2019)

Riha, A. (u.å.). lineage: Tools for genetic genealogy and the analysis of consumer DNA test results (Versjon 3.0.1) [Python, OS Independent]. Hentet fra https://github.com/apriha/lineage

Treece, J. (2019). Jeftreece/dnamatch-tools [Python]. Hentet fra https://github.com/jeftreece/dnamatch-tools (Original work published 2019)

2 Likes

Regarding haplogroup/haplotype:
The haplogroup is defined by SNP-mutations of the yDNA which occured over time. So if a new (more recent) SNP mutation is found which creates a new subclade further down and is more specific, your old haplogroup should already contain it (but was less specific). I’m unsure how often it really happens that your first prediction of your haplogroup was so wrong that you changed to another branch/haplogroup of the tree and your first haplogroup did not contain the new subclade. Is that still a common problem?

I think the only way to auto update/change/rename to newer haplogroup/haplotype would be having a raw DNA file to analyze in gramps, so probably your own haplogroup (where a raw data file exists) could be updated, but all other persons where the haplogroup was entered manually would have to be changed manually again. Or does anyone know another solution?

1 Like

I can’t that much about DNA, I was just thinking that different test could give different result, also regarding haplogroups, and in historic research ALL data should be saved and stored, and it should be possible to use the old data for comparing changes over time…

I think that of all data, not only DNA, Thats why I really hope for dated attributes or even better quadruplets, because then it will be possible to use the attributes to create custom linked data, that is not normaly done by roles, tags and events…

I know that a lot of what I wish for is something noone else use or even see any benefits from, but its still something I think can be of help for those using research tools outside Gramps and really wnat to use Gramps as a primary database for as much of their data as possible…

Gramps are already a really good choice for storing most humanities research data, but there are some things to be done before its easy to use the data outside of Gramps, in software not created for gedcom, most research tools still use csv or one of the network specific json standards out there…

This is of course way out of the Gramps usage area, but it would be great if it would be possible to make the data exchange even more “Open Data” like… as it is now, gramps is one of few Feature Rich Genealogy Software (commercial or free) that actually do work with csv, json and xml…
I think the gedcom 5.x.x standard should be banned, because I don’t want to be limited by a lineage-linke research approach…

But thats just me… I have understud that most users of genealogy software don’t actually understand or care about the difference between lineage-linked, Event-based and Document/Fact/Evidence-Based research…

Users are already using the existing Forms addon to do this. The markers are recorded as attributes in a “DNA Test” event. My experimental gramplet can then be used to compare markers across related people in a family tree. What we didn’t discuss at the time, was how to import data from files provided by the test companies.

1 Like

Yes, that seems to be a good starting point.

Do you have an example data file? What do you want the comparison chart to look like?