# Datamining standard libraries for Gramps? **URL:** \ **Category:** Development\ **Tags:** prerequisite-mgmt\ **Created:** [October 19, 2020, 6:57pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839 "2020-10-19T18:57:10Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![emyoulation](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/emyoulation/32/67_2.png) [@emyoulation](https://gramps.discourse.group/u/emyoulation)\ **Post date:** [October 19, 2020, 6:57pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/1 "2020-10-19T18:57:10Z")
While looking at a [new feature for Gramps](https://gramps-project.org/wiki/index.php/Addon:DNAgramplet), it struck me that the add-on was storing CSV data in a Gramps Note and some of the bugs were standard parsing problems. And that this new Note type probably needed to embed headers for future-proofing. Particularly since the DNA segment map data could come from different sources… where the same (untransformed) data can be expected but in differing order and with differing header labels. Which is the most simple crosswalk situation. Then I thought about the [CSV Import module](https://gramps-project.org/wiki/index.php/Gramps_5.0_Wiki_Manual_-_Manage_Family_Trees:_CSV_Import_and_Export#Example_CSV_with_multiple_areas) for Gramps that already did a lot of flexibility work with external CSVs: parsing handling blanks lines, multi-section with differing content validation, header recognition, and so forth. With a little bit of surfing I discovered Python has [CSV libraries](https://docs.python.org/3/library/csv.html) that build header dictionaries. Should we be using those? Are there standard libraries Gramps already uses that developers should be leveraging rather than writing their own parsers? ## Data Mining Libraries - [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - [Scrapy](https://docs.scrapy.org/en/latest/intro/overview.html) (Structured) ## Data Processing & Modeling Libraries - [Pandas](https://pandas.pydata.org/) - [NumPy](https://numpy.org/) - [SciPy](https://www.scipy.org/about.html) - TensorFlow (_example:_ [Using Nucleus and TensorFlow for DNA Sequencing Error Correction](https://colab.research.google.com/github/google/nucleus/blob/master/nucleus/examples/dna_sequencing_error_correction.ipynb)) - [PyTorch](https://wikipedia.org/wiki/PyTorch) ## Data Visualization Libraries - [Matplotlib](https://matplotlib.org/tutorials/introductory/pyplot.html) - [Seaborn](https://seaborn.pydata.org/introduction.html) - [Plotly](https://plotly.com/) - [Pydot](https://pypi.org/project/pydot/) - [Bokeh](https://docs.bokeh.org/en/latest/index.html) ## Genealogy or Genetics libraries - [SNPs](https://github.com/apriha/snps) : tools for reading, writing, merging, and remapping SNPs 🧬 - [Lineage](https://pypi.org/project/lineage/) : tools for genetic genealogy and the analysis of consumer DNA test results. (Prerequisites: Python 3.5, [numpy](http://www.numpy.org/), [pandas](http://pandas.pydata.org/), [matplotlib](http://matplotlib.org/), [atomicwrites](https://github.com/untitaker/python-atomicwrites), [snps](https://github.com/apriha/snps)) ---
**Author:** ![StoltHD](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@StoltHD](https://gramps.discourse.group/u/StoltHD)\ **Post date:** [October 19, 2020, 7:14pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/2 "2020-10-19T19:14:28Z")
I have asked something similar before… There are also python modules for working with DNA… like [lineage](https://pypi.org/project/lineage/) And there are libraries that can be used to import data to i.e. Cytoscape, that are a full blown bio-research tools… just saying… Interoperability is dangerous! ---
**Author:** ![emyoulation](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/emyoulation/32/67_2.png) [@emyoulation](https://gramps.discourse.group/u/emyoulation)\ **Post date:** [October 19, 2020, 7:22pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/3 "2020-10-19T19:22:37Z")
In some ways, this an interoperability question. (Standards conformance and such) But it is actually more basic than that since it doesn’t (yet!) stray into piping data with flow control between APIs. This is mostly about simplifying development by re-use of existing code. Becoming proficient with passing data INSIDE Gramps is fundamental to fleshing out the API for interoperability. I know some people have doing impressive things in Gramps with the visualizations. And from the postings about prerequisites & the gramplet docs, that includes leveraging GraphViz. ---
**Author:** ![StoltHD](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@StoltHD](https://gramps.discourse.group/u/StoltHD)\ **Post date:** [October 19, 2020, 7:48pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/4 "2020-10-19T19:48:49Z")
Well, by using many of those existing libraries, interoperability comes as a positive addition. And I asked about it only for use in Gramps, LONG before I started to understand all that about lack of interoperability in the export formats and the benefit of using other tools in addition to gramps… ---
**Author:** ![prculley](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/prculley/32/41_2.png) [@prculley](https://gramps.discourse.group/u/prculley)\ **Post date:** [October 19, 2020, 9:44pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/5 "2020-10-19T21:44:14Z")
FYI Gramps has been around a lot longer than some of those libraries; just so you understand why some things seem to be done less than optimally. Since Gramps for Windows and OSX needs to include in the bundle any library used (at least ones that use compiled code), we want to use care in adding new dependencies. For instance, NumPy is a huge dependency, probably near the same size as the rest of Gramps, so we have resisted adding it even though at least one addon could use it. We recently have started using a couple of pure Python dependencies in addons, the addon itself loads the dependency during its first run. Even there the Python module cannot depend on having dependencies of its own available; this can get out of hand quickly. But this would be quite difficult to do in the more general case. If you have a good use case for a library that is not already present, discuss it with the developers on the developers email list; if it is small, or might be more generally useful, we may want to make it part of the standard set. ---
**Author:** ![Mattkmmr](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/mattkmmr/32/410_2.png) [@Mattkmmr](https://gramps.discourse.group/u/Mattkmmr)\ **Post date:** [October 19, 2020, 11:51pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/6 "2020-10-19T23:51:52Z")
> [@emyoulation](#): > > While looking at a [new feature for Gramps](https://gramps-project.org/wiki/index.php/Addon:DNAgramplet), it struck me that the add-on was storing CSV data in a Gramps Note and some of the bugs were standard parsing problems. And that this new Note type probably needed to embed headers for future-proofing. Particularly since the DNA segment map data could come from different sources… where the same (untransformed) data can be expected but in differing order and with differing header labels. Which is the most simple crosswalk situation. You don’t need new libraries for that. An exceptions in the DNA gramplet code would be enough to handle bad formatted data in the note and stop Gramps or the addon from crashing. Ignore/skip the data and inform the user with a message which note causes problems. ---
**Author:** ![StoltHD](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@StoltHD](https://gramps.discourse.group/u/StoltHD)\ **Post date:** [October 20, 2020, 12:07am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/7 "2020-10-20T00:07:34Z")
Seriously, I have got the really good answer “fork it and do it your self” enough times now, trying to ask for updated libraries or inclutions to the windows AIO… so I do not see the point anymore! ---
**Author:** ![emyoulation](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/emyoulation/32/67_2.png) [@emyoulation](https://gramps.discourse.group/u/emyoulation)\ **Post date:** [October 20, 2020, 12:16am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/8 "2020-10-20T00:16:15Z")
That Gramps predates libraries isn’t much of a surprise. Data Mining and ‘Big Data’ have become major buzz words in the last decade. I expect that a lot of the functions that used to be built from scratch are now simply standardized library calls. And those routines have probably been through quite a few optimization cycles for advanced interactions like multi-threaded CPUs. Although the opposite can happen where a custom coded routine is more agile. It may not be buried under all the bells & whistles. Error handling for edge cases can add a lot of overhead. ---
**Author:** ![emyoulation](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/emyoulation/32/67_2.png) [@emyoulation](https://gramps.discourse.group/u/emyoulation)\ **Post date:** [October 20, 2020, 12:30am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/9 "2020-10-20T00:30:42Z")
Well, there’s the other approach. If you have a local college doing Python programming or data analysis courses, the professors are often looking for good, limited-scope projects to assign to advanced students. (Projects that are not “yet another” game!) They cannot use last semester’s assignments because students have always been very efficient cheaters. Besides a grade, a sole coder credit for a project add-on (where there are 10s of thousands of users) is a good resume item for a student. With Open Source, a prospective employer can look at their code and documentation. Plus they can review their interactions with other developers & clients. So, if you have people skills (something I lack), try promoting Gramps as a project platform. Ask the professor to provide an example of what they hand out as a Design Spec. Then write up a design spec for what you want. ---
**Author:** ![StoltHD](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@StoltHD](https://gramps.discourse.group/u/StoltHD)\ **Post date:** [October 20, 2020, 2:03am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/10 "2020-10-20T02:03:56Z")
sorry, don’t know anyone that want to use time on a lineage-linked system… I have already tried, but when they needed to also create import/exports for interchangeable formats, just to be able to use Gramps with other tools (they are like me, they seriously think that gedcom should be deprecated from all systems), they just wasn’t interested as long as their Excel Sheets and networkx and plotly scripts did the job for them. And I have already started convert my projects that I was going to share to other formats, so I stopped the job getting it to Gramps xml, it was to much hassle for me since I’m neither a xml developer nor a python programmer. so my hacks will stay private I can only contribute ideas, but when those ideas is not of interest to implement, then its not much more to do. ---
**Author:** ![emyoulation](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/emyoulation/32/67_2.png) [@emyoulation](https://gramps.discourse.group/u/emyoulation)\ **Post date:** [October 20, 2020, 2:45am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/11 "2020-10-20T02:45:12Z")
You making an incorrect assumption about the approach. The teachers & student don’t need to have an interest in the final productivity in Genealogy. It is simply a gradeable exercise in a real-world task. A class of 60 could each have different small exercises or larger 3-5 person exercises. The modularity of Gramps makes it an ideal free platform for a wide variety of exercises. ---
**Author:** ![StoltHD](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@StoltHD](https://gramps.discourse.group/u/StoltHD)\ **Post date:** [October 20, 2020, 8:03am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/12 "2020-10-20T08:03:44Z")
I dont think that’s how it works in Norway… and anyway, as I say I dont know any “professors” ---
**Author:** ![emyoulation](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/emyoulation/32/67_2.png) [@emyoulation](https://gramps.discourse.group/u/emyoulation)\ **Post date:** [October 22, 2020, 11:55am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/13 "2020-10-22T11:55:23Z")
**Mathplotlib overhead** > [@Hotspot heatmap for event clusters](https://gramps.discourse.group/t/hotspot-heatmap-for-event-clusters/776/17): > > If you use matplotlib, it needs at least 8MB of libraries and if you use cartopy, it also needs matplotlib, so the gramps package will grow aproximately 16MB for windows and mac users. > It’s difficult to add more libraries to gramps. > > The gramps package is already very big. (100MB for Macs and 65MB for Windows) > Linux is better managed for that. > > matplotlib uses its own maps. > cartopy can use some other maps (stamen, osm, google …), ---
**Author:** ![StoltHD](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@StoltHD](https://gramps.discourse.group/u/StoltHD)\ **Post date:** [October 22, 2020, 4:36pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/14 "2020-10-22T16:36:00Z")
> [@Hotspot heatmap for event clusters](https://gramps.discourse.group/t/hotspot-heatmap-for-event-clusters/776/17): > > The gramps package is already very big. (100MB for Macs and 65MB for Windows) > Linux is better managed for that. For us Windows users, 60-100MB software installations are not much… even my Notebook software are bigger than that… But I do agree that it’s good to try to limit the size… But 50MB extra for great functionality are not a problem for most researchers, when other software packages takes 4-500MB and some of them up over 1,5GB. I have photography and video software on my computer that each takes 1,5-3GB when installed… ---
**Author:** ![DavidMStraub](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/davidmstraub/32/7_2.png) [@DavidMStraub](https://gramps.discourse.group/u/DavidMStraub)\ **Post date:** [October 24, 2020, 8:08am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/15 "2020-10-24T08:08:11Z")
I don’t see the point why people here are arguing over whether a dependency is too big of an overhead or not. Let the user decide! It’s totally obvious to me that all the packages mentioned by @emyoulation in the original post would not be dependencies of the core Gramps, but of addons. So where is the problem? Why can’t we simply allow arbitrary Python dependencies _in addons_? Just declare the dependencies in the `register` method and let the user take care of the install. Something like: ```python import pkg_resources from importlib.metadata import PackageNotFoundError, version for requirement in addon.requirements: try: version(pkg_resources.Requirement.parse(requirement)) except PackageNotFoundError: raise SomeAppropriateGrampsError( "This addon requires the installation of the {} package" .format(requirement) ) ``` If a user wants to use `scipy` for something, I’m pretty sure they also know how to install it! You could even go one step further and install the package programmatically. This is done e.g. by one of my favourite Python packages, where I also took the inspiration for the above snippet: [see here](https://github.com/home-assistant/core/blob/dev/homeassistant/util/package.py). ---
**Author:** ![Mattkmmr](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/mattkmmr/32/410_2.png) [@Mattkmmr](https://gramps.discourse.group/u/Mattkmmr)\ **Post date:** [October 24, 2020, 9:13am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/16 "2020-10-24T09:13:48Z")
> [@DavidMStraub](#): > > If a user wants to use `scipy` for something, I’m pretty sure they also know how to install it! @DavidMStraub Addon devs yes, but I’m afraid that many Gramps users just might be overchallenged with additional installations. It would be great if we could provide the installation of addon dependencies together with the download in the addon manager. ---
**Author:** ![DavidMStraub](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/davidmstraub/32/7_2.png) [@DavidMStraub](https://gramps.discourse.group/u/DavidMStraub)\ **Post date:** [October 24, 2020, 11:29am UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/17 "2020-10-24T11:29:28Z")
Yes, that would certainly be optimal. But that only makes sense when installing the dependencies with `pip`. But I don’t see a problem with that, even on Windows (even though I have never tried). By the way, concerning > [@prculley](#): > > We recently have started using a couple of pure Python dependencies in addons, the addon itself loads the dependency during its first run. Even there the Python module cannot depend on having dependencies of its own available; this can get out of hand quickly. I don’t quite agree because every single package mentioned by @emyoulation can be installed with `pip` without any problems nowadays, thanks to wheels. Even PyTorch and Scipy! This is very different from the situation 3 or 4 years ago when one had to compile C extensions. I’m not saying that we should start `pip`-installing dependencies of the _core_ Gramps package - that would be a bad idea. But having an addon manager that automatically installs addon dependencies with `pip` in the background would be perfectly possible and very useful in my opinion. ---
**Author:** ![emyoulation](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/emyoulation/32/67_2.png) [@emyoulation](https://gramps.discourse.group/u/emyoulation)\ **Post date:** [November 4, 2020, 2:44pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/18 "2020-11-04T14:44:53Z")
My initial interest was in what functionalities were being done manually by Add-ons but are now available in Python? (And where the built-ins might have more extensive error handling and multi-core threading support.) (Is Gramps 5.1.3 at Python 3.3? I see GitHub is at 3.5) The particular functionality that that inspired question is the redundantly implemented variations of CSV import & export. My thought was that we might at least encourage new add-on development into leveraging those built-ins. ---
**Author:** ![Mattkmmr](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/mattkmmr/32/410_2.png) [@Mattkmmr](https://gramps.discourse.group/u/Mattkmmr)\ **Post date:** [November 4, 2020, 4:00pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/19 "2020-11-04T16:00:48Z")
@emyoulation regarding multi-core threading support: I don’t think there is a single addon right now doing so heavy calculations that multi-core threading is needed. The other addons are limited by Gramps and the database read/write, so multi-core threading added in addons wouldn’t speed anything up. ---
**Author:** ![DavidMStraub](https://yyz2.discourse-cdn.com/free1/user_avatar/gramps.discourse.group/davidmstraub/32/7_2.png) [@DavidMStraub](https://gramps.discourse.group/u/DavidMStraub)\ **Post date:** [November 4, 2020, 4:19pm UTC](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839/20 "2020-11-04T16:19:35Z")
I actually opened a pull request (still open) with a very small change allowing to use the CLI database handler class to open the database in read-only mode, which would allow multi-threaded read-only operations on an SQLite Gramps DB. Feel free to add your 👍😉 The motivation for this is using multithreading in a web application ([https://github.com/gramps-project/web-api/issues/16](https://github.com/gramps-project/web-api/issues/16)). [Next page](https://gramps.discourse.group/t/datamining-standard-libraries-for-gramps/839.md?page=2)