Forms add-on/plugin

Help
41

I am sorry but we are losing the plot here, we are looking for the simplest possible (least labour intensive) way of getting “Family History” data from numerous sources into GRAMPS. The big commercial operators and even the free one are not under any circumstance going to let the general public access their raw data in whatever format and it is why they persist in using GEDCOM for us plebs (there is no financial gain for them and never will be) because of its limitations and because they are all indebted in some way to the The Latter Days Saints organisation from which they either evolved or they negotiated access to the data and resources of that organisation.

Everything other than being able directly to scan convert/transcribe and import images from certificates, census, wills etc, etc into GRAMPS is just causing needless/wasted effort whether by retyping or software development.

Last word GEDCOM is a side issue as long as there is a plugin/addon that can convert/export a GRAMPS database to which ever version you want then do no more, forget it, walk away.
phil

1 Like
42

Ultimately the only way to do so would be through web requests or simulation programs like selenium if there is some degree of direct access protection. Actually once I got some of the stuff for my forms addon worked out a lot of the code could be reused for something like this. The only thing is that each website would need a script written specifically for it, but it wouldn’t be very hard. The user could store credentials for sites for the program to use, or it could even open a window for the user to enter credentials directly before requests are sent via selenium. Authenticated requests could then be sent and whether through structured apis if the site uses that or through web parsing with something like beautiful soup all he information a user sees when using the site could be accessed.

I did something similar when I made a program to download all my matches trees off of FTDNA to help find MRCAs but when I posted the program to a facebook group I got a lot of angry messages about privacy infringement so I took it down.

43

Hmmm. I thought a lot of angry messages on Facebook go with posting ‘hello’, ‘goodbye’ or ‘what?’. A lot of unhappy people venting there.

That program sounds interesting too. My brother has a FTDNA upload and we haven’t had much luck wading through the matches. That tool sounds like it would help.

Maybe it could be adapted to the RootsTech cousins download too.

44

I get that, but what does that mean? When we accept that the big guys and gals don’t make their raw data available, and focus on the small ones instead, there are still lots of sources available, in a large variety of formats, some as CSV, others as PDFs, from people that made transcriptions from forgotten church books, often formatted to be human readable.

And then there are also lots of sites local sites for which it’s often not worth the trouble to write a screen scraper for, or a more advanced client that simulates a web browser and extracts the data in a format that we like, from which we simply copy paste some formatted texts that do contain relevant data.

And if such text already exist in notes, in Evernote, Gramps, Joplin, Word, or Zotero, just to name a few, they often have a structure that you can use to extract data, although it probably can’t be fully automated. That could mean that like in that old version of Family Tree Maker, you can mark pieces of text with a particular meaning, so that they can be translated to real structured source data. I have many of such notes, in Gramps and Word.

45

“I get that, but what does that mean? When we accept that the big guys and gals don’t make their raw data available, and focus on the small ones instead, there are still lots of sources available, in a large variety of formats, some as CSV, others as PDFs, from people that made transcriptions from forgotten church books, often formatted to be human readable.”

I am a firm believer in the art of the possible and not the completely unimaginable.

We all probably have thousands of documents we have paid for (or not)and then may have hand transcribed. If in the future we could continue paying (or not) for the downloads in whatever form and then could use OCR (never had great success with this) or AI to pass this to a GRAMPS form I would be happy.

“the big guys and gals” these are the type of people you would need a very long handled spoon to sup from the same bowl with.
They will never give access to others for free what they perceive to be their meal ticket for life.

It would be nice to dream of such a Utopia but we are dealing here with Family History not Fairy Tales.

Also it mainly covers the legal aspects of the situation as long as you are not going to publish in major works and are not infringing copyright. As Ancestry does at this minute for example by accepting uploads of copies of the UK 1921 Census and then because or their terms and conditions they lay claim to these documents.

The only legal way of doing this presently lies with the likes of FTM (other programs are available) syncing to Ancestry and from recent conversations I have had with unhappy FTM users I would not go down that route.

phil

46

For me, it’s not so much the person data that I want to more easily harvest, but rather the details that I use when creating sources and their repository references. From a single web page (a FamilySearch catalog entry), I might create several such objects in Gramps. When I look at the page source, I can see that there’s some structured data within a script. Maybe some kind of browser extension could extract it?

For example,within the source of this page there is the following data. (Ignore the bolding; Discourse is doing that, I’m not trying to emphasize that part.)

var data = {“titleno”:“8141”,“articleno”:“”,“title”:“Parish registers”,“subtitle”:“”,“inclusive_dates”:“1841-1920”,“title_subtitle_sort”:“parish register”,“non_file_chars”:“0”,“edition”:“”,“statement_responsibility”:“”,“summary_contents”:“”,“numeric_designation”:“”,“media_type”:“MAN”,“format”:“Manuscript”,“format_addendum”:“Manuscript on Film”,“available_online”:“Y”,“oclc_record_number”:“866780495”,“is_collection_record”:“”,“author”:[{“seq”:“1”,“authorno”:“281712”,“givenname”:“”,“surname”:“Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario)”,“date”:“”,“numerals”:“”,“title”:“”,“work”:“”,“identification”:“”,“fullname”:“Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario)”,“display_text”:“Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario)”,“type”:“Main Author”},{“seq”:“2”,“authorno”:“281712”,“givenname”:“”,“surname”:“Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario)”,“date”:“”,“numerals”:“”,“title”:“”,“work”:“”,“identification”:“”,“fullname”:“Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario)”,“display_text”:“Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario)”,“type”:“Repository”}],“language”:[{“seq”:“1”,“text”:“English”},{“seq”:“3”,“text”:“Latin”}],“note”:[{“seq”:“1”,“text”:“<font color="red">Ontario, Roman Catholic Church Records are available online, click <a href="/search/collection/1927566"Target=”_blank">here.”,“type”:“RSLNK”,“note_link”:{“text”:“<font color="red">Ontario, Roman Catholic Church Records are available online, click <a href="/search/collection/1927566"Target=”_blank">here.”}},{“seq”:“2”,“text”:“Microfilm of original records in possession of the Purification of the Blessed Virgin Mary Rectory, Lindsay, Ontario.”,“type”:“LOC”},{“seq”:“3”,“text”:“Text in English and Latin.”,“type”:“LANG”},{“seq”:“4”,“text”:“Also known as St. Mary’s.”,“type”:“SUMM”},{“seq”:“5”,“text”:“Also covers Emily and Ops Township in Victoria County, Ontario.”,“type”:“SUMM”}],“film_note”:[{“seq”:“1”,“text”:“Baptisms and marriages, 1841-1849 Marriages, 1847-1850 Baptisms, marriages, confirmations, 1850-1865 Baptisms, marriages, burials, 1866-1886 (includes confirmations to 1907–some out of chron. order) Births and marriages, 1841-1883 Marriages, 1887-1902 (partially indexed)”,“type”:“FCONT”,“record_type”:“”,“volume”:“”,“inclusive_dates”:“”,“location”:“Family History Library”,“copy_location”:[“”,“Howell Michigan Family History Center”,“Kyle Texas Family History Center”,“Los Angeles California FamilySearch Library”,“Cody Wyoming Family History Center”,“BYU Family History Center”,“Kingston Ontario Family History Center”,“Hamilton Ontario Family History Center”,“Calgary Alberta Family History Center”,“Palmer Alaska Family History Center”,“Littleton Colorado Family History Center”,“San Mateo California Family History Center”,“Puyallup Washington Family History Center”,“Edmonton Alberta Riverbend Family History Center”,“Orlando Florida Family History Center”,“Lansing Michigan Family History Center”],“copy_location_ex”:[{“copy_location_id”:“”,“copy_location_name”:“”},{“copy_location_id”:“3335372”,“copy_location_name”:“Howell Michigan Family History Center”},{“copy_location_id”:“3340147”,“copy_location_name”:“Kyle Texas Family History Center”},{“copy_location_id”:“3312593”,“copy_location_name”:“Los Angeles California FamilySearch Library”},{“copy_location_id”:“3342875”,“copy_location_name”:“Cody Wyoming Family History Center”},{“copy_location_id”:“3340228”,“copy_location_name”:“BYU Family History Center”},{“copy_location_id”:“3343979”,“copy_location_name”:“Kingston Ontario Family History Center”},{“copy_location_id”:“3343839”,“copy_location_name”:“Hamilton Ontario Family History Center”},{“copy_location_id”:“3327868”,“copy_location_name”:“Calgary Alberta Family History Center”},{“copy_location_id”:“3329585”,“copy_location_name”:“Palmer Alaska Family History Center”},{“copy_location_id”:“3326594”,“copy_location_name”:“Littleton Colorado Family History Center”},{“copy_location_id”:“3331865”,“copy_location_name”:“San Mateo California Family History Center”},{“copy_location_id”:“3342190”,“copy_location_name”:“Puyallup Washington Family History Center”},{“copy_location_id”:“3343383”,“copy_location_name”:“Edmonton Alberta Riverbend Family History Center”},{“copy_location_id”:“3332594”,“copy_location_name”:“Orlando Florida Family History Center”},{“copy_location_id”:“3335178”,“copy_location_name”:“Lansing Michigan Family History Center”}],“geo_collection”:“United States & Canada”,“shelf”:“2nd Floor Film”,“item_image_start_no”:“”,“filmno”:“1305530”,“items”:“”,“digital_film_no”:“5106861”,“digital_film_rights”:“UNREST”,“fs_indexed”:“Y”,“fs_indexed_collection_id”:“1927566”,“fs_indexed_film_number”:“5106861”},{“seq”:“2”,“text”:“Baptisms, 1887-1902 (indexed; includes a few marriages licenses at the end; includes confirmations) Burials, 1887-1902 Baptisms, 1902-1910 (copy; indexed) Marriages, 1902-1907 Marriages, 1905-1910”,“type”:“FCONT”,“record_type”:“”,“volume”:“”,“inclusive_dates”:“”,“location”:“Family History Library”,“copy_location”:[“”,“Howell Michigan Family History Center”,“Kyle Texas Family History Center”,“Los Angeles California FamilySearch Library”,“Cody Wyoming Family History Center”,“San Mateo California Family History Center”,“BYU Family History Center”,“Edmonton Alberta Riverbend Family History Center”,“Tucson Arizona West Family History Center”,“Orlando Florida Family History Center”,“Sudbury Ontario Family History Center”],“copy_location_ex”:[{“copy_location_id”:“”,“copy_location_name”:“”},{“copy_location_id”:“3335372”,“copy_location_name”:“Howell Michigan Family History Center”},{“copy_location_id”:“3340147”,“copy_location_name”:“Kyle Texas Family History Center”},{“copy_location_id”:“3312593”,“copy_location_name”:“Los Angeles California FamilySearch Library”},{“copy_location_id”:“3342875”,“copy_location_name”:“Cody Wyoming Family History Center”},{“copy_location_id”:“3331865”,“copy_location_name”:“San Mateo California Family History Center”},{“copy_location_id”:“3340228”,“copy_location_name”:“BYU Family History Center”},{“copy_location_id”:“3343383”,“copy_location_name”:“Edmonton Alberta Riverbend Family History Center”},{“copy_location_id”:“3330494”,“copy_location_name”:“Tucson Arizona West Family History Center”},{“copy_location_id”:“3332594”,“copy_location_name”:“Orlando Florida Family History Center”},{“copy_location_id”:“3343960”,“copy_location_name”:“Sudbury Ontario Family History Center”}],“geo_collection”:“United States & Canada”,“shelf”:“2nd Floor Film”,“item_image_start_no”:“”,“filmno”:“1305531”,“items”:“”,“digital_film_no”:“5106860”,“digital_film_rights”:“UNREST”,“fs_indexed”:“Y”,“fs_indexed_collection_id”:“1927566”,“fs_indexed_film_number”:“5106860”},{“seq”:“3”,“text”:“Baptisms, 1902-1920 (original; includes 1 entry for 1921) Baptisms, 1902-1919 (copy; indexed; another filming of 1902-1910) Marriages, 1902-1907 (another filming), 1917-1920 Marriages, 1911-1917 (partially indexed)”,“type”:“FCONT”,“record_type”:“”,“volume”:“”,“inclusive_dates”:“”,“location”:“Family History Library”,“copy_location”:[“”,“San Mateo California Family History Center”,“St. Catharines Ontario Family History Center”,“Brampton Ontario Family History Center”,“Barrie Ontario Family History Center”,“Lansing Michigan Family History Center”],“copy_location_ex”:[{“copy_location_id”:“”,“copy_location_name”:“”},{“copy_location_id”:“3331865”,“copy_location_name”:“San Mateo California Family History Center”},{“copy_location_id”:“3343847”,“copy_location_name”:“St. Catharines Ontario Family History Center”},{“copy_location_id”:“3344096”,“copy_location_name”:“Brampton Ontario Family History Center”},{“copy_location_id”:“3366480”,“copy_location_name”:“Barrie Ontario Family History Center”},{“copy_location_id”:“3335178”,“copy_location_name”:“Lansing Michigan Family History Center”}],“geo_collection”:“United States & Canada”,“shelf”:“2nd Floor Film”,“item_image_start_no”:“”,“filmno”:“2028184”,“items”:“Items 10-13”,“digital_film_no”:“8265616”,“digital_film_rights”:“NO_ACC”}],“physical”:{“seq”:“1”,“materialtp”:“”,“length”:“on 3 microfilm reels”,“physical_chars”:“”,“dim_use”:“35 mm.”,“acc_material”:“”,“acc_desc”:“”,“physical_display”:“on 3 microfilm reels ; 35 mm.”,“display_text”:“on 3 microfilm reels ; 35 mm.”},“publisher”:{“seq”:“1”,“pos”:“1”,“name”:“Filmed by the Genealogical Society of Utah”,“place”:“Salt Lake City, Utah”,“date”:“1981-1996”},“subject”:[{“seq”:“1”,“subjectno”:“406235”,“text”:“Canada, Ontario, Victoria, Lindsay - Church records”,“type”:“TRACE”,“geo_name”:“174000”},{“seq”:“2”,“subjectno”:“406236”,“text”:“Canada, Ontario, Victoria, Emily Township - Church records”,“type”:“TRACE”,“geo_name”:“173991”},{“seq”:“3”,“subjectno”:“406237”,“text”:“Canada, Ontario, Victoria, Ops Township - Church records”,“type”:“TRACE”,“geo_name”:“174006”}],“xref”:{“titleno”:“1927566”,“title”:“Ontario, Roman Catholic Church records”,“subtitle”:“COLLECTION RECORD”,“link_type”:“Digital Collection”,“inclusive_dates”:“1760-1923”,“author_name”:“Catholic Church parishes (Ontario)”},“display_title”:“Parish registers, 1841-1920”,“subjectSname”:null,“subjectLocality”:[{“seq”:“1”,“subjectno”:“406235”,“text”:“Canada, Ontario, Victoria, Lindsay - Church records”,“type”:“TRACE”,“geo_name”:“174000”},{“seq”:“2”,“subjectno”:“406236”,“text”:“Canada, Ontario, Victoria, Emily Township - Church records”,“type”:“TRACE”,“geo_name”:“173991”},{“seq”:“3”,“subjectno”:“406237”,“text”:“Canada, Ontario, Victoria, Ops Township - Church records”,“type”:“TRACE”,“geo_name”:“174006”}],“subjectLibCongress”:null};
var details = [[“authorsLabel”,[“<a href="#" onclick="FS.PushState.pushState(‘count=20&query=%2Bauthor_id%3A281712’, ‘/search/catalog/results’); return false;" target="_self">Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario) (Main Author)”,“<a href="#" onclick="FS.PushState.pushState(‘count=20&query=%2Bauthor_id%3A281712’, ‘/search/catalog/results’); return false;" target="_self">Catholic Church. Purification of the Blessed Virgin Mary (Lindsay, Ontario) (Repository)”]],[“formatLabel”,[“Manuscript/Manuscript on Film”]],[“langLabel”,[“English”,“Latin”]],[“publicationLabel”,[“Salt Lake City, Utah : Filmed by the Genealogical Society of Utah, 1981-1996”]],[“physicalLabel”,[“on 3 microfilm reels ; 35 mm.”]],[“referencesLabel”,[“(Digital Collection) <a href="/search/catalog/1927566 " target="_self">Ontario, Roman Catholic Church records : COLLECTION RECORD, 1760-1923 / Catholic Church parishes (Ontario)”]]];
var availability = “Family%20History%20Library”;
var getPermissions = true;

47

Renee

Matters of privacy and copyright always invoke strong feelings.

If you have a public tree on a DNA matching website in my opinion you have surrendered your privacy rights in order to gain the benefits (caveat you should have not put living people on this platform).

Where it becomes an issue is if by chance you (as a third party) then have given access to “living” people to others and I have had some conversations and confrontations about peoples belief’s in this area.

This why I use GRAMPS offline and therefore I can record everything I want without it going elsewhere (where this leaves GrampsWeb for the future is anyone’s guess). I do not use the living tag or the private tags (too much ongoing maintenance). Because I have my own definition of what is publishable or not
“Born before 1939 and provably dead for at least 10 years” (can be published) and can filter out on this basis for exporting information.

phil

1 Like
48

WOW
When I posted this simple question on consensus about the use of the Form plugin I could not have envisioned a deluge of 46 replies…

49

It would appear that this post has been seen as offensive, which was never my intention and hereby apologise unreservedly for that.
I am a cynic at heart and opposed to the profits being made by commercial organisations on the back of thousands of unpaid volunteers working over many years and lately very poorly paid individuals in many parts of the world.
I am especially cynical about these organisations and their seeming inability to respond to suggestions, criticism or complaint
phil

2 Likes
50

Nothing offensive in that text what so ever…

So, you have nothing to apologize for…

51

I have absolutely no idea where that may have come from Philip. No offense taken here.

52

Let’s not delve into it further.

I share Phil’s cynicism about the big commercial players. When they charge customers to share their personal research and then charge visitors to view that shared information, it is usurious. When they lay claim to copyright of that information, it is purely theft. When they buy competitors with assurances of continued access but hidden plans to kill it off, they are using capitalism in a duplicitous and malicious way.

So, when a conversation touches on these evils, that evil cannot help but contaminate the rest of the reply.

2 Likes
53

Ha! Just means you have inspired some interesting conversation.

1 Like
54

I’ve gone ahead and added this to my list of things to do after finishing my forms extension update. Absolutely doable.

I’ve been collecting research articles in Zotero in anticipation for incorporating this. Not only is it helpful for everyone, it has the added benefit of improving accessibility to Family History for individuals with waning or absent eyesight. Given the number of older folks interested in genealogy I think this is especially important.

2 Likes
55

A new plugin type may be helpful for this. People could then write an addon to extract data from a website that they use frequently.

2 Likes
56

It’s a very popular addon, and has been since I wrote the original about 15 years ago.

I have always seen source-based data entry as a three part process:

Firstly, the document should be transcribed. This is what the the Forms addon was intended to do. No knowledge of the source contents is required for this. It should be possible to transcribe a document in a foreign language that the user doesn’t understand.

Next, interpretations can be added to the raw transcription. This is where parts of the document can be given meaning. We can match a column to a type for example. This is something that Renée is working on. Users will be able to define mappings in templates that will allow them to create conclusion objects. Other interpretations may include translation and transliteration, matching places, calculating dates from ages, expanding abbreviations etc…

Finally we can use the information from the source as evidence for conclusions. The ability to automatically generate conclusion objects is an often requested enhancement.

This approach is not for everyone and the Forms addon is not the only way to achieve the goal. Personally, I find Forms most useful for census data, and I don’t use them for much else.

I am really pleased that we have a new developer who is taking this project forward.

4 Likes
57

Hear, hear! And I hope she helps us identify things that deter new developers from contributing from our project.

We’ve lost active participation from some really good people. And I’ve had several confide privately about being frustrated that their contributions had become so bogged down that they’d given up.

There are a several experimental items targeting novice UX that are in limbo. They need more exposure to generate the feedback so they can become real features. But how can that happen?

1 Like
58

A new thread to discuss the options would be a good starting point.

1 Like
59

As part of the new thread can we include something to deal with the following.
I have been asked several times to recommend a Family History program I usually give 2 or 3 alternatives including GRAMPS. On talking later some had tried GRAMPS but had chosen others.
The reasons being

  1. Too complex
  2. Too steep a learning curve
    Maybe would encourage others to use and develop if they could see a quicker return for their efforts.
    phil
60

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.