Scraping genealogical web site to add or update a person

[move this over to ideas if it does not exist.]

does Gramps have a means to scrape a web page about a person’s information and apply it to a person (create if not yet there) in the tree one is working on? for instance after i load my gedcom from Ancestry, i want to add some additional (about 30-40) people from Wikitree and Familysearch.

1 Like

I changed the category to ideas.

There is an addon Web Connect Pack which does searching websites, but I’m not aware of any addons scraping sites for information yet.

Dave & I recently revisited the US WebConnect for link rot. And that lead to a bunch of fixed links & new searches leveraging fuzzy dates. (We also expanded the documentation so you could fix the searches yourself. Although your customization are not stored in an .ini and will get wiped out with the next Gramplet tweak.)

I didn’t use that WebConnect tool before but I CERTAINLY use it a lot now.

But we also discovered that it needs some love from a Python coder. We didn’t see a way to pass some parameters to make the websearching URLs.

For instance, you can easily insert searching for first, last names, birth or death year. But if there is a way to insert a maiden name (or a custom attribute or Citation Page Number) then it isn’t clear how to reference those fields. You can’t search using other Category values… like searching a library for a Source’s Title or Author.

And if you configure Source web resources like WikiTree, FamilySearch, FindAGrave, HathiTrust, internet archive; you can not insert the WikiProfileID, profileID, memorialID, catalogRecordID, external-identifier into the PageNumber in a Citation and use Webconnect as a linkrot resistent way to reference online resources.

2 Likes

Hey,

Gramps itself doesn’t have a built-in feature for scraping web pages to automatically add or update people in your family tree. You’d likely need to create a custom script using Python with libraries like BeautifulSoup or Scrapy to extract the data from Wikitree or Familysearch, then format it properly to import into Gramps.

Good luck!