Solved Bächlingen showing as Bächlingen

[StoltHD]

Connie,

I have had the same problems as you regarding the CSV from Gramps, but for some reason it has not been any pattern in it that I have managed to find… sometime it opens correct, sometimes the double- byte chatacters are wrong, as if they are exxported in a combination of windows default and utf-8…

Changing it to UTF-8 with BOM in Notepad++ has nearly always solved it with the files from Gramps, but I have had other CSV files that has been reported as UTF-8 in notepad and first after I convert them they shows up correct…

Two of them was because of depricated libraries (this had nothing to do with Gramps, but the problem was identical).

[philhoward]

having the correct BOM in there should make in work in all coding environments. way back when Unicode suggested having 16 bit coding for its 16 bit characters i felt a gut punch regarding the byte order issue. i have done some assembly programming on PDP-11 computers and remember the issues all to well just for numbers. then when they went to larger codes, another gut punch because now there are 3 different byte orders for 32-bit values. UTF-8 was the way that just worked well, but Microsoft hadn’t chosen it (in general, though Window can work with it). if programs always do I/O as 8-bit with UTF-8 encoding they can be portable. Python2 could do it and Python3 does it by default. Linux systems only have 8-bit I/O on x86 computers.

[StoltHD]

Yes its lot of problems regarding this on Windows Computers…

The latest Windows 10 can be set to use UTF-8 and also have a feature to set UTF-8 for any “Legacy” software, but that really mess things up…

I ried that with Legacy familtree (it is not a unicode programmed software), and even though they use a jet database (jet has been unicode/utf-8 enabled for years now), all the data with double byte characters are a mess…

sadly there are still to many libraries and scripts (even Pyton and JAVA libraries) that are coded with CP-1252 or ISO/IEC 8859-1:1998…

And when a library like that are used on a system (software, OS, etc) that use an unicode codepage, there will be a lot of mess…

I don’t know if that the case with the CSV export from Gramps, or IF there are some problem with Openrefine, I don’t use Libre-/Open Office for this, so for my part it has to be some other library.

I have had the same problem when exporting some Graphviz files from a few other software, and after some investigating it was the vis.js library that was the problem…

Same happen with the graphviz import to cytoscape, that add-on use a aged library, and couldn’t support utf-8.

The same thing has happened a few times for me with the CSV’s from Gramps, but I have not been able to find where the problem actually are, if it is the export or if it is the import to the different software I use. Only thing I know is if the file is opened in Notepad++ and set to utf-8 with bom, or converted if the first doesn’t work (even though the file is reported as utf-8), it usually import to any of the software without problems… but the problem is that this is also true for files edited in Excel and saved as utf-8…

A while back I asked if there could updates the D3.js libraries for some of the reports and graphs, because the library in use was like 4-5 years old (or more, don’t remember) and there was some known security problems with them and it was done.

Maybe it would be an idea to do a check and do an update of all libraries in Gramps that export- or create content that can be output to both text and graphical text files (i.e. graphviz)?