Diacritics in UTF-8 Glossary File Not Working

[DigitalProf]

I have attempted several times to upload a glossary such that diacritics in French will be handled correctly. I am sure is in UTF-8 and have verified in various ways.

A related issue is that Excel by default includes a Byte Order Mark (BOM), which I have seen is included in the first item in the glossary. Using Notepad++ I have converted the encoding of the file to exclude the BOM, but that does not correct the problem. Here is the link to the glossary file that I uploaded to DeepL using Python.

I am also attaching below:

1. A screen grab of the "Save as" screen from Excel
2. The results of a dump of the most recent glossary that I uploaded as well as a screen grab of the problems int the translated document
3. The actual CSV
   
   
   

[daniel-jones-deepl]

Hi @DigitalProf Mike, thanks for creating this issue.

I reproduced the problem that the first term contains the byte order mark (BOM); there seems to be an error on our side. I've reported this issue to the team.

I also looked into Excel. Exporting using "CSV UTF-8 (Comma delimited)" gives the correct encoding, but includes the BOM. Unfortunately I could not find an easy way to omit the BOM.

As a workaround (until we can resolve the BOM issue on our side), could you try entering a dummy first entry in the CSV? For example "entry-to-be-ignored,entry-to-be-ignored". Your remaining glossary entries should be unaffected and work correctly. Please make sure the entries appear correctly in Excel -- when I open your link above, many of the entries already include wrong characters (I guess because Excel assumed the wrong encoding as the file does not contain a BOM).

[DigitalProf]

Thanks for the action on this issue, Daniel! I apologize for how the link to the Excel file works. The access to a OneDrive file via the browser does not give one a chance to state that the file is in fact in UTF-8. When the file is opened in Excel, the software asks for confirmation that the file is indeed in UTF-8. I should have tested the link myself. Sorry about that! :-)

As to the Byte Order Mark (BOM), Excel does in fact place that into the file by default. I have checked, but do not see how to do otherwise for exporting from Excel. I have, however, tested this aspect of the problem by opening the file in Notepad++ and changing the encoding scheme to remove the BOM. I have tested that, but it does not change how DeepL handles the file.

Last night, I sent along the Python code I used to upload the glossary. The code is from your site, but in copying the code into my message last evening, I believe that I now see the problem. I have checked this out, but I am thinking that I simply need to open the file in Python with UTF-8 encoding by adding this:

, encoding="utf-8"

If this is in fact the issue, I suggest that the sample code be changed on GitHub. It appears about half-way down the page on the GitHub site in the section, “Creating a glossary.”

Cheers,

Mike

Python Code for Creating a Glossary

[DigitalProf]

Thanks for your note! I responded on GitHub. Cheers, Mike Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Daniel Jones ***@***.***> Sent: Monday, August 15, 2022 4:55:09 AM To: DeepLcom/deepl-python ***@***.***> Cc: ***@***.*** ***@***.***>; Mention ***@***.***> Subject: Re: [DeepLcom/deepl-python] Diacritics in UTF-8 Glossary File Not Working (Issue #44) Hi @DigitalProf<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDigitalProf&data=05%7C01%7C%7Ccddafffa0bd04ff2b99308da7eaca336%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637961577199202339%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pzY0Hgp%2BjQRo%2F9f%2ByuEJKaV6dgy7%2FOjbl8%2FxhEmBuiw%3D&reserved=0> Mike, thanks for creating this issue. I reproduced the problem that the first term contains the byte order mark (BOM); there seems to be an error on our side. I've reported this issue to the team. I also looked into Excel. Exporting using "CSV UTF-8 (Comma delimited)" gives the correct encoding, but includes the BOM. Unfortunately I could not find an easy way to omit the BOM. As a workaround (until we can resolve the BOM issue on our side), could you try entering a dummy first entry in the CSV? For example "entry-to-be-ignored,entry-to-be-ignored". Your remaining glossary entries should be unaffected and work correctly. Please make sure the entries appear correctly in Excel -- when I open your link<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2F1drv.ms%2Fu%2Fs!ArbxA1RlknTQmuIju1W2LQOAzAtIWw%3Fe%3DbKC71P&data=05%7C01%7C%7Ccddafffa0bd04ff2b99308da7eaca336%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637961577199202339%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f87vHLxvK9gR7oWIWRAfqXl9pU46mkF%2F7pH52F2Aodo%3D&reserved=0> above, many of the entries already include wrong characters (I guess because Excel assumed the wrong encoding as the file does not contain a BOM). — Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDeepLcom%2Fdeepl-python%2Fissues%2F44%23issuecomment-1214883525&data=05%7C01%7C%7Ccddafffa0bd04ff2b99308da7eaca336%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637961577199202339%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mw7%2Fd2Nc%2B%2BYG%2BhMF3ge2kPJ9nt3YDHGgPIFZzWLRHkI%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABXOPRLPABSCHPDXUT5CNMTVZIOY3ANCNFSM56O4OF6A&data=05%7C01%7C%7Ccddafffa0bd04ff2b99308da7eaca336%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637961577199202339%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=90sjZQQROHDN8kckGULNYFnRh%2BmduZNwEdRh34lRBmU%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>