[MacPorts] #52420: Umlauts, accents are wrongly displayed in neomutt
#52420: Umlauts, accents are wrongly displayed in neomutt -----------------------------------+-------------------------------- Reporter: michele.marcionelli@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Keywords: | Port: neomutt -----------------------------------+-------------------------------- I updated mutt-devel to neomutt. Now the umlauts (ö,ü,...) and accents (è,á,...) are not displayed correctly anymore. For instance I get a '?' or '\123'. A fix would be very appreciated ;-) Bests, Michele -- Ticket URL: <https://trac.macports.org/ticket/52420> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Changes (by ryandesign@…): * owner: macports-tickets@… => leonardo@… -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:1> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by leonardo.schenkel@…): That's very odd. I use all sorts of accented characters as well and I never experienced any issue. I'll need some more details: - Does it happen with all e-mails or just some e-mails? - Does it happen in the UI as well? Try typing a mutt command with accents. - Which terminal do you use? The built-in Terminal? iTerm? Something else? - Could you please run the command `locale` and post the output? - Could you please run the command `mutt -v` and post the output? -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:2> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by cal@…): Try `unset LC_CTYPE` (and possibly also `unset LC_ALL`) before starting neomutt. It seems that a number of Terminals have a feature to helpfully automatically set the locale variables, but end up setting them to an unsupported value. Not setting them does not cause problems, but setting them to unsupported values renders special characters broken. I bisected this to https://github.com/neomutt/neomutt/commit/f28b4cace4a1f26eace3120ae08e0b9435..., which comes from mutt upstream, so is not a neomutt problem. In iTerm, unticking "Preferences > Profiles > $yourprofile > Terminal > Environment > Set locale variables automatically" and opening a new Shell session fixed this for me. Terminal.app has a similar setting. -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:3> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by michele.marcionelli@…): Just for your information: the old mutt-devel (Mutt 1.6.0 (2016-04-01)) was working well with the default settings, i.e. with the "set locale variables automatically" setting for Terminal (or iTerm) which correspond to following {{{locale}}}: {{{ LANG= LC_COLLATE="C" LC_CTYPE="UTF-8" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL= }}} Now with the new neomutt (NeoMutt 20160916 (1.7.0) - just installed today) I get following results: 1) with default {{{locale}}} (see above): - in the mail listing I read the subject "Z??rich" instead of "Zürich" - while reading the email I see in the header "Subject: Z??rich" and in the body "Z\303\274rich" 2) with {{{unset LC_CTYPE}}} I get this output of {{{locale}}} {{{ LANG= LC_COLLATE="C" LC_CTYPE="C" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL= }}} and I see: - in the mail listing I read the subject "Zrich" instead of "Zürich" - while reading the email both header and body is OK 3) with {{{export LC_ALL=en_US.UTF-8}}} I get this output of {{{locale}}} {{{ LANG= LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL="en_US.UTF-8" }}} and everything seems to work fine; also {{{export LC_ALL=en_US}}} oder {{{export LC_ALL=de_CH}}} work find. What is the difference with (1)? Why (1) don't work correctly with neomutt? I also noticed that not setting the LC_ALL but all other LC_* event to different values mutt will display umlauts and accents correctyle; but if I just unset one of the LC_* mutt will fail: {{{ LANG= LC_COLLATE="de_AT" LC_CTYPE="de_DE" LC_MESSAGES="en_US" LC_MONETARY="de_CH" LC_NUMERIC="fr_FR" LC_TIME="en_GB" LC_ALL= }}} ==> is ok for mutt {{{ LANG= LC_COLLATE="de_AT" LC_CTYPE="de_DE" LC_MESSAGES="en_US" LC_MONETARY="C" LC_NUMERIC="fr_FR" LC_TIME="en_GB" LC_ALL= }}} ==> is no more ok for mutt HTH ;-) Bests, Michele -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:4> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by cal@…): Yes, mutt-devel is expected to not have this problem, because it doesn't have the commit that changes this. Instead of updating mutt-devel to 1.7.0 (which has the same problem) we switched to neomutt based on mutt 1.7.0. A "C" locale not supporting special characters is not surprising to me. I'm not sure how mixing a single "C" locale into others interacts. I do not really know why some combinations don't work. The `mosh` remote shell client, for example, aborts on connection with LC_CTYPE=UTF-8 telling me that this is an invalid locale (even though `/usr/share/locale/UTF-8/LC_CTYPE` exists). We should probably report this to the developers of mutt, but the answer may well be that there's a problem with our locale configuration rather than with their code. -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:5> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by leonardo.schenkel@…): This is what I suspected, but I just wanted to confirm before blaming your set-up. Unfortunately the LC_CTYPE="UTF-8" that Terminal.app sets by default is not a valid recognized locale by a lot of software (the locale is supposed to be something like "de_CH.UTF-8"). The macOS userland can cope with it (and many BSD as well), but if you ever ssh to a Linux box for example you'll instantly get this warning: `setlocale: LC_CTYPE: cannot change locale (UTF-8)` because the value is rejected by the GNU userland. Mutt 1.7 is now being affected by this. Before 1.7 it didn't pay attention to your environment's locale (only to the locale you set in the configuration file), but that setting was removed and it is now determing the locale and the charset based on the environment variables, like virtually all other software does. Unfortunately it got bitten by the broken macOS defaults. This is so annoying in general that on my shell initialization scripts I have the following: {{{ if [ "${LC_CTYPE}" == "UTF-8" ]; then # OS X likes to set LC_CTYPE="UTF-8". In general this is supported by BSD # systems, but not Linux (which will become a problem when ssh'ing to # foreign systems). Check if it is possible to unset LC_CTYPE without # changing the charset, and unset it if that is true. if [ "$(LC_CTYPE= locale charmap)" == "UTF-8" ]; then unset -v LC_CTYPE fi fi }}} If you change your shell initialization scripts to contain `export LANG=de_CH.UTF-8` (or whatever is your locale of choice) *before* the snippet above, then LC_CTYPE will be automatically unset and you should not have any more of these issues. Note that as "cal" have correctly said, the change of behavior that you have experienced was introduced vanilla Mutt, not NeoMutt neither this port. All I can do in this situation is to suggest you to change your settings. -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:6> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by larryv@…): Replying to [comment:6 leonardo.schenkel@…]:
Unfortunately the LC_CTYPE="UTF-8" that Terminal.app sets by default is not a valid recognized locale by a lot of software (the locale is supposed to be something like "de_CH.UTF-8").
FWIW, my Terminal.app automatically uses a correct locale. My shell startup settings do not touch `LANG`. {{{ LANG=en_US.UTF-8 }}} I don’t know why it would use just “UTF-8” for anything. Perhaps this happens when the system Language & Region settings are customized? I use the stock “United States” settings. -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:7> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by leonardo.schenkel@…): I don't touch LANG as well. In all Macs I have ever owned, in all OS X versions I can remember, Terminal.app sets the locale like this: {{{ LANG=<locale from regional settings>.UTF-8 LC_CTYPE=UTF-8 }}} I never saw plain "C" for LANG, though, which is curious. Why it uses "UTF-8" for LC_CTYPE and not a well-formed locale name is a mystery to me, especially since it can do that just well for LANG — and since LANG is already defined as a UTF-8 locale, there's no need to set LC_CTYPE at all. I'm not an expert, but my understanding is that the relevant specs say that all locales must be in the form of `lang_country_variant.charset` with notable exceptions ("C", "POSIX"). Even though "UTF-8" exists as a locale in the Mac, it's not supposed to be a valid locale name. I even tried to create such a locale via `locale-gen` on Linux systems and it refuses to do so. -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:8> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by michele.marcionelli@…): Replying to [comment:7 larryv@…]:
Replying to [comment:6 leonardo.schenkel@…]: I don’t know why it would use just “UTF-8” for anything. Perhaps this happens when the system Language & Region settings are customized? I use the stock “United States” settings.
That was the problem. I'm using as Language "English" and Region "Switzerland" and this generate the wrong default {{{locale}}} mentioned above. Setting as Language "Italian (Switzerland)" or "German (...)" or "French (...)" then everything works fine. This is definitely not a *mutt problem... rather an OS X one. Thank you guys helping me with this problem. Bests, Michele -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:9> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by larryv@…): Replying to [comment:8 leonardo.schenkel@…]:
I don't touch LANG as well. In all Macs I have ever owned, in all OS X versions I can remember, Terminal.app sets the locale like this: {{{ LANG=<locale from regional settings>.UTF-8 LC_CTYPE=UTF-8 }}}
I have seen the opposite: I have never had `LC_CTYPE` set automatically. My environment has always contained `LANG` alone. If I had to, I would guess that the system doesn’t try to guess a country if you tweak the system settings in a way that doesn’t explicitly denote one. But I have no idea. UTF-8 is obviously just a character encoding without any locale information at all. -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:10> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
#52420: neomutt: Umlauts, accents are wrongly displayed ------------------------------------+------------------------ Reporter: michele.marcionelli@… | Owner: leonardo@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.3.4 Resolution: | Keywords: Port: neomutt | ------------------------------------+------------------------ Comment (by larryv@…): FYI, more information on the Mac locale fun: https://bugs.python.org/issue18378#msg215215 -- Ticket URL: <https://trac.macports.org/ticket/52420#comment:11> MacPorts <https://www.macports.org/> Ports system for the Mac operating system
participants (1)
-
MacPorts