[MacPorts] #43145: Perl PortGroup fails to reinplace non-ascii Makefiles
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles ---------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Keywords: | Port: p5-yaml ---------------------+-------------------------------- When building p5-yaml on 10.8 and 10.9, sed fails with {{{ DEBUG: Executing reinplace: /usr/bin/sed {/^CCFLAGS *=/s/$/ /} < /opt/local/var/macports/build/_opt_mports_dports_perl_p5-yaml/p5.16-yaml/work/YAML-0.90/Makefile
@ file14 DEBUG: sed: RE error: illegal byte sequence }}} (see https://build.macports.org/builders/buildports-mavericks- x86_64/builds/2454) because the `Makefile` contains a non-ascii character (in Latin 1 encoding, invalid UTF).
This comes from the perl PortGroup: {{{ # CCFLAGS can be passed in to "configure" but it's not necessarily inherited. # LDFLAGS can't be passed in (or if it can, it's not easy to figure out how). post-configure { fs-traverse file ${configure.dir} { if {[file isfile ${file}] && [file tail ${file}] eq "Makefile"} { ui_info "Fixing flags in [string map "${configure.dir}/ {}" ${file}]" reinplace "/^CCFLAGS *=/s/$/ [get_canonical_archflags cc]/" ${file} reinplace "/^OTHERLDFLAGS *=/s/$/ [get_canonical_archflags ld]/" ${file} } } } }}} Suggestions from IRC: neverpanic:
set LC_ALL to C I think that's a common issue with ill-formatted UTF-8 files /usr/bin/sed changed on 10.8 and above You can either patch that to be valid UTF-8 or set configure.env LC_ALL=C in the Portfile It is kind of weird to use the locale settings for that. I mean you could set the correct locale for the file, but then any messages to your terminal would be broken…
anddam:
I'd escape that dollar sign
Dar1us:
it refuses to process files which aren't validly encoded given the current LC settings .. and by default that is UTF-8
-- Ticket URL: <https://trac.macports.org/ticket/43145> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Changes (by mojca@…): * port: p5-yaml => p5-test-base p5-yaml -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:2> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by dluke@…): see also: https://lists.macosforge.org/pipermail/macports- dev/2012-August/019993.html (yes the perl portgroup should be updated -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:3> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Old description:
When building p5-yaml on 10.8 and 10.9, sed fails with {{{ DEBUG: Executing reinplace: /usr/bin/sed {/^CCFLAGS *=/s/$/ /} < /opt/local/var/macports/build/_opt_mports_dports_perl_p5-yaml/p5.16-yaml/work/YAML-0.90/Makefile
@ file14 DEBUG: sed: RE error: illegal byte sequence }}} (see https://build.macports.org/builders/buildports-mavericks- x86_64/builds/2454) because the `Makefile` contains a non-ascii character (in Latin 1 encoding, invalid UTF).
This comes from the perl PortGroup: {{{ # CCFLAGS can be passed in to "configure" but it's not necessarily inherited. # LDFLAGS can't be passed in (or if it can, it's not easy to figure out how). post-configure { fs-traverse file ${configure.dir} { if {[file isfile ${file}] && [file tail ${file}] eq "Makefile"} { ui_info "Fixing flags in [string map "${configure.dir}/ {}" ${file}]" reinplace "/^CCFLAGS *=/s/$/ [get_canonical_archflags cc]/" ${file} reinplace "/^OTHERLDFLAGS *=/s/$/ [get_canonical_archflags ld]/" ${file} } } } }}}
Suggestions from IRC:
neverpanic:
set LC_ALL to C I think that's a common issue with ill-formatted UTF-8 files /usr/bin/sed changed on 10.8 and above You can either patch that to be valid UTF-8 or set configure.env LC_ALL=C in the Portfile It is kind of weird to use the locale settings for that. I mean you could set the correct locale for the file, but then any messages to your terminal would be broken…
anddam:
I'd escape that dollar sign
Dar1us:
it refuses to process files which aren't validly encoded given the current LC settings .. and by default that is UTF-8
New description: When building p5-yaml on 10.8 and 10.9, sed fails with {{{ DEBUG: Executing reinplace: /usr/bin/sed {/^CCFLAGS *=/s/$/ /} < /opt/local/var/macports/build/_opt_mports_dports_perl_p5-yaml/p5.16-yaml/work/YAML-0.90/Makefile
@ file14 DEBUG: sed: RE error: illegal byte sequence }}} (see https://build.macports.org/builders/buildports-mavericks- x86_64/builds/2454) because the `Makefile` contains a non-ascii character (in Latin 1 encoding, invalid UTF).
This comes from the perl PortGroup: {{{ # CCFLAGS can be passed in to "configure" but it's not necessarily inherited. # LDFLAGS can't be passed in (or if it can, it's not easy to figure out how). post-configure { fs-traverse file ${configure.dir} { if {[file isfile ${file}] && [file tail ${file}] eq "Makefile"} { ui_info "Fixing flags in [string map "${configure.dir}/ {}" ${file}]" reinplace "/^CCFLAGS *=/s/$/ [get_canonical_archflags cc]/" ${file} reinplace "/^OTHERLDFLAGS *=/s/$/ [get_canonical_archflags ld]/" ${file} } } } }}} Suggestions from IRC: neverpanic:
set LC_ALL to C I think that's a common issue with ill-formatted UTF-8 files /usr/bin/sed changed on 10.8 and above You can either patch that to be valid UTF-8 or set configure.env LC_ALL=C in the Portfile It is kind of weird to use the locale settings for that. I mean you could set the correct locale for the file, but then any messages to your terminal would be broken…
Dar1us:
it refuses to process files which aren't validly encoded given the current LC settings .. and by default that is UTF-8
-- Comment (by and.damore@…): The portgroup cannot possibly know what encoding will the produced Makefile use. Port p5-yaml is using ExtUtils::MakeMaker that in turn relies on Pod::Man. Pod::Man can produce utf8 Makefiles, [http://perldoc.perl.org/Pod/Man.html search utf8 in in this page] but seems that MakeMaker doesn't have any option to specify that, see [http://article.gmane.org/gmane.comp.lang.perl.modules.extutils- makemaker.devel/2074/ this request]. Since the portgroup is patching Makefiles we could sanitize the file by adding a new variable specifying a charset to convert from. In this case the portfile writer would specify something like {{{ perl5.makefile_encoding iso8859-1 perl5.setup YAML 0.90 }}} and the post-configure phase would take care of convert the file to UTF-8 (or any other target encoding depending on the darwin release) before reinplaces. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:4> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by dluke@…): ... or we could just set LANG=C and things will work just like they used to. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:5> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Changes (by jmr@…): * cc: jeremyhu@… (added) Comment: The nicest solution would be to iconv the Makefile.PL to UTF-8 first so any info that ends up being installed doesn’t have a weird encoding. But in any case, this is the exact reason why reinplace has a -locale flag. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:6> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by mojca@…): Just to explain where the weird encoding comes from. The original file `dist.ini` contains UTF-8 encoded {{{ author = Ingy döt Net }}} Then this apparently gets translated into {{{ "AUTHOR" => "Ingy d\x{f6}t Net", }}} in `Makefile.PL`. During the configure phase, the following command is called (more or less): {{{ perl5.18 Makefile.PL }}} which results in {{{ # --- MakeMaker ppd section: # Creates a PPD (Perl Package Description) for a binary distribution. ppd : $(NOECHO) $(ECHO) '<SOFTPKG NAME="$(DISTNAME)" VERSION="$(VERSION)">' > $(DISTNAME).ppd $(NOECHO) $(ECHO) ' <ABSTRACT>YAML Ain'\''t Markup Language (tm)</ABSTRACT>' >> $(DISTNAME).ppd $(NOECHO) $(ECHO) ' <AUTHOR>Ingy döt Net</AUTHOR>' >> $(DISTNAME).ppd $(NOECHO) $(ECHO) ' <IMPLEMENTATION>' >> $(DISTNAME).ppd $(NOECHO) $(ECHO) ' <ARCHITECTURE NAME="darwin-thread- multi-2level-5.18" />' >> $(DISTNAME).ppd $(NOECHO) $(ECHO) ' <CODEBASE HREF="" />' >> $(DISTNAME).ppd $(NOECHO) $(ECHO) ' </IMPLEMENTATION>' >> $(DISTNAME).ppd $(NOECHO) $(ECHO) '</SOFTPKG>' >> $(DISTNAME).ppd }}} in Latin 1 encoding. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:7> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by jmr@…): If the output is always Latin 1, that at least makes it easy to pass the right locale to reinplace. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:8> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by and.damore@…): Replying to [comment:6 jmr@…]:
The nicest solution would be to iconv the Makefile.PL to UTF-8 first so any info that ends up being installed doesn’t have a weird encoding.
I'm not sure that this would help since Pod::Man needs a specific flag to enable utf8 and seems that ExtUtils::MakeMaker isn't passing it when calling Pod::Man. Also you'd need to specify the =encoding in the temporary produced POD file. {{{ utf8 By default, Pod::Man produces the most conservative possible *roff output to try to ensure that it will work with as many different *roff implementations as possible. […] Be aware that, when using this option, the input encoding of your POD source must be properly declared unless it is US-ASCII or Latin-1. POD input without an =encoding command will be assumed to be in Latin-1, and if it's actually in UTF-8, the output will be double-encoded. }}}
But in any case, this is the exact reason why reinplace has a -locale flag.
Correct, so we could specify a ''perl5.reinplace_locale'' variable and pass that. Since the post-configure is traversing the filesystem checking for all "Makefile" named files and the encoding isn't necessarily the same for all files I think that this ''reinplace_locale'' should be a structure of {{file_name file_locale} …} couples that should be parsed per each file. Replying to [comment:8 jmr@…]:
If the output is always Latin 1, that at least makes it easy to pass the right locale to reinplace.
The output could be plain ASCII (but I guess that wouldn't be a problem), but not all modules use ExtUtiles::MakeMaker, I've seen at least on Makefile.PL directly generating the Makefile. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:9> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by mojca@…): We are neither recoding files nor changing any non-ascii portions of the text. So I would say that encoding is pretty much irrelevant even if it's not Latin 1, as long as it's kept intact. The "C" local should do just as well. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:10> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by mojca@…): See also #44630. At least the following ports are affected (but there might be more): * p5-io-all * p5-spiffy * p5-test-base * p5-yaml * p5-yaml-libyaml -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:11> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by dluke@…): Replying to [comment:10 mojca@…]:
We are neither recoding files nor changing any non-ascii portions of the text. So I would say that encoding is pretty much irrelevant even if it's not Latin 1, as long as it's kept intact. The "C" local should do just as well.
+1 for adding -locale C to the reinplace calls in the perl5 portgroup -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:12> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: closed Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: fixed | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Changes (by dluke@…): * status: new => closed * resolution: => fixed Comment: r123724 -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:13> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: closed Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: fixed | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by mojca@…): Please also remove the patch in the other four ports then. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:14> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: closed Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: fixed | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by dluke@…): Replying to [comment:14 mojca@…]:
Please also remove the patch in the other four ports then.
done, although since the perl5 portgroup change didn't break them, I don't believe I really had an obligation to correct them. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:15> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: closed Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: fixed | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by mojca@…): Thank you. I'm just cross-referencing the numbers (r123726, r123727, r123728, r123729). Sure, the ports weren't broken, but we would probably forget to remove the patch(es) later and could keep them forever without any good reason. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:16> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: closed Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: fixed | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by dluke@…): Replying to [comment:16 mojca@…]:
Sure, the ports weren't broken, but we would probably forget to remove the patch(es) later and could keep them forever without any good reason.
historically, it has been up to the individual port maintainers to handle non-breaking modifications to base (or portgroups), not the person/persons making changes to base or portgroups. -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:17> MacPorts <http://www.macports.org/> Ports system for OS X
#43145: Perl PortGroup fails to reinplace non-ascii Makefiles -----------------------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: closed Priority: Normal | Milestone: Component: ports | Version: 2.2.1 Resolution: fixed | Keywords: Port: p5-test-base p5-yaml | -----------------------------------+-------------------------------- Comment (by mojca@…): (You can probably also move the checksums out of the brackets in `p5-test- base`.) -- Ticket URL: <https://trac.macports.org/ticket/43145#comment:18> MacPorts <http://www.macports.org/> Ports system for OS X
participants (1)
-
MacPorts