Item13424: Malformed UTF-8 character (unexpected end of string) fails for some languages on some perl versions.

The failure appears to be related to perl versions. It's failing on trunk.foswiki.org, which is running perl 5.16.3. It fails for only a few languages:

• Spanish
• French

And it currently fails on 5.16.3 (tested on foswiki.org and local perlbrew), Works with 5.18.2, 5.20.1 and 5.22-RC1

Malformed UTF-8 character (unexpected end of string) in string ne at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538.
 at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 379.
   CGI::Carp::realdie("Malformed UTF-8 character (unexpected end of string) in strin"...) called at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 468
   CGI::Carp::die("Malformed UTF-8 character (unexpected end of string) in strin"...) called at /usr/home/trunk.foswiki.org/core/lib/AssertOn.pm line 15
   Assert::__ANON__("Malformed UTF-8 character (unexpected end of string) in strin"...) called at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538
   Locale::Maketext::_compile(Foswiki::I18N::fr=HASH(0x804c38ac8), "Des id\x{e9}es, requ\x{ea}tes ou probl\x{e8}mes concernant [_1]\x{a0}? <a href='["...) called at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 224
   Locale::Maketext::maketext("<nop>Foswiki.org development testing", "mailto:webmaster\@foswiki.org?subject=Foswiki.org development "...) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/I18N.pm line 272
   Foswiki::I18N::maketext(Foswiki::I18N::fr=HASH(0x804c38ac8), "Ideas, requests, problems regarding [_1]? <a href='[_2]'>Send"..., "<nop>Foswiki.org development testing", "mailto:webmaster\@foswiki.org?subject=Foswiki.org development "...) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/MAKETEXT.pm line 57
   Foswiki::MAKETEXT(Foswiki=HASH(0x802e41ea0), Foswiki::Attrs=HASH(0x80536f258), Foswiki::Meta=HASH(0x80565a5a0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3405
   Foswiki::_expandMacroOnTopicRendering(Foswiki=HASH(0x802e41ea0), "MAKETEXT", "\"Ideas, requests, problems regarding [_1]? <a href='[_2]'>Sen"..., Foswiki::Meta=HASH(0x80565a5a0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3267
   Foswiki::_processMacros(Foswiki=HASH(0x802e41ea0), "<span class=\"foswikiRight\"> <a href=\"http://foswiki.org/\"><im"..., CODE(0x802d845a0), Foswiki::Meta=HASH(0x80565a5a0), 15) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3285
   Foswiki::_processMacros(Foswiki=HASH(0x802e41ea0), "%WEBCOPYRIGHT%", CODE(0x802d845a0), Foswiki::Meta=HASH(0x80565a5a0), 16) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3064
   Foswiki::innerExpandMacros(Foswiki=HASH(0x802e41ea0), SCALAR(0x8051e6030), Foswiki::Meta=HASH(0x80565a5a0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/INCLUDE.pm line 300
   Foswiki::__ANON__() called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 421
   eval {...} called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 413
   Error::subs::try(CODE(0x805659888), HASH(0x8057b5690)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/INCLUDE.pm line 356
   Foswiki::_includeTopic(Foswiki=HASH(0x802e41ea0), Foswiki::Meta=HASH(0x80370e7c8), HASH(0x8051e7558), Foswiki::Attrs=HASH(0x80567d528)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/INCLUDE.pm line 415
   Foswiki::INCLUDE(Foswiki=HASH(0x802e41ea0), Foswiki::Attrs=HASH(0x80567d528), Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3405
   Foswiki::_expandMacroOnTopicRendering(Foswiki=HASH(0x802e41ea0), "INCLUDE", "\"System.WebBottomBar, System.WebBottomBarExample\"", Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3267
   Foswiki::_processMacros(Foswiki=HASH(0x802e41ea0), " %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"..., CODE(0x802d845a0), Foswiki::Meta=HASH(0x80370e7c8), 16) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3064
   Foswiki::innerExpandMacros(Foswiki=HASH(0x802e41ea0), SCALAR(0x802d956c0), Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3590
   Foswiki::expandMacros(Foswiki=HASH(0x802e41ea0), " %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"..., Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Meta.pm line 3353
   Foswiki::Meta::expandMacros(Foswiki::Meta=HASH(0x80370e7c8), " %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"...) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI/View.pm line 479
   Foswiki::UI::View::_prepare(" %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"..., Foswiki::Meta=HASH(0x80370e7c8), 0) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI/View.pm line 465
   Foswiki::UI::View::view(Foswiki=HASH(0x802e41ea0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI.pm line 374
   Foswiki::UI::__ANON__() called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 421
   eval {...} called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 413
   Error::subs::try(CODE(0x8010914f8), HASH(0x802e41978)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI.pm line 500
   Foswiki::UI::_execute(Foswiki::Request=HASH(0x802e196a8), CODE(0x802c3be28), "view", 1) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI.pm line 326
   Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x802e196a8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Engine/CGI.pm line 99
   Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x80218d4b0)) called at /home/trunk.foswiki.org/core/bin/view line 29.

-- GeorgeClark - 22 May 2015

The following seems fixes it, but it is far from the optimal solution.

Principe, do the decode not at the Locale::Maketext::Lexicon level but higher, in the Foswiki's %MAKETEXT% macro level.

Currently, the Lexicon returns unicode characters, e.g. doing the decode_utf8 internally (and seems, something is wrong).

After the following patch, the Lexicon returns utf8 encoded byte string and Foswiki will do the decode. So moving the borders a bit closer into Foswiki. Not optimal, because for the same string will do more times the decode_utf8 but solves the problem. (at least in my test - needs more testing ).

--- refmast/foswiki/core/lib/Foswiki/I18N.pm   2015-05-21 14:20:40.000000000 +0200
+++ devmast/foswiki/core/lib/Foswiki/I18N.pm   2015-05-22 11:19:45.000000000 +0200
@@ -80,7 +80,7 @@
         unless (
             eval {
                 Locale::Maketext::Lexicon->import(
-                    { _decode => 1, $lang => [ Gettext => $langFile ] } );
+                    { $lang => [ Gettext => $langFile ] } );
                 1;
             }
           )
--- refmast/foswiki/core/lib/Foswiki/Macros/MAKETEXT.pm   2015-04-11 16:42:03.000000000 +0200
+++ devmast/foswiki/core/lib/Foswiki/Macros/MAKETEXT.pm   2015-05-22 11:36:56.000000000 +0200
@@ -54,7 +54,7 @@
     }
 
     # do the magic:
-    my $result = $this->i18n->maketext( $str, @args );
+    my $result = Encode::decode_utf8($this->i18n->maketext( $str, @args ));
 
     # replace accesskeys:
     $result =~

Also, the above would needed to do at every place, where the i18n->maketext is called.

-- JozefMojzis - 22 May 2015

I think we can downgrade this. I have not seen any occurrences of this since we changed the site to utf-8. Since thats the default now, and Actually they are still happening. But this seems to be a perl issue, marking this Normal, so its not a release blocker.

-- GeorgeClark - 12 Jun 2015

Commenting is disabled while not logged in