Item12444: Wysiwyg error: "Wide character in subroutine entry"

Error when trying to view wiki text for DefaultPreferences. Editing using "View wiki text" doesn't have any problem.

Foswiki version: 1.1.8 Perl version: v5.14.2

==============================LocalSite.cfg========
$Foswiki::cfg{Site}{Locale} = 'en_US.ISO-8859-1';
$Foswiki::cfg{UserInterfaceInternationalisation} = 0;
==============================

==============================Error log=============

[Sat Mar 23 14:04:22 2013] view: [Sat Mar 23 14:04:22 2013] view: AccessControlException: Access to CHANGE Home.AlfahedPM for BaseUserMapping_222 is denied. access not allowed on web
[Sat Mar 23 14:04:22 2013] view:  at /usr/share/perl/5.14/CGI/Carp.pm line 379
[Sat Mar 23 14:04:22 2013] view:        CGI::Carp::realdie('[Sat Mar 23 14:04:22 2013] view: AccessControlException: Acce...') called at /usr/share/perl/5.14/CGI/Carp.pm line 475
[Sat Mar 23 14:04:22 2013] view:        CGI::Carp::die('Foswiki::AccessControlException=HASH(0xa898638)') called at /var/www/foswiki/lib/CPAN/lib/Error.pm line 150
[Sat Mar 23 14:04:22 2013] view:        Error::throw('Foswiki::AccessControlException=HASH(0xa898638)') called at /var/www/foswiki/lib/CPAN/lib/Error.pm line 396
[Sat Mar 23 14:04:22 2013] view:        Error::subs::try('CODE(0x95a4968)', 'HASH(0x9b1d498)') called at /var/www/foswiki/lib/Foswiki/UI.pm line 435
[Sat Mar 23 14:04:22 2013] view:        Foswiki::UI::_execute('Foswiki::Request=HASH(0x9b29c70)', 'CODE(0x9b07b58)', 'view', 1) called at /var/www/foswiki/lib/Foswiki/UI.pm line 274
[Sat Mar 23 14:04:22 2013] view:        Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x9b29c70)') called at /var/www/foswiki/lib/Foswiki/Engine/CGI.pm line 41
[Sat Mar 23 14:04:22 2013] view:        Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x97775a4)') called at /var/www/foswiki/bin/view line 24
Wide character in subroutine entry at /usr/lib/perl/5.14/Encode.pm line 174.
 at /usr/lib/perl/5.14/Encode.pm line 174
        Encode::decode('windows-1252', '---+!! <nop>%WIKITOOLNAME% Site-Level Preferences

This topic...', 520) called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 666
        Foswiki::Plugins::WysiwygPlugin::Handlers::returnRESTResult('Foswiki::Response=HASH(0x8924b94)', 200, '---+!! <nop>%WIKITOOLNAME% Site-Level Preferences

This topic...') called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 771
        Foswiki::Plugins::WysiwygPlugin::Handlers::_restHTML2TML('Foswiki=HASH(0x8924720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x8924b94)') called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin.pm line 86
        Foswiki::Plugins::WysiwygPlugin::_execute('_restHTML2TML', 'Foswiki=HASH(0x8924720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x8924b94)') called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin.pm line 70
        Foswiki::Plugins::WysiwygPlugin::__ANON__('Foswiki=HASH(0x8924720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x8924b94)') called at /var/www/foswiki/lib/Foswiki/Func.pm line 704
        Foswiki::Func::__ANON__('Foswiki=HASH(0x8924720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x8924b94)') called at /var/www/foswiki/lib/Foswiki/UI/Rest.pm line 242         
        Foswiki::UI::Rest::rest('Foswiki=HASH(0x8924720)') called at /var/www/foswiki/lib/Foswiki/UI.pm line 316                                                                                
        Foswiki::UI::__ANON__() called at /var/www/foswiki/lib/CPAN/lib/Error.pm line 379                                                                                                       
        eval {...} called at /var/www/foswiki/lib/CPAN/lib/Error.pm line 371                                                                                                                    
        Error::subs::try('CODE(0x834e968)', 'HASH(0x895cbf0)') called at /var/www/foswiki/lib/Foswiki/UI.pm line 435                                                                            
        Foswiki::UI::_execute('Foswiki::Request=HASH(0x88d3e54)', 'CODE(0x88d3c4c)', 'rest', 1) called at /var/www/foswiki/lib/Foswiki/UI.pm line 274                                           
        Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x88d3e54)') called at /var/www/foswiki/lib/Foswiki/Engine/CGI.pm line 41                                                             
        Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x85219a0)') called                                                                                                                
Wide character in subroutine entry at /usr/lib/perl/5.14/Encode.pm line 174.                                                                                                                    
 at /usr/lib/perl/5.14/Encode.pm line 174                                                                                                                                                       
        Encode::decode('windows-1252', '---+!! <nop>%WIKITOOLNAME% Site-Level Preferences                                                                                                       
                                                                                                                                                                                                
This topic...', 520) called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 666
        Foswiki::Plugins::WysiwygPlugin::Handlers::returnRESTResult('Foswiki::Response=HASH(0x9348b94)', 200, '---+!! <nop>%WIKITOOLNAME% Site-Level Preferences

This topic...') called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 771
        Foswiki::Plugins::WysiwygPlugin::Handlers::_restHTML2TML('Foswiki=HASH(0x9348720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x9348b94)') called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin.pm line 86
        Foswiki::Plugins::WysiwygPlugin::_execute('_restHTML2TML', 'Foswiki=HASH(0x9348720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x9348b94)') called at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin.pm line 70
        Foswiki::Plugins::WysiwygPlugin::__ANON__('Foswiki=HASH(0x9348720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x9348b94)') called at /var/www/foswiki/lib/Foswiki/Func.pm line 704
        Foswiki::Func::__ANON__('Foswiki=HASH(0x9348720)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x9348b94)') called at /var/www/foswiki/lib/Foswiki/UI/Rest.pm line 242
        Foswiki::UI::Rest::rest('Foswiki=HASH(0x9348720)') called at /var/www/foswiki/lib/Foswiki/UI.pm line 316
        Foswiki::UI::__ANON__() called at /var/www/foswiki/lib/CPAN/lib/Error.pm line 379
        eval {...} called at /var/www/foswiki/lib/CPAN/lib/Error.pm line 371
        Error::subs::try('CODE(0x8d72968)', 'HASH(0x9380bf0)') called at /var/www/foswiki/lib/Foswiki/UI.pm line 435
        Foswiki::UI::_execute('Foswiki::Request=HASH(0x92f7e54)', 'CODE(0x92f7c4c)', 'rest', 1) called at /var/www/foswiki/lib/Foswiki/UI.pm line 274
        Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x92f7e54)') called at /var/www/foswiki/lib/Foswiki/Engine/CGI.pm line 41
        Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x8f459a0)') called
[Sat Mar 23 14:26:24 2013] save: Wide character in print at /var/www/foswiki/lib/Foswiki/Store/VC/Handler.pm line 994.
[Sat Mar 23 14:26:39 2013] rest: Parsing of undecoded UTF-8 will give garbage when decoding entities at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin/HTML2TML.pm line 183.
[Sat Mar 23 14:26:57 2013] rest: Parsing of undecoded UTF-8 will give garbage when decoding entities at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin/HTML2TML.pm line 183.
[Sat Mar 23 14:27:37 2013] rest: Parsing of undecoded UTF-8 will give garbage when decoding entities at /var/www/foswiki/lib/Foswiki/Plugins/WysiwygPlugin/HTML2TML.pm line 183.
==============================

-- AlfahedPM - 23 Mar 2013

IRC Logs at http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2013-03-23,Sat&sel=135#l131

Other notes:

• Problem happened on first edit of DefaultPreferences on new 1.1.8 install
• Saving with Wiki Text editor resolved the issue.
• System is in India, System locale: C C.UTF-8 POSIX

Set task to urgent to encourage review.

-- GeorgeClark - 23 Mar 2013

I reviewed this before but didn't comment because I couldn't reproduce it. I just tried harder, with all the different settings described above and in the IRC log, and I still can't reproduce it. Without the users' LocalSite.cfg it's pretty tough to debug this sort of thing.

Later: not mentioned in the report, but this happens when you hit the "wikitext" button in TMCE. The client sends UTF-8 encoded data for conversion via HTML2TML.

Later again: I looked through the code, and there's a load of character set handling appeared in the HTML2TML convertor that should not be there. html2tml should work on site-encoded octets only.

George, can you contact me to discuss? Thanks.

-- CrawfordCurrie - 14 May 2013

It took a lot of hunting and cleaning up, but I finally nailed the source of the problem. Specifically, that topic has an entity, •, that is expanded when the containing span is flattened. Unfortunately the call to HTML::Entity::decode_entities is one that decodes all entities, not just those that are "safe" to target to the site charset. As a result we end up with a UTF8 character in the text.

-- CrawfordCurrie - 15 May 2013

Commenting is disabled while not logged in