SiteMap failure: utf8

I've been tracking down and fixing utf8 "does not map to Unicode" characters in my .txt files. I think I've found them all.

However, my System.SiteMap is giving me an error

Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.

utf8 "\xE9" does not map to Unicode

I tracked down the problem down to one line in the SiteMap SEARCH code.

The code contains this line:

  [[%web%.%HOMETOPIC%][%web%]]

If I change that so the link is not created, i.e.

  %web%.%HOMETOPIC%

the SiteMap page renders without errors.

I don't see any obvious unexpected characters in any of my web names. But something in the link creation code is dying.

if it helps, the Apache error log contains

grep 'Fri Sep 18 11:47:13 2015. .error. .client 192.168.1.207. ' /var/log/apache2/error_log*
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] utf8 "\\xAE" does not map to Unicode at /System/Library/Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162.
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207]  at /System/Library/Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tEncode::decode('utf-8', '%META:TOPICINFO{author="VickiBrown" comment="save topic" date...', 1) called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Store.pm line 259
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Store::decode('%META:TOPICINFO{author="VickiBrown" comment="save topic" date...') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Store/Rcs/Store.pm line 116
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Store::Rcs::Store::readTopic('Foswiki::Store::RcsWrap=HASH(0x100cee660)', 'Foswiki::Meta=HASH(0x102a1da20)', undef) called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Meta.pm line 1156
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Meta::loadVersion('Foswiki::Meta=HASH(0x102a1da20)') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Store/Rcs/Store.pm line 360
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Store::Rcs::Store::getVersionInfo('Foswiki::Store::RcsWrap=HASH(0x100cee660)', 'Foswiki::Meta=HASH(0x102bacd98)') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Meta.pm line 1603
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Meta::getRevisionInfo('Foswiki::Meta=HASH(0x102bacd98)') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Render/ToolTip.pm line 49
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Render::ToolTip::render('Foswiki=HASH(0x10085ba60)', 'BAAPT', 'WebHome', '$username - $date - r$rev: $summary') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Render.pm line 1369
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Render::_renderExistingWikiWord('Foswiki::Render=HASH(0x100f859a0)', 'BAAPT', 'WebHome', 'BAAPT', '', '') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Render.pm line 1293
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Render::_renderWikiWord('Foswiki::Render=HASH(0x100f859a0)', 'BAAPT', 'WebHome', 'BAAPT', '', 1, undef, '') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Render.pm line 192
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Render::internalLink('Foswiki::Render=HASH(0x100f859a0)', 'BAAPT', 'WebHome', 'BAAPT', '', 1, undef, 1, '', ...) called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Render.pm line 1565
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Render::_handleSquareBracketedLink('Foswiki::Render=HASH(0x100f859a0)', 'Foswiki::Meta=HASH(0x100f7aa98)', undef, undef) called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Render.pm line 513
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Render::getRenderedVersion('Foswiki::Render=HASH(0x100f859a0)', '---+ <nop>CFCL Sitemap
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] This table is updated automatically ba...', 'Foswiki::Meta=HASH(0x100f7aa98)') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Meta.pm line 3367
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Meta::renderTML('Foswiki::Meta=HASH(0x100f7aa98)', '---+ <nop>CFCL Sitemap
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] This table is updated automatically ba...') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/UI/View.pm line 480
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::UI::View::_prepare('---+ <nop>%WIKITOOLNAME% Sitemap
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] This table is updated automa...', 'Foswiki::Meta=HASH(0x100f7aa98)', 0) called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/UI/View.pm line 459
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::UI::View::view('Foswiki=HASH(0x10085ba60)') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/UI.pm line 374
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::UI::__ANON__() called at /Library/Perl/5.10.0/Error.pm line 419
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \teval {...} called at /Library/Perl/5.10.0/Error.pm line 411
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tError::subs::try('CODE(0x100805140)', 'HASH(0x10085b460)') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/UI.pm line 500
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::UI::_execute('Foswiki::Request=HASH(0x100ba9118)', 'CODE(0x100823bb8)', 'view', 1) called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/UI.pm line 326
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::UI::handleRequest('Foswiki::Request=HASH(0x100ba9118)') called at /Volumes/Datastore/Local/Protected/WWW/web/Foswiki-2.0.1/lib/Foswiki/Engine/CGI.pm line 99
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] \tFoswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x100a2e738)') called at /WWW/web/stage/foswiki/bin/view line 29.
/var/log/apache2/error_log:[Fri Sep 18 11:47:13 2015] [error] [client 192.168.1.207] File does not exist: /WWW/web/stage/favicon.ico, referer: http://stage.cfcl.com/foswiki/bin/view/System/SiteMap

-- VickiBrown - 18 Sep 2015

From your apache log, it's not E9, it's xAE, which is the registered tradmark symbol ® at least in Windows cp-1252 and iso-8859-1. It might be showing up in a tooltip message for the link depending upon which plugins you have enabled.

-- GeorgeClark - 19 Sep 2015

It turns out that there are several WebHome pages causing the utf8 error, with codes EA, 96, and 9A. (I found the AE).

So, what you're saying is that Foswiki, in building the link, wants to extract info for a totip (http title option) and bumps into a "bad" character...

-- VickiBrown - 19 Sep 2015

Check your setting for LINKTOOLTIPINFO. Default is turned off, but if enabled, it can put various information into a tooltip when a link is generated.

-- GeorgeClark - 19 Sep 2015

It could also be characters in the WEBSUMMARY setting, which is expanded as part of the SiteMap page.

-- GeorgeClark - 19 Sep 2015