Item13455: PlainFileStore fails to clean up ,pfv directories for utf-8 topics

The Fn_SEARCH tests are extremely noisy on my system. I've tried two different perl versions, 5.14.2 (ubuntu) and 5.20.2 (perlbrew)

I've added debug messages into the _rmtree function in PlainFileStore. Also lists the directory on failure. Seems to show that the following were left behind:

• OkÃTopic,pfv
• SomeOtherÃÃÃTopic,pfv

exporting FOSWIKI_ASSERTS=1 for extra checking; disable by exporting FOSWIKI_ASSERTS=0
Options: -clean
Assert checking on 1
DEVELOPER WARNING: taint mode could not be enabled. Is Taint::Runtime installed?
Starting CWD is /var/www/foswiki/distro/core/test/unit 
Running Fn_SEARCH
        Fn_SEARCH::test_nofinalnewline
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/SomeOtherÃÃÃTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/SomeOtherÃÃÃTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/SomeOtherÃÃÃTopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/.changes
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkÃTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkÃTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkÃTopic.txt
total 16
drwxr-xr-x  4 gac gac   4096 Jun  9 21:05 .
drwxr-xr-x 15 gac users 4096 Jun  9 21:05 ..
drwxr-xr-x  2 gac gac   4096 Jun  9 21:05 OkÃTopic,pfv
drwxr-xr-x  2 gac gac   4096 Jun  9 21:05 SomeOtherÃÃÃTopic,pfv
Unexpected exception while removing web TemporarySEARCHTestWebSEARCH
PlainFile: Failed to delete /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH:  at /var/www/foswiki/distro/core/lib/Foswiki/Store/PlainFile.pm line 1476.
 at /var/www/foswiki/distro/core/test/unit/FoswikiTestCase.pm line 45
        FoswikiTestCase::__ANON__('PlainFile: Failed to delete /var/www/foswiki/distro/core/data...') called at /var/www/foswiki/distro/core/lib/Foswiki/Store/PlainFile.pm line 1476
        Foswiki::Store::PlainFile::_rmtree('/var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH') called at /var/www/foswiki/distro/core/lib/Foswiki/Store/PlainFile.pm line 791
        Foswiki::Store::PlainFile::remove('Foswiki::Store::PlainFile=HASH(0x4289ee8)', 'BaseUserMapping_666', 'Foswiki::Meta=HASH(0x44f9898)', undef) called at /var/www/foswiki/distro/core/lib/Foswiki/Meta.pm line 2618
        Foswiki::Meta::removeFromStore('Foswiki::Meta=HASH(0x44f9898)') called at /var/www/foswiki/distro/core/test/unit/FoswikiTestCase.pm line 821
        FoswikiTestCase::__ANON__() called at /usr/share/perl5/Error.pm line 416
        eval {...} called at /usr/share/perl5/Error.pm line 408
        Error::subs::try('CODE(0x44e0db0)', 'HASH(0x45c61e8)') called at /var/www/foswiki/distro/core/test/unit/FoswikiTestCase.pm line 827
        FoswikiTestCase::removeWebFixture('Fn_SEARCH=HASH(0x3732620)', 'Foswiki=HASH(0x425cea8)', 'TemporarySEARCHTestWebSEARCH') called at /var/www/foswiki/distro/core/test/unit/FoswikiFnTestCase.pm line 129
        FoswikiFnTestCase::tear_down('Fn_SEARCH=HASH(0x3732620)', 'Fn_SEARCH::test_nofinalnewline') called at /var/www/foswiki/distro/core/test/unit/Fn_SEARCH.pm line 194
        Fn_SEARCH::tear_down('Fn_SEARCH=HASH(0x3732620)', 'Fn_SEARCH::test_nofinalnewline') called at /var/www/foswiki/distro/core/lib/Unit/TestRunner.pm line 604
        Unit::TestRunner::runOne('Unit::TestRunner=HASH(0x2606e48)', 'Fn_SEARCH=HASH(0x3732620)', 'Fn_SEARCH', 'test_nofinalnewline') called at /var/www/foswiki/distro/core/lib/Unit/TestRunner.pm line 163
        Unit::TestRunner::start('Unit::TestRunner=HASH(0x2606e48)', 'Fn_SEARCH::test_nofinalnewline') called at ../bin/TestRunner.pl line 143.

Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/.changes
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv/1

Unit test run Summary:
 - Tested with perl v5.14.2 from /usr/bin/perl

All tests passed (1)
1..21

-- GeorgeClark - 10 Jun 2015

The following patch fixes the failures, however I don't think it's right. I think the issue is that _rmtree() is encoding something that is already encoded, so the _decode removes the double encoding. However removing the _encode at the top of _rmtree() breaks a lot of things.

diff --git a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
index bea2b56..1ed9463 100644
--- a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
+++ b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
@@ -1449,6 +1449,8 @@ sub _rmtree {
             $entry = "$root/$1";
             if ( -d $entry ) {
                 _rmtree($entry);
+                _rmtree(_decode($entry)) if -e $entry;
+                die "STILL THERE" if -e $entry;
             }
             elsif ( !unlink($entry) && -e $entry ) {
                 my $mess = 'PlainFile: Failed to delete file '

-- GeorgeClark - 13 Jun 2015

Another fix also appears to work fine. Move the call to _encode up to the callers of _rmtree, so that there is no encoding of recursive calls. I assume the issue is that the directory read returns bytes, so it does not need further encoding.

diff --git a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
index bea2b56..569d103 100644
--- a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
+++ b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
@@ -782,14 +782,14 @@ sub remove {
 
         # Topic or attachment
         _unlink( _latestFile( $meta, $attachment ) );
-        _rmtree( _historyDir( $meta, $attachment ) );
-        _rmtree( _getPub($meta) ) unless ($attachment);    # topic only
+        _rmtree( _encode(_historyDir( $meta, $attachment )) );
+        _rmtree( _encode(_getPub($meta) ) ) unless ($attachment);    # topic only
     }
     else {
 
         # Web
-        _rmtree( _getData($meta) );
-        _rmtree( _getPub($meta) );
+        _rmtree( _encode(_getData($meta)) );
+        _rmtree( _encode(_getPub($meta)) );
     }
 
     return unless ( $Foswiki::Store::STORE_FORMAT_VERSION < 1.2 );
@@ -1438,7 +1438,7 @@ sub _mkPathTo {
 
 # Remove an entire directory tree. $root must be encoded.
 sub _rmtree {
-    my $root = _encode(shift);
+    my $root = shift;
     my $D;
     if ( opendir( $D, $root ) ) {