Item13654: Foswiki::Func::saveFile was changed in Foswiki 2.0 to use utf-8 encoding.

The function Foswiki::Func::saveFile is not protected by any sort of file locking or multiple access serialization. This will lead to corruption of files stored using this function. Additionally, the function forces utf-8 encoding. Neither of these issues are documented in the function. If the function is being used, these issues should be part of the documentation in large warning text, or the restrictions should be removed altogether, i.e. implement file locking and remove the forced encoding.

-- JohnKnutson - 31 Aug 2015

Yep

-- MichaelDaum - 01 Sep 2015

The utf8 IS documented: Text is encoded using utf-8 before saving. but yes - because we already have special Topic manipulation functions, so the low-level Foswiki::Func::saveFile should be RAW to allow plugins to save any file content.

-- JozefMojzis - 01 Sep 2015

I checked some of the plugins that use this and they all use this for temp files and files to be used by 3rd party software. I do not see why the change to UTF8 inside Foswiki should force plugins to save in UTF8.

Yes it is documented. It is documented that someone made a none backwards compatible change to a Func call as part of 2.0. It should have been made as an enhancement keeping raw as default. We cannot know what people use these low level Func calls for and the content may never hit the wiki engine. As it is done now some plugnis will have to change so they bypass the Func calls and write directly to a file and is that what we want people to do?

-- KennethLavrsen - 03 Sep 2015

Marking this urgent. It sounds like it has the potential to corrupt files. Just like Attachments, we should not be encoding/decoding, I don't think. Crawford... any comments?

-- Main.GeorgeClark - 03 Sep 2015 - 14:41

Doing some more testing. If saveFile is intended to be used transparently with binary data, then extensions will need to first convert from unicode to utf8 before callling the function. Moving the encoding up into the caller.

-- GeorgeClark - 03 Sep 2015

Concur that the "utf8" thing is documented. Nevertheless, saveFile is still dangerous from the stand-point that being a Foswiki function implies that it is safe to use in a Foswiki context when it is not. It might be safe only if used in conjunction with something like mktemp, but if there's even the slightest chance of two sessions writing to the same file, corruption is likely. IMO this API should either be protected using flock, or removed entirely with the extensions that use it updated to use Perl's open/close directly instead, leaving any necessary file locking up to the extension.

-- JohnKnutson - 03 Sep 2015

I believe that this task breaks down into two areas:

1. Adding locking to saveFile. That's an enhancement, and should be considered for 2.1.0 (Opened Tasks.Item13678)
2. utf-8 encoding should probably be addressed in this task for 2.0.2 or 2.0.3. This is an API change from Foswiki 1.1.9. The statement "Text is encoded using utf-8 before saving." is new in 2.0.0. There is nothing specific in the API that states $text passed to saveFile can't be an arbitrary binary string. I think we need to remove the utf-8 encoding from the API and default to raw I/O as in Foswiki 1.1.x, and add a flag to the call to indicate any desired encoding.

-- GeorgeClark - 06 Sep 2015

I favour removing the API completely. It was ill-considered from the very beginning, and is something I wanted to do a long time ago. For compatibility, a "compatible" saveFile can be cooked up that saves bytes.

-- CrawfordCurrie - 07 Sep 2015

After release meeting discussion:

• Revert default behaviour to pre Foswiki 2.0. Read/write raw bytes.
• Add flag to API which enables utf-8 encoding layer

-- GeorgeClark - 07 Sep 2015

Commenting is disabled while not logged in