Item13566: Speed-aware reading encoded files

Benchmarked different file-reading and decoding methods, and got some surprising results.

So asked an question in the Stackoverflow. http://stackoverflow.com/questions/31598938/benchmarking-utf8-file-read-explanation-of-the-differences

The answer pointed me to the module CPAN:Unicode::UTF8, it allows reading files 5 times faster as using Encode. The question contains some benchmarking code too.

Therefore, (and learned from the CGI-problem lesson too), IMHO Foswiki shouldn't use any CPAN modules directly, but thru its own interface modules, like Foswiki::Encode - instead of the plain perl's Encode:. Such approach could allow easily modify, replace the backend implementation, e.g. in this case the file-reading routine could be:

if( the_store_is_utf8 ) {
    read RAW file and use Unicode::UTF8 for the speed gain
} else {
    read the file using Encode
}

-- JozefMojzis - 27 Jul 2015

Setting this to confirmed. But does need more investigation.

-- GeorgeClark - 25 Sep 2016

Jozef, which perl version have you been benchmarking. Note that anything befor 5.22 is not worth it. I just looked at Unicode::UTF8's activity on github. ... which logs most recent commit back in 2013. Hm.

-- MichaelDaum - 26 Sep 2016

Commenting is disabled while not logged in