Item2504: Search for windows should be setup for Pureperl instead of Forking

Document that Pureperl SEARCH is needed for Windows

was METASEARCH and SEARCH inconsistent between PurePerl and Forking

In addition neither one works the same way as it used to. Whatever I do either one part of our site or another ends up broken.

Take this simple METASEARCH

<table>
%METASEARCH{type="parent" web="%WEB%" topic="WebHome" format="<tr><td>[[$web.$topic][$topic]]</td></tr>"}% 
</table>

This successfully shows the children of WebHome as delivered with Foswiki 1.0.8 (stricly 1.0.6 upgraded with 1.0.7 and 1.0.8 upgrades), but not my own topics which have this as their parent.

Using degug=raw to show you one topic (of about 4) that does not get found (I had to remove % in front of META as it gets removed on save otherwise):

META:TOPICINFO{author="JulianLevens" date="1260800812" format="1.1" version="1.1"}%
META:TOPICPARENT{name="WebHome"}%
---+!! !PCI

Whereas this provided topic (WebNotify) with foswiki:

META:TOPICINFO{author="ProjectContributor" date="1231502400" format="1.1" version="1"}%
META:TOPICPARENT{name="WebHome"}%

Is found as part of that search.

However, if I switch to PurePerl, then both items are found.

But

This search:

%SEARCH{ "*." topic="FileSend*Converter" scope="text" type="regex" nosearch="on" nonoise="on" nototal="off" format="$n()---+++$topic%BR%$percntINCLUDE{\"$topic\" section=\"summary\"}$percnt"}%

Works fine with the Forking algorithm, but pure perl gives me this output on the page:

Could not perform search. Error was: Quantifier follows nothing in regex; marked by <-- HERE in m/* <-- HERE ./ at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Store/SearchAlgorithms/PurePerl.pm line 41, line 1. at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Store/SearchAlgorithms/PurePerl.pm line 41 Foswiki::Store::SearchAlgorithms::PurePerl::__ANON__('META:TOPICINFO{author="JulianLevens" date="1249485343" forma...') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Store/SearchAlgorithms/PurePerl.pm line 47 Foswiki::Store::SearchAlgorithms::PurePerl::search('*.', 'ARRAY(0x1f43e0c)', 'HASH(0x1fa2494)', 'C:/PROGRA~1/Foswiki/Foswiki_1_0_8_pa/data/Main/', undef, 'Main') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Store/RcsFile.pm line 332 Foswiki::Store::RcsFile::searchInWebContent('Foswiki::Store::RcsLite=HASH(0x1ff31ac)', '*.', 'ARRAY(0x1f43e0c)', 'HASH(0x1fa2494)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Store.pm line 2029 Foswiki::Store::searchInWebContent('Foswiki::Store=HASH(0xf088e4)', '*.', 'Main', 'ARRAY(0x1f43e0c)', 'HASH(0x1fa2494)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Search.pm line 260 Foswiki::Search::_searchTopics('Foswiki::Search=HASH(0x1780f14)', 'Main', 'text', 'regex', 'HASH(0x1780ef4)', 'ARRAY(0x1f6050c)', 'FileSendBinConverter', 'FileSendCountsConverter', 'FileSendEZTSVConverter', ...) called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Search.pm line 680 Foswiki::Search::searchWeb('Foswiki::Search=HASH(0x1780f14)', 'inline', 1, 'topic', 'FileSend*Converter', 'search', '*.', 'basetopic', 'FileSend', ...) called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki.pm line 3836 Foswiki::__ANON__() called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/CPAN/lib//Error.pm line 379 eval {...} called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/CPAN/lib//Error.pm line 371 Error::subs::try('CODE(0x17800d4)', 'HASH(0x1780e54)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki.pm line 3845 Foswiki::SEARCH('Foswiki=HASH(0x74fe2c)', 'Foswiki::Attrs=HASH(0x1780d14)', 'FileSend', 'Main', 'Foswiki::Meta=HASH(0x1d7d0a4)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki.pm line 2872 Foswiki::_expandTagOnTopicRendering('Foswiki=HASH(0x74fe2c)', 'SEARCH', ' "*." topic="FileSend*Converter" scope="text" type="regex" no...', 'FileSend', 'Main', 'Foswiki::Meta=HASH(0x1d7d0a4)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki.pm line 2777 Foswiki::_processTags('Foswiki=HASH(0x74fe2c)', '---+!! File Send\x{a}%TOC%\x{a}%STARTSECTION{type="include"}%\x{a}---++ I...', 'CODE(0xe13954)', 16, 'FileSend', 'Main', 'Foswiki::Meta=HASH(0x1d7d0a4)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki.pm line 2694 Foswiki::expandAllTags('Foswiki=HASH(0x74fe2c)', 'SCALAR(0xe14144)', 'FileSend', 'Main', 'Foswiki::Meta=HASH(0x1d7d0a4)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki.pm line 3022 Foswiki::handleCommonTags('Foswiki=HASH(0x74fe2c)', '---+!! File Send\x{a}%TOC%\x{a}%STARTSECTION{type="include"}%\x{a}---++ I...', 'Main', 'FileSend', 'Foswiki::Meta=HASH(0x1d7d0a4)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/UI/View.pm line 388 Foswiki::UI::View::_prepare('---+!! File Send\x{a}%TOC%\x{a}%STARTSECTION{type="include"}%\x{a}---++ I...', 'Foswiki=HASH(0x74fe2c)', 'Main', 'FileSend', 'Foswiki::Meta=HASH(0x1d7d0a4)', 0) called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/UI/View.pm line 368 Foswiki::UI::View::view('Foswiki=HASH(0x74fe2c)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/UI.pm line 304 Foswiki::UI::__ANON__() called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/CPAN/lib//Error.pm line 379 eval {...} called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/CPAN/lib//Error.pm line 371 Error::subs::try('CODE(0x936c54)', 'HASH(0x1d845dc)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/UI.pm line 391 Foswiki::UI::_execute('Foswiki::Request=HASH(0xec2874)', 'CODE(0xec755c)', 'view', 1) called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/UI.pm line 275 Foswiki::UI::handleRequest('Foswiki::Request=HASH(0xec2874)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/lib/Foswiki/Engine/CGI.pm line 29 Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0xd75c04)') called at C:/Program Files/Foswiki/Foswiki_1_0_8_pa/bin/view line 45 

---

• I cannot reproduce your METASEARCH problem on a clean pseudo-install of the current release branch (what will become 1.0.9), nor can I see anything wrong using my production 1.0.8 wiki.
• I do see the error when running a modified version of your SEARCH, but that's because "*." is not a valid regexp (but ".*" is, and this works fine). So the concern is: why doesn't the Forking implementation give an error for this malformed regexp?

Can you tell us more about the environment you're running: What Operating System/version, what version of perl?

-- PaulHarvey - 15 Dec 2009

Thank's for the SEARCH correction. I assumed everything was OK, I've upgraded from TWiki recently and everything was working fine there.

As for the Forking search not finding my WebHome parents. I'll dig further, I have quite a few plugins installed which could be interfering. I'm running on Windows (Mswin32 5.1 (MSWin32-x86-multi-thread)) and perl 5.10 (strawberry) and Apache/2.2.13 (Win32). Standard CGI engine at the moment.

-- JulianLevens - 15 Dec 2009

The malformed regex doesn't give an error with Forking because it uses 'grep', and grep is tolerant to a '*' at the start of the regex, where perl is not. Try grep '*.' *.txt on the command-line to see what I mean.

-- CrawfordCurrie - 16 Dec 2009

After some testing I found this in the Apache error logs. Crucially, the Sandbox::sysCommand error is not created when I switch to PurePerl searching.

This would suggest that the forking algorithm is not entirely successful calling grep under Windows. Is this a clue?

[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] [Tue Dec 22 04:53:12 2009] CGI.pm: Use of uninitialized value $_ in -d at C:/strawberry/perl/lib/CGI.pm line 4083., referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] [Tue Dec 22 04:53:12 2009] CGI.pm: Use of uninitialized value $_ in -d at C:/strawberry/perl/lib/CGI.pm line 4083., referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8885), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8782), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8897), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8873), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8765), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8899), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8796), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8911), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8887), referer: http://tw4-wiki/
[Tue Dec 22 13:53:14 2009] [error] [client 10.132.96.221] WARNING: Sandbox::sysCommand commandline probably too long (8779), referer: http://tw4-wiki/

This certainly gave me an idea. I created a topic called WikiChild1 and that appears even under the forking algorithm. This possibly explains why the apparently standard topics Web... and Wiki... appear whereas mine earlier in the alphabet do not. It suggests the possibility to me that a list of topics to search is passed to grep and when this list is too big the early entries are ignored. Remember I'm running on Windows (more details above) which may be relevant.

-- JulianLevens - 22 Dec 2009

Hi Julian, thanks for the good information!

If you leave a bug WaitingFor yourself, it won't be noticed by busy developers who rely on accurate CurrentState/WaitingFor fields.

Changed back to "New", we need a developer to recreate the problem on windows.

-- PaulHarvey - 26 Dec 2009

If the problem is that grep cannot take all the topics under Windows because of command line max, then there is no solution other than PurePerl which is not a bad solution I would say.

Then the actions would be

• Change the configure setting to non expert
• Document in configure that PurePerl should be used for native Windows (not cygwin) - if I understand the conclusion right
• Put a note in InstallationGuide about this.

-- KennethLavrsen - 05 Jan 2010

Crawford - is my assumption correct that there is no real solution to this using grep and that pure perl is the search we should recommend as standard for native Windows installation?

If so I will take the steps I suggest above. Just let me know if I am right

-- KennethLavrsen - 07 Jan 2010

Had a chat with Sven and my assumption is wrong.

Here is the IRC log

[02:11] <Lavr> Sven what is your view on http://foswiki.org/Tasks/Item2504? Should Windows always search with PurePerl?
[02:12] <SvenDowideit> no :)
[02:12] <SvenDowideit> my opinion is that we should fix our bugs
[02:13] <SvenDowideit> grep used to work quite well on windows, but somewhere it stopped being as reliable
[02:14] <SvenDowideit> its a bit surprising because i recal running the unit tests on windows last time i had time
[02:14] <Lavr> The submitters argument is that it fails because Windows has a limit on max number of characters on command line. So maybe it will always fail if we pass too long a string to grep?
[02:14] <SvenDowideit> that has always been the case
[02:14] <SvenDowideit> on unix too
[02:14] <SvenDowideit> thats why there is code there attempting to deal with it
[02:15] <Lavr> it seems this is the reporters problem. Try and read his follow up carefully.
[02:15] <SvenDowideit> but the attempt is very simplistic -
[02:15] <SvenDowideit> if i had time, i would have already
[02:15] <SvenDowideit> at this point i'm already stealing time to try the rc
[02:15] <Lavr> That is OK. I just wanted to hear your view and I got it. Thanks
[02:16] * toffe82 has joined #foswiki
[02:16] <SvenDowideit> i did put a comment in the sandbox code wrt how we should recode it iirc
[02:16] <SvenDowideit> atm it chooses a number of topics to add to the command line each time
[02:17] <SvenDowideit> what it should to is calculate based on the length of the data path and the topic names +1space
[02:17] <SvenDowideit> and then compare to the known length of the command buffer
[02:17] <Lavr> Ah so it can fail if people on the site uses very long topic names
[02:18] <SvenDowideit> (also slightly doccoed in the code i think)
[02:18] <SvenDowideit> worse
[02:18] <SvenDowideit> it will fail more the longer the path to the data dir
[02:18] <Lavr> blast
[02:18] <SvenDowideit> ie - c:\ProgramFiles\foswiki\foswiki\data
[02:19] <SvenDowideit> but the maths would not be that hard to impl
[02:19] <Lavr> I'll copy this IRC trail to the report.

-- KennethLavrsen - 08 Jan 2010

There is certainly an opportunity for some work on this code. A complementary approach on Windows would be to use the 8-character file/directory names e.g. Program Files -> Progra~1, though I'm not sure what happens on UTF-8 filesystems (I think NTFS can do UTF-8). It all depends if a Windows dev prepared to invest time on this can be found.

-- CrawfordCurrie - 08 Jan 2010

I have been digging in the history. And I have a point. In all the documentation we have written in Support web in the supplemental documents, we ALWAYS specify pureperl search for Windows and I recall having heard these issues before.

So I will so what I proposed above. I will document the PurePerl search as the recommended for Windows and then someone else can save the whales and make external grep program work.

-- KennethLavrsen - 09 Jan 2010

Waiting for release to get this in 1.0.9 release note.

So to Julian - PurePerl seems to be what you should do. Unless you want to experiment with short path to Foswiki. I have not tested this. You may want to and provide feedback.

-- KennethLavrsen - 09 Jan 2010

PurePerl will be OK with documentation updated for other Windows users. I'll try to look at fixing the forking option, but no idea how long that will take, my workload is way too high right now. As I intend to move to Fast CGI eventually, will the NativeSearchContrib be another alternative or does that also depend on grep and have similar issues?

-- JulianLevens - 11 Jan 2010

I use NativeSearchContrib with FastCGI and ModPerl and it works well as far as i can tell, but I never tested it on windows.

-- GilmarSantosJr - 11 Jan 2010

Rleated? Support.Question380

-- PaulHarvey - 14 Jan 2010

I am going to put this back in Waiting For Release and later close it because we have done actions to this that I want in the release notes.

Further work should be done on new bug item which we can tie to a later release.

I will create this bug report for Julian and put it waiting for him.

-- KennethLavrsen - 16 Jan 2010