Item9867: Setting a variable from a regex search is broken in 1.1.0

I am trying to set a variable in Main.!SitePreferences which contains the result of a regex search.. I am trying to create a drop down of options from a typical bullet list such as this:

• dubtest1
• dubtest2
• dubtest3
• dubtest4
• dubtest5

I am trying to reuse this search in a number of wiki apps which was working correctly on 1.0.9.. Here is the search and the setting of the variable:

• Set DROPDOWN = dubtest1 dubtest2 dubtest3 dubtest4 dubtest5

Where the search is:

%<nop>SEARCH{ "   \*[ ]*dubtest.*?" web=%WEB% topic="%TOPIC%" type="regex" separator=" " multiple="on" casesensitive="on" nonoise="on"  format="<option>$pattern(.*   \*\s*([^\n]+).*)</option>" }%

where the select shows:

dubtest1 dubtest2 dubtest3 dubtest4 dubtest5

You can see that the search is working correctly with 1.0.9 (which is the current version of foswiki.org but the set variable DROPDOWN is not outputting the value into the option field although it is returning the correct number of options on 1.1.0+ as can be seen here: http://trunk.foswiki.org/Sandbox/TestTopic44590

-- PadraigLennon - 22 Oct 2010

I can confirm this bug.

You can assign a value to a macro with a SEARCH. But the minute the SEARCH has a format that contains $pattern the result is an empty macro.

This is a very strange bug. And a very serious one. It will breaks many wiki applications and must be fixed for 1.1.1

I have changed the search above so it works also with standard grep search and so the example is self contained. You cannot use \s. There was also a bug in the regex.

To reproduce the bug locally simply copy this entire topic to your local installation.

-- KennethLavrsen - 22 Oct 2010

I have tested a lot

And it is not in general that $pattern does not work this way. It is more strange. It is the [^\n] in the $pattern that causes the touble.

Example

• Set DROPDOWN2 = dubtest1 dubtest2 dubtest3 dubtest4 dubtest5

or

   * #Set DROPDOWN2 = %SEARCH{ "   \*[ ]*dubtest.*?" web="%WEB%" topic="%TOPIC%" type="regex" separator=" " multiple="on" casesensitive="on" nonoise="on"  format="<option>$pattern(.*   \*\s*([a-z1-9]+).*)</option>" }%

gives

dubtest1 dubtest2 dubtest3 dubtest4 dubtest5

So the observation until now is that it is the \n in a character class that makes the code fail. But only when the SEARCH is a macro assignment. And only inside $pattern. That we need to find the reason for.

-- KennethLavrsen - 22 Oct 2010

I am getting near. It is a taint issue.

-- KennethLavrsen - 22 Oct 2010

More information.

The observation I have is that when _extractPattern is called from a SEARCH all the \ are escaped as double \ (I cannot write it here as the renderer also escapes new lines)

But when the _extractPattern is called from a SEARCH from the result of showing a Macro the

\n

inside the $pattern is not escaped with \\. Something removes the \. And only in 1.1

The result is that Foswiki::validatePattern sees the pattern as dangerrous and returns undefined (which you can also see in the server error log).

So the next step for me is to find out why the

\\n

becomes

\n

in this case.

This little test

• Set TESTMACRO = [\n] [\\n]

[ ] [\n]

.. shows that both in 1.0 and 1.1 we do replace a backslash n in the renderer when we render the macro TESTMACRO. I have to double escape it to show it.

But the difference between 1.0 and 1.1 is that when this is inside a search, the 1.0 does the SEARCH before altering anything. 1.1 molest the content. It is 1.1 that does it wrong. There is a bug here. Someone has changed the order of the rendering.

-- KennethLavrsen - 23 Oct 2010

The removal of the backslash is not happening in the Search.pm code at all.

It is already molested when the SEARCH is being called.

-- KennethLavrsen - 23 Oct 2010

distro:cfe42b42bdd7 was the checkin that introduced this bug.

-- KennethLavrsen - 23 Oct 2010

I have found the reason. The untaint function cannot handle data with newlines.

It takes one little /s on the regex to fix this.

Potentially I still find it worrying why one of the \ is stripped off but that is a generic thing I need to search for.

I am checking in the fix for the untaint function now and it fixes this issue.

-- KennethLavrsen - 23 Oct 2010

I will close this bug and open a new bug that addresses the \\n problem, so I can get the specific issue in release note of 1.1.1.

-- KennethLavrsen - 23 Oct 2010

I must admit i'm highly impressed with your bug-fixing capabilities Kenneth. Thank you for the quick resolution on this...

-- PadraigLennon - 24 Oct 2010

For the record

I opened Item9871 to analyse the newline thing and after some analysis concluded that this is exactly how it has worked since 2004 and maybe earlier and that the way it works makes sense once you understand how it works.

And as can be seen it all works fine with the untaint function capable of multiline.

If someone wonders why it worked in 1.0.X then it was because there was a bug in the untaint sub so it returned the input string instead of returning undef when it could not check the input string.

Crawford fixed this so it returns the correct undef. But that exposed the old bug that it could not handle multiline strings.

I believe it is all fixed and working now.

-- KennethLavrsen - 24 Oct 2010

Commenting is disabled while not logged in