Item9569: Macros are processed by commonTagsHandler within verbatim blocks of included topics

Macros that use commonTagsHandler are rendered within verbatim blocks that are included from another topic.

So this verbatim block (COMMENT macro mangled to prevent messing up this topic):

<verbatim>
%SERVERTIME%
%COmmENT%
</verbatim>

Renders correctly when it appears in the base topic but renders as:

%SERVERTIME%
<form method="post" action="http://trunk.foswiki.org/bin/save/Sandbox/TestTopic11" enctype="multipart/form-data" name="above0" id="above0">
<div class="commentPlugin commentPluginPromptBox">
<table border="0" cellpadding="0" cellspacing="0">
  <tr valign="middle">
    <td>
      <textarea  rows="3" cols="70"\
        name="comment" class="foswikiInputField" wrap="soft"\
        onfocus="if(this.value=='')this.value=''"\
        onblur="if(this.value=='')this.value=''"></textarea>
    </td>
    <td>&nbsp;<input  type="submit"\
      value="Add comment" class="foswikiButton" />
    </td>
  </tr>
</table>
</div><!--/commentPlugin-->
<input type="hidden" name="comment_action" value="save"  />
<input type="hidden" name="comment_type" value="above"  />
<input type="hidden" name="comment_index" value="0"  /></form>

when it appears in an included topic.

See Sandbox.TestTopic10 & Sandbox.TestTopic11 for a live example.

-- BryanThale - 27 Aug 2010

This appears to be a bug in the CommentPlugin. If you've found any other extensions that exhibit this behavior, they need tasks opened.

It is the responsibility of any extension using commonTagsHandler to remove verbatim blocks if so desired. from the EmptyPlugin docs: Note that when this handler is called, <verbatim> blocks are still present in the text.

Note that the comment plugin is badly broken on this example in trunk. My comment was added to the verbatim block. Marking urgent.

-- GeorgeClark - 05 Sep 2010

It is also like this in 1.0.9 and probably back to Cairo.

I am downgrading this to Normal. If we could live with it so long we can also live with it for 1.1.0. It will be solved when Crawford check in his changes to CommentPlugin where he defined Comment as a real tag instead of commonTagsHandler.

I do not take in his update for 1.1.0 because it took me 30 seconds to find the first severe bug for that and I am sure such a major rewrite of CommentPlugin hides at least another 3-5 much more severe bugs than this one.

The update to CommentPlugin can be released as a plugin update between 1.1.0 and 1.2.0 without having to release Foswiki.

-- KennethLavrsen - 05 Sep 2010

I just looked at EmptyPlugin.pm in the trunk and it still says:

285	*NOTE:* when this handler is called, <verbatim> blocks have been
286	removed from the text (though all other blocks such as <pre> and
287	<noautolink> are still present).

I think you were looking at the note for beforeCommonTagsHandler (emphasis is mine):

322	This handler is called before Foswiki does any expansion of its own
323	internal variables. It is designed for use by cache plugins. Note that
324	when this handler is called, <verbatim> blocks are still present
325	in the text.

Other plugins exhibit the same behavior, I just used CommentPlugin in my example because it was installed on foswiki.org and easy to demonstrate the behavior.

If you look at the expandMacros code in Foswiki.pm it looks like the intention is to have the verbatim blocks removed during processing of commonTagsHanlders

3193	    $this->innerExpandMacros( \$text, $topicObject );
3194	
3195	    $text = takeOutBlocks( $text, 'verbatim', $verbatim );
3196	
3197	    # Plugin Hook
3198	    $this->{plugins}
3199	      ->dispatch( 'commonTagsHandler', $text, $topicContext, $webContext, 0,
3200	        $topicObject );

...

3228	    putBackBlocks( \$text, $verbatim, 'verbatim' );
3229	
3230	    # Foswiki Plugin Hook (for cache Plugins only)
3231	    $this->{plugins}
3232	      ->dispatch( 'afterCommonTagsHandler', $text, $topicContext, $webContext,
3233	        $topicObject );
3234	
3235	    return $text;
3236	}

The problem is in the INCLUDE logic in Macros/INCLUDE.pm it doesn't remove and restore verbatim blocks like expandMacros does.

291	            $this->innerExpandMacros( \$text, $includedTopicObject );
292	
293	            # 4th parameter tells plugin that its called for an included file
294	            $this->{plugins}
295	              ->dispatch( 'commonTagsHandler', $text, $includedTopic,
296	                $includedWeb, 1, $includedTopicObject );
297	
298	            # We have to expand tags again, because a plugin may have inserted
299	            # additional tags.
300	            $this->innerExpandMacros( \$text, $includedTopicObject );

I don't have a real life example at hand, but I'd be willing to bet that if a commonTagsHandler produced a verbatim block, the non-trivial macros contained in the block would also be expanded inappropriately.

-- BryanThale - 05 Sep 2010

The latest CommentPlugin doesn't use commonTagsHandler, so this problem should have gone.

Rejected for inclusion in 1.1, so has to wait for 1.1.1

-- CrawfordCurrie - 07 Sep 2010

Yes, but the problem isn't limited to CommentPlugin. Comment Plugin was just a convenient example to illustrate the problem.

The problem is that the INCLUDE logic doesn't remove verbatim blocks like the normal expansion loop does.

-- BryanThale - 07 Sep 2010

Bryan, the commonTagsHandler is documented as not removing verbatim blocks, included or inline. Is the problem demonstrable with plugins implementing a registered Tag?

-- GeorgeClark - 08 Sep 2010

Um... somebody had better tell EmptyPlugin then because it clearly says that verbatim blocks have been removed when commonTagsHandler is called.

264	=begin TML
265	
266	---++ commonTagsHandler($text, $topic, $web, $included, $meta )
267	   * =$text= - text to be processed
268	   * =$topic= - the name of the topic in the current CGI query
269	   * =$web= - the name of the web in the current CGI query
270	   * =$included= - Boolean flag indicating whether the handler is
271	     invoked on an included topic
272	   * =$meta= - meta-data object for the topic MAY BE =undef=
273	This handler is called by the code that expands %MACROS% syntax in
274	the topic body and in form fields. It may be called many times while
275	a topic is being rendered.
276	
277	Only plugins that have to parse the entire topic content should implement
278	this function. For expanding macros with trivial syntax it is *far* more
279	efficient to use =Foswiki::Func::registerTagHandler= (see =initPlugin=).
280	
281	Internal Foswiki macros, (and any macros declared using
282	=Foswiki::Func::registerTagHandler=) are expanded _before_, and then again
283	_after_, this function is called to ensure all %MACROS% are expanded.
284	
285	*NOTE:* when this handler is called, <verbatim> blocks have been
286	removed from the text (though all other blocks such as <pre> and
287	<noautolink> are still present).
288	
289	*NOTE:* meta-data is _not_ embedded in the text passed to this
290	handler. Use the =$meta= object.
291	
292	*NOTE:* Read the developer supplement at
293	Foswiki:Development.AddToZoneFromPluginHandlers if you are calling
294	=addToZone()= from this handler
295	
296	*Since:* $Foswiki::Plugins::VERSION 2.0
297	
298	=cut
299	
300	#sub commonTagsHandler {
301	#    my ( $text, $topic, $web, $included, $meta ) = @_;
302	#
303	#    # If you don't want to be called from nested includes...
304	#    #   if( $included ) {
305	#    #         # bail out, handler called from an %INCLUDE{}%
306	#    #         return;
307	#    #   }
308	#
309	#    # You can work on $text in place by using the special perl
310	#    # variable $_[0]. These allow you to operate on $text
311	#    # as if it was passed by reference; for example:
312	#    # $_[0] =~ s/SpecialString/my alternative/ge;
313	#}

Registered tag handlers are fine because _processTags cum _processMacros removes verbatim blocks before entering its parsing loop.

-- BryanThale - 08 Sep 2010

Um... this issue hasn't actually been fixed. The rewrite of CommentPlugin has side-stepped the issue for CommentPlugin but the issue still remains for all other macros that continue to be rendered using the commonTagsHandler

The problem is in the implementation of macros/INCLUDE.pm, it needs to remove and restore verbatim blocks just like _processMacros does.

-- BryanThale - 26 Oct 2010

Re-opened

and changed to core bug

-- KennethLavrsen - 26 Oct 2010

Defer to 1.1.5

-- GeorgeClark - 13 Dec 2011

Re-opening to fix broken unit test PluginHandlerTests::test_commonTagsHandlersINCLUDE

-- OlivierRaginel - 11 Mar 2012

Re-opening to fix broken unit test PluginHandlerTests::test_commonTagsHandlersINCLUDE

-- CrawfordCurrie - 12 Mar 2012

Commenting is disabled while not logged in