Item14437: Macro expansion runs away, reprocessing some macros many times.

Found this while trying to debug the "Friendly" attribute parser.

When a "common tags" macro is encountered, expansion "fails", and it's placed back onto the stack and re-processed with much longer argument strings. A simple example is a %TABLE{sort="off"}%, followed by a table. Below is an abbreviated debug trace. It appeared to go through this parsing process 15 times.

QUEUE:'TABLE{sort="off"}'
      '%'
...
CONSIDER %TABLE{sort="off"}
ATTRS Called with sort="off"
=============================EXPAND TABLE FAILED
...
CONSIDER %TABLE{sort="off"}%
     | *Parameter* | *Description* | *Default* |
     | ="string"= | String to encode | "" (empty string) |
     | =type= | Use a predefined encoding (see below). | Default is 'url'. Parameter =type= not be used if =old= or =new= are given. |
     | =old= | Comma-separated list of tokens to replace. Tokens are normally single characters, but can also be sequences of characters. The standard [[format tokens]] may be used in this list. Each token must be unique - you cannot list the same token twice. | May not be used with =type=; required if =new= is used |
     | =new= | comma-separated list of replacement tokens. The elements in this list match 1:1 with the elements in the =old= list. Again, the standard [[format tokens]] may be used. An empty element in the =new= list will result in the corresponding token in the =old= list being deleted from the string. If the =new= list is shorter than the =old= list it will be extended to the same length using the empty element. Tokens do not have to be unique. <div class="foswikiHelp">
...
CONSIDER %TABLE{sort="off"}%
     | *Parameter* | *Description* | *Default* |
     | ="string"= | String to encode | "" (empty string) |
     | =type= | Use a predefined encoding (see below). | Default is 'url'. Parameter =type= not be used if =old= or =new= are given. |
     | =old= | Comma-separated list of tokens to replace. Tokens are normally single characters, but can also be sequences of characters. The standard [[format tokens]] may be used in this list. Each token must be unique - you cannot list the same token twice. | May not be used with =type=; required if =new= is used |
     | =new= | comma-separated list of replacement tokens. The elements in this list match 1:1 with the elements in the =old= list. Again, the standard [[format tokens]] may be used. An empty element in the =new= list will result in the corresponding token in the =old= list being deleted from the string. If the =new= list is shorter than the =old= list it will be extended to the same length using the empty element. Tokens do not have to be unique. <div class="foswikiHelp"><img src="/pub/System/DocumentGraphics/warning.png" alt="ALERT!" title="ALERT!" width="16" height="16" /> 
...
ATTRS Called with sort="off"}%
     | *Parameter* | *Description* | *Default* |
     | ="string"= | String to encode | "" (empty string) |
     | =type= | Use a predefined encoding (see below). | Default is 'url'. Parameter =type= not be used if =old= or =new= are given. |
     | =old= | Comma-separated list of tokens to replace. Tokens are normally single characters, but can also be sequences of characters. The standard [[format tokens]] may be used in this list. Each token must be unique - you cannot list the same token twice. | May not be used with =type=; required if =new= is used |
     | =new= | comma-separated list of replacement tokens. The elements in this list match 1:1 with the elements in the =old= list. Again, the standard [[format tokens]] may be used. An empty element in the =new= list will result in the corresponding token in the =old= list being deleted from the string. If the =new= list is shorter than the =old= list it will be extended to the same length using the empty element. Tokens do not have to be unique. <div class="foswikiHelp"><img src="/pub/System/DocumentGraphics/warning.png" alt="ALERT!" title="ALERT!" width="16" height="16" /> When using =old= and =new=, be aware that the results of applying earlier tokens are not processed again using later tokens. (see examples below)</div> | May not be used with =type=; required if =old= is used |
If =ENCODE= is called with no optional parameters (e.g. =%<nop>ENCODE{"string"
=============================EXPAND TABLE FAILED

-- GeorgeClark - 15 Jul 2017

I wonder if there is a way to improve this. By allowing extensions to "register" tags that will be expanded in a commonTagsHandler as a "hint" to the macro processing might improve things. Then when a macro is encountered, it could be temporarily protected, and restored at the end of the parsing. On the SpreadSheetPlugin page, the CALC macro was processed 200 times with increasing length arguments.

ATTRS CALLED length:23   $PROPERSPACE(WikiGuest)...) 
ATTRS CALLED length:142   $PROPERSPACE(WikiGuest)}%
ATTRS CALLED length:373   $PROPERSPACE(WikiGuest)}%
ATTRS CALLED length:586   $PROPERSPACE(WikiGuest)}%
...
ATTRS CALLED length:21259   $PROPERSPACE(WikiGuest)}%

I'm not sure how to address it, but I notices another issue. Macros escaped with exclamation mark are removed from consideration by encoding the %. However macros escaped with <nop> are not actually escaped and get considered repeatedly.

-- GeorgeClark - 16 Jul 2017

Changing this to Proposal needed. This would be a pretty big change.

-- GeorgeClark - 12 Dec 2017

Commenting is disabled while not logged in