Feature Proposal: Use a different assignment operator to delay macro evaluation inside parameters

Motivation

In HereDocumentSyntaxForMacros MichaelDaum wrote:

"This proposal tries to address one of the weaknesses of TML. Lots of wiki apps got their main logic inside params to some %MACRO. So moving it out - for instance using HERE or INCLUDE is a good practice. That way multiple levels of escapes like $dollardollarpercnt can be omitted. However, from what I was able to get reading the stuff above, we still have these standard escapes in the HERE documents even though they are outside of the normal %MACRO{param} nesting. I wonder if it is possible to diverge from the normal left-to-right-inside-out in a way to get around this problem."

Michael, has a very good point there.

It has long been an issue that macros are always expanded inside-out-left-right. There are many cases where outside-in-left-right is the required order; not least when evaluating the format of a SEARCH. So, let's say (thought experiment) we used syntax (a different kind of equals sign, say ^=) to change the eval order of a parameter to outside-in? So:

%IF{"condition" then="%X%" else="%Y%}%

will expand %X and %Y before expanding the %IF, which is not always what you want (especially if %X and %Y are compute intensive). But if you used:

%IF{"condition" then^="%X%" else^="%Y%"}%

instead, it would delay expansion of %X and %Y until after the condition was tested.

Interesting idea, as it has the effect of embedding cleanly into existing TML; no surprises if you only use the "old" syntax. Really, really, uber-geeky, though.

Description and Documentation

TBD

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

Implementation

-- Contributors: MichaelDaum, CrawfordCurrie, MichaelTempest, PaulHarvey

Discussion

Outside-in param assignment options:

=	Example	Note
#	param#"a %MACRO%"	As above...
:=	param:="a %MACRO%"	All hail pascal!
@	param@"a %MACRO%"
==	param=="a %MACRO%"	No comment...
$=	param$="a %MACRO%"	$ is a popular char to be confused with in various programming languages
*=	param*="a %MACRO%"
|=	param|="a %MACRO%"
^=	param^="a %MACRO%"
#=	param#="a %MACRO%"
=	param!="a %MACRO%"
@=	param@="a %MACRO%"
.=	param.="a %MACRO%"
+=	param+="a %MACRO%"
~	param~"a %MACRO%"	Conveniently confused with query syntax

Other options:

1. Back-tick ( ` ) instead of double-quotes around the string
   1. Pipe-char ( | )
   2. Single-quote ( ' )
   3. Back-tick, single-quote, Eg. : param=`"Outside-in!"`
   4. Bracket-quotes, Eg. : param=["Outside-in!"]
2. Use different type of braces other than curlies ( { and }). Eg:
   1. %MACRO(param="now everything is outside-in!")%

-- PaulHarvey - 17 Oct 2009

The biggest problem implementing this is how we rewrite the parser to handle it. At the moment the parser works by splitting on % signs and recompiling the strings from the inside out. The advantage of this is that you don't need any information about the context a macro is used in, and users can be quite lackadaisical about syntax. Changing the eval order requires a rather stricter approach. For example, an outside-in construct like this:

%OUTER{param^="%INNER{param="blah"}%"}%

requires you to actually parse the outer and inner macros to identify and match the double quotes. This conflicts with the inside-out approach where the quote matching on the INNER eliminates any chance of confusion in the quotes for OUTER, because INNER is expanded and replaced in-place before OUTER is considered.

Mind you, having said that, the current approach has horrible problems with something like this:

%OUTER{param^="%INNER{param="blah}%"}%

(note the missing close " on the INNER) so I guess we can't actually make it much worse.

-- CrawfordCurrie - 05 Nov 2009

That's why I'd only do it for HERE documents.

-- MichaelDaum - 05 Nov 2009

Whatever syntax we choose, it should also be applicable to the default parameter. This means that we should not look to modifying the assignment operator (i.e. the = ). Instead, we should mark the whole macro for delayed processing of its parameters (e.g. by changing or augmenting the %{ }%, or we should mark specific parameters for delayed processing, which means marking it where it is quoted.

If we choose to delay processing for specific parameters, then the markup for doing so should also be usable for HERE documents, or we should use HERE document quoting as the markup for delayed processing (as Michael suggested). I support the latter, i.e. Michael's suggestion.

However, suppose we need to support both options i.e. there must be a way to specify if macros in HERE documents are evaluated before the outer macro, or after the outer macro. We could add prefix, which means "delay processing of this parameter's value". Here are some examples:

%FOO{!"%THIS% is processed after FOO"}%
%FOO{delay:"%THIS% is processed after FOO"}%

However, what if processing certain markup should be delayed more than once? Foswiki currently allows for this with $dollardollarpercnt. How about %(id:nn) ... (id)% as the markup for doing this? The id is an alphanumeric identifier for the block, intended to make it easier for the parser to match the beginning and end. The nn specifies how many times to delay processing of the macros. I deliberately used %-based markup because that might be easier to add to the existing parser. Some examples:

• %(a:1)%FOO%(a)% would be equivalent to $percntFOO$percnt
• The classic nested search example, using old and new markup where appropriate:

%SEARCH{
"culture" web="System" 
format="   * $topic is referenced by: %(inner:1)%SEARCH{ 
    "$topic" web="System" 
    format="$dollartopic" 
    nosearch="on" nototal="on" separator=", " 
    }%(inner)%" 
nosearch="on" nototal="on" }%

Personally, I would go for delayed macro evaluation using HERE documents, and use $dollar, $percnt or nested HERE documents if I need to delay more than once.

-- MichaelTempest - 23 Jan 2010

I updated the HereDocumentSyntaxForMacros proposal to delay expansion of macros in HERE documents.

-- MichaelTempest - 24 Jan 2010

This is perfectly consistent if you think about the parameter expansion happening at the point of assignment (yes, I know that's not what really happens, but bear with me). Think about param="value" (or param=<<HERE). The "value" (or <<HERE) is identified and read verbatim first; then, when the = operator is processed, it is recursively macro-expanded. If you think that way, it's easy to logically slot in an alternate operator in place of =.

Michael's implemented approach of delaying evaluation of the HERE doc, but maintaining the eval order of "" params, is a nice compromise IMHO. but we need as many people as possible to try it, to see if we can find any problems.

-- CrawfordCurrie - 25 Jan 2010

The here-document syntax needs some more thought, so I do not think we can depend on it to change order of evaluation. Also, here-documents did not eliminate the need for $dollartopic escaping.

The "alternate operator in place of =" approach does not take into account the _DEFAULT parameter. There is no = in %FOO{"bar"}% - the assignment is implicit, so there is no assignment operator to modify. I favour markup that says do something to this parameter before assignment, because then it may also be applied to the _DEFAULT parameter. This might simply mean that we use param=^"value" instead of param^="value", so that the _DEFAULT parameter is given as %FOO{ ^"%BAR%"}%.

There are other issues and situations related to delayed-evaluation that the operator approach does not address.

• What if I want to delay expansion of part of a parameter?
  e.g. format="%FOO%: $percntBAR$percnt"
• What if I want to delay expansion more than once?
  e.g. format="$dollarpercntINCLUDE{\\"$dollartopic\\" section=\\"summary\\"}$dollarpercnt"

We could borrow syntax from perl's POD. POD markup like C<< code >> uses a variable number of <'s and >'s so that it is always possible to quote content that uses the markup symbols, but the number of opening <'s must equal the number of closing >'s.

We could use % in the markup to make it amenable to parsing e.g. with this markup: % n-single-quotes { content } n-single-quotes %
For example: format="%'{ %FOO{ %''{ %BAR% }''% }'% is %BORKEN%"

I don't like the example I have given (it is ugly), but I hope it conveys my intent. This concept works for POD, so perhaps we could use the idea, preferably in a more palatable form

-- MichaelTempest - 21 Feb 2010

I like your thinking here. I'm not sure we're all clear on what "late eval" actually means. To me, the intent of

format="$percntBLAH$percnt"

is to say "I want to pass a valid macro call in the format parameter to this macro, without the macro processor expanding it first". If you look at it that way, $percnt is simply an unwieldy escape. There is another type of escape already used - viz \" - that is much cleaner but unfortunately wasn't thought all the way through, so only applies to ", and there is no support for \\" (i.e. $dollarquot). Anyway, my point is that the simplest way of thinking about the late eval syntax is that it's just a way of saying "reverse the escape translation in this string".

• \\" is supported and is widely used by MichaelDaum. See Item8404 -- MichaelTempest - 24 Feb 2010

Let's say we supported concatenation of strings in parameters, so %BLAH{"blah""%BLAH%""blah""\""}% was the equivalent of %BLAH{"blah%BLAH%blah\""}%. Further, say we picked on a different quote character - say, backtick ` - to express the "reverse escape translation"intent. We can then write %BLAH{"blah"`%BLAH%`"blah"`"`}%. This is "read" as %BLAH{"blah""$percntBLAH$percnt""blah""$quot"}% and thus %BLAH{"blah$percntBLAH$percntblah$quot"}%.

This way we cab have clean translation steps throughput the parsing process, and clear equivalences.

-- CrawfordCurrie - 22 Feb 2010

In my new MultiTopicSavePlugin I solved this problem by inventing a delay parameter.

Primitive but efficient. The delay feature in MultiTopicSavePlugin will cause the plugin to return itself without expansion and with all its quotes as $quot and its leading and trailing % as $percnt. This enables users of MultiTopicSavePlugin to be able to write

%SEARCH{"MultiTopicSaveTestTarget" scope="topic" nonoise="on"
format="| $topic <br /> \
%MULTITOPICSAVEINPUT{"Option" type="text" size="20" web="$web" topic="$topic" value="$value" delay="1" editmode="%URLPARAM{"multiedit" default="off"}%" lockmode="on"}% | \
%MULTITOPICSAVEINPUT{"Description" type="textarea" web="$web" topic="$topic" size="25x6" value="$value" delay="1" editmode="%URLPARAM{"multiedit" default="off"}%" lockmode="on"}% | \
%MULTITOPICSAVEINPUT{"Type" type="radio" size="1" delay="1" web="$web" topic="$topic" value="$value" options="%TYPES%" editmode="%URLPARAM{"multiedit" default="off"}%" lockmode="on"}% | \
%MULTITOPICSAVEINPUT{"Animal" type="checkbox" size="1" delay="1" web="$web" topic="$topic" value="$value" options="%ANIMALS%" editmode="%URLPARAM{"multiedit" default="off"}%" lockmode="on"}% | \
%MULTITOPICSAVEINPUT{"Friend" type="select" size="0" multiple="on" delay="1" web="$web" topic="$topic" value="$value" options="%FRIENDS%" editmode="%URLPARAM{"multiedit" default="off"}%" lockmode="on"}% |"
}%

Note that the delay decrements each time it returns. When it reaches 0 it expands to its real value. In case with the MultiTopicSavePlugin to a HTML input form field with the content of the form field which must happen after the search is complete. Ie. outside -> in. In a two level nested SEARCH you would set delay="2".

This plugin had to deal with both the $dollar/$quot hell and the encoding problem inside TML tables when using formatted searches.

It is a plugin, so it can do things its own way. I am not saying a delay feature is the way. Just giving this as imspiration to how things can also be done.

The most common problem with SEARCH is nested SEARCH. So a delay-alike feature in SEARCH could perhaps remove many $dollardollar horrors.

-- KennethLavrsen - 22 Feb 2010

Interesting idea... I think we could implement this in the engine so that the macro never even sees the parameter

-- MichaelTempest - 23 Feb 2010

Agreed, an interesting idea; presumably delay=2 yields $dollarpercnt rather than $percnt?

• No, in MultiTopicSavePlugin the delay=2 still only encodes $percnt. In addition it decrements its own delay to 1. So it keeps on re-encoding the macro until delay is 0. This is not an efficient way to do it in core. That is a plugin way to do it. I would implement it differently in a macro parser. I just wanted to clarify the principle I used in MultiTopicSavePlugin -- KennethLavrsen - 28 Feb 2010

As you suggest, Michael, the macro parser is the right place for this; and it gives us scope for using alternative syntax rather than a macro parameter (which we ought to avoid). Not sure why the numerical value for delay is required, though; if it's done within the macro parser, embedding should suffice. Assume that an @ symbol indicates a "delayed evaluation":

%SEARCH{... format="%@SEARCH{format="%@SEARCH{format="$topic"}%"}%"}%

expands to

%SEARCH{... format="%@SEARCH{format="$percntSEARCH{format=$quot$dollartopic$quot}$percnt"}%"}%

which expands to

%SEARCH{... format="$percntSEARCH{format=$quot$dollarpercntSEARCH{format=$dollarquot$dollardollartopic$dollarquot}$dollarpercnt$quot}$percnt"}%

which can then be passed to SEARCH.pm

-- CrawfordCurrie - 23 Feb 2010

How about not introducing even more syntax? Instead we could switch the parser from inside-outside to outside-inside for part of a TML expression. It is not the syntax that is wrong. It is the evaluation order that we want to manipulate. Maybe adding even more syntax is not the right approach.

-- MichaelDaum - 24 Feb 2010

Instead we could switch the parser from inside-outside to outside-inside for part of a TML expression - without introducing more syntax? I'd be impressed....

-- CrawfordCurrie - 24 Feb 2010

Well, of course you need some construct to flag evaluation order. Maybe I should have said: let's not add YAFES (yet another funky escape sequence).

So instead of tricking outside-inside evaluation order into an inside-outside parser - as we have done with all this $percnt yada before - we could tell the parser to traverse the syntax tree slightly different as intended originally.

-- MichaelDaum - 24 Feb 2010

You have to "tell" the parser, somehow. That's what the funky %@ATSIGN does.

-- CrawfordCurrie - 24 Feb 2010

%SEARCH{
"culture" web="System" 
format="   * $topic is referenced by: %(inner:1)%SEARCH{ 
    "$topic" web="System" 
    format="$dollartopic" 
    nosearch="on" nototal="on" separator=", " 
    }%(inner)%" 
nosearch="on" nototal="on" }%

as in the HERE doc I think we need to go back to writing what you really should be doing for maintainability and reuse:

%SEARCH{
    "culture" 
    web="System" 
    format="   * $topic is referenced by: $percentINCLUDE{"%BASEWEB%.%BASETOPIC%" section="inner" topic="$topic"}$percent" 
    nosearch="on"
    nototal="on" 
}%

%STARTSECTION{"inner"}%%SEARCH{ 
    "%topic%" 
    web="System" 
    format="%topic%" 
    nosearch="on" 
    nototal="on" 
    separator=", " 
    }%%STOPSECTION{"inner"}%

which then can be simplified using my $include() proposal

%SEARCH{
    "culture" 
    web="System" 
    format="   * $topic is referenced by: $include('%BASEWEB%.%BASETOPIC%' section='inner' topic='$topic')" 
    nosearch="on"
    nototal="on" 
}%

%STARTSECTION{"inner"}%%SEARCH{ 
    "%topic%" 
    web="System" 
    format="%topic%" 
    nosearch="on" 
    nototal="on" 
    separator=", " 
    }%%STOPSECTION{"inner"}%

or when combined with my Here Doc's suggestion

%SEARCH{
    "culture" 
    web="System" 
    format="   * $topic is referenced by: $p(section='inner' topic='$topic')" 
    nosearch="on"
    nototal="on" 
}%

%STARTSECTION{"inner"}%%SEARCH{ 
    "%topic%" 
    web="System" 
    format="%topic%" 
    nosearch="on" 
    nototal="on" 
    separator=", " 
    }%%STOPSECTION{"inner"}%

Crawford has mentioned on IRC that his main worry wrt $include() is that it introduces a new way to call macros - I think this isn't a big issue (though is could be) specifically because this is probably the only macro that needs to be treated this way, as all others can be delayed using this one.

note to MichaelT - the choice of 'P' and 'p' is totally arbitary, and is pandering to the 'don't make me type' crowd - I think we need it to be something, rather than anonymous brackets of braces - and I need to amend the other proposal to show show that there should be a difference between the normal eval order and the delay one.

-- SvenDowideit - 24 Feb 2010

Sorry Sven, I just don't buy into the $include approach, for three reasons:

• You have to fully understand and appreciate why it is being done before it makes sense. It's just not expressive enough; I look at the examples above, and my brain starts to melt.
• I am really uncomfortable with having a whole new way to call macros. $include may not be a problem, but it's the thin end of a wedge.
• (not really important) it has to be nailed into a lot of existing code (it can't be managed entirely from the macro processor).

Just to make sure we have covered all the bases, there is another way to do this that doesn't involve new syntax, but abuses the macro parser somewhat:

%SEARCH{
    "culture" 
    web="System" 
    format="   * $topic is referenced by: %ESCAPE{"%SEARCH{ 
       "$topic" 
       web="System" 
       format="$topic" 
       nosearch="on" 
       nototal="on" 
       separator=", " 
    }%"}%" 
    nosearch="on"
    nototal="on" 
}%

i.e. a "special" macro %ESCAPE escapes all of its parameters (which obviously have to be well formed, with balancing {}).

So far of all the proposals I personally like the backtick best.

%SEARCH{
    "culture" 
    web="System" 
    format="   * $topic is referenced by: "`%SEARCH{ 
       "$topic" 
       web="System" 
       format="$topic" 
       nosearch="on" 
       nototal="on" 
       separator=", " 
    }%"` 
    nosearch="on"
    nototal="on" 
}%

I have a horrible feeling this will all come down to gut feel.

-- CrawfordCurrie - 25 Feb 2010

Sven - I think your example should look like this. Note that I changed %topic% to %outertopic% and I changed the inner format from %topic% to $topic.

%SEARCH{
    "culture" 
    web="System" 
    format="   * $topic is referenced by: $inc(section='inner' outertopic='$topic')" 
    nosearch="on"
    nototal="on" 
}%

%STARTSECTION{"inner"}%%SEARCH{ 
    "%outertopic%" 
    web="System" 
    format="$topic" 
    nosearch="on" 
    nototal="on" 
    separator=", " 
    }%%STOPSECTION{"inner"}%

I do prefer %INC{inner outertopic='$topic'}% to $p(section='inner' outertopic='$topic'), based on what you wrote in HereDocumentSyntaxForMacros.

-- MichaelTempest - 25 Feb 2010

agreed, to both - I've updated your updated eg

-- SvenDowideit - 25 Feb 2010

/me still can't see the difference between

   format="   * $topic is referenced by: $inc(section='inner' outertopic='$topic')" 

and

   format="   * $topic is referenced by: $percentINCLUDE{$quot$topic$quot section=$quotinner$quot}$percnt" 

except that the former is inventing new (and IMHO unnecessary) syntax. If you must invent new syntax, at least do it in a generic way. Again IMHO this is much more consistent and readable:

   format="   * $topic is referenced by: %@INCLUDE{"$topic" section="inner"}%" 

and, because it is general purpose and handled in the macro parser, it works for any macro, anywhere.

-- CrawfordCurrie - 25 Feb 2010

@Crawford: No. %@INCLUDE would only work automatically for macros handled by the macro parser; it would not work for macros handled by commonTagsHandler, such as the infamous %CALC%. In this way, %@INCLUDE would enjoy the same level of support as $include(), because (as far as I understand Sven) $include() would be processed by the macro parser, after tag handler function returns.

@Sven: In HereDocumentSyntaxForMacros, you had %INC{...}%, not $inc(...). I had thought the %INC was intentional - I assume I had that wrong, but note that %INC does not involve new syntax whereas $inc does.

I think the %@MACRO syntax could be generalised: expansion is delayed by the number of @'s. So %@@@MACRO% would be equivalent to $dollardollarpercntMACRO$dollardollarpercnt.

%@MACRO and %INC{paramname}%... (newline)...%END{paramname}% do not have to be mutually exclusive.

-- MichaelTempest - 25 Feb 2010

I like the idea if %@ being a generic solution. And %@@ for delaying two levels out. That would be brilliant if this could work on all macros.

-- KennethLavrsen - 25 Feb 2010

Now I'm well confused. because $include() would be processed by the macro parser. So the result of the expansion of the $include would be embedded in the parameter value for the enclosing macro, at parse time. So %BLAH{"$include(blah)"}% is exactly the same as %BLAH{"%INCLUDE{"%TOPIC%" section="blah"}%"}% isn't it? Or is the difference that there is some protection for the string returned by the $include, that means it doesn't engage in the parsing of the containing macro? Which is what %@INCLUDE would achieve, isn't it?

Note that I'm not trying to start a syntax war here; I'm trying to understand the relative strengths and weaknesses of the proposals. AFAICT none of the proposals so far will work for commonTagsHandler type macros.

-- CrawfordCurrie - 25 Feb 2010

No, %BLAH{"$include(blah)"}% is not the same as %BLAH{"%INCLUDE{"%TOPIC%" section="blah"}%"}%, it will be the same as %BLAH{"$percentINCLUDE{"%TOPIC%" section="blah"}$percent"}%. (that is pretty much the entirety of the spec for it - so long as you allow me to evade exactly how I deal with the quotes inside the braces)

The endpoint of the extractFormat feature I'm working towards in 1.1 and 2.0, is to create a single reuseable API to implement all Macro's header, format, sort, page etc parameters. for 1.1, I've aimed at just extracting the core of that code from the Search code, and to attempt to abstract the input 'list' into an iterator that we can use. At the moment, I'm trying to validate that work by implementing the MongoDB search and query accelerator, which will then also be extended into a fully blown store (optionally) - this has already lead to my identifying and resolving some bugs, and refactoring some code I had been hoping that I could do later.

The point of this story, is that before 2.0 (which realistically is the timeframe for this and the heredocs feature), there will be a simple point in the processing of the format parameter (which all macro's will call) to expand the $inc() sectional delay (right at the end...) - and that approach will work even for commonTagsHandler type macros.

when you think about it tho, if you ignore the dirty few, the $inc() expansion can be shoehorned onto the end of the MACRO switchboard - I wasn't intending to do that, but

One of the reasons I think this $inc() delay is so 'clever' is that ir really is just a syntactic sugar shortcut, which then reveals to the user that the $dollar escaping business is quite totally un-necessary, except in cases where its faster not to create another section. Any time that you think you need to escape something, create yourself a reusable include section, using INCLUDE params to pass values.

-- SvenDowideit - 27 Feb 2010

Indeed, it's syntactic sugar; but the main problem I have with it is it's a whole new family of syntactic sugar. The thin end of this particular wedge was $formfield, which looked like a good idea on the face of it, but is really just a hamstrung $percntFORMFIELD. As soon as people want to do more with $formfield we redirect them to $percntFORMFIELD (or $percentQUERY now, I hope) because we don't want to start developing a whole new syntax based on $formfield for calling macros - which is exactly what I fear $inc is evidence of. Add on top of that the fact that this particular syntactic sugar requires extensive code changes to support it. And on top of that, the sectional includes approach, while it does force people to think in terms of structure, is not particularly user friendly, as you acknowledge.

We have enough trouble with the two macro languages we have for formatting (TMPL and TML macros). Adding a third - even a hamstrung third - strikes me as a really bad idea.

I am far from convinced this new calling protocol is even slightly necessary. If we think in terms of the proposed generic syntax for delayed eval:

%SEARCH{... format="%@INCLUDE{"%TOPIC%" section="fred"}%"}%

(aside: it occurs to me that if we made the _DEFAULT parameter to INCLUDE default to the current topic, and $ the delayed eval char, then the following is getting darn close to $inc, except it builds on the existing macro language.

%SEARCH{... format="%$INCLUDE{section="fred"}%"}%

or (in what is probably a step too far), replace the %$ with $, the { with ( and the }% with )

%SEARCH{... format="$INCLUDE(section="fred")"}%

crucially it is still 100% consistent with the existing macro language, and outside of the parser can re-use all the existing code.

-- CrawfordCurrie - 28 Feb 2010

$MACRO() as the delayed syntax may clash with the SSP's "functions".

$MACRO() does not provide for multiple levels of escaping.

I think it would be straightforward to support %@MACRO{}% in the macro parser. I suspect that $MACRO() would be harder to implement and whilst I can think of more than one way to do it (e.g. in the macro parser or in the attributes parser), the different approaches have different problems. Implementing it in the attributes parser would mean a far more complex parser (with associated problems of lower maintainability and increased likelihood of bugs). Implementing it in the macro parser would mean overlap between attributes parser and macro parser, and problems may occur with an imperfect overlap.

I am concerned that the corner cases for $MACRO() might be nasty. I acknowledge that I (the "new guy") am making a "gut-level" complaint here. I will try and think of some examples to illustrate what I mean.

-- MichaelTempest - 01 Mar 2010

No worries, I was just trying to illustrate the connection between $inc and "real" macros, to try and avert the risk of a new syntax being invented. I don't actually think using $ is a good idea in any respect. I'm not a huge fan of %@MACRO (it isn't beautiful), but it's practical.

-- CrawfordCurrie - 01 Mar 2010

I like $inc and/or $include() mostly because it fits in with my plans to move all macros to use a common format codebase.

I also do not think there is a need to delay evaluate any macro except INCLUDE (because in just using INCLUDE, delaying evaluation becomes a function call with parameters) - so in that sense, I see %@MACRO% as more complex than necessary.

that said, I could live with %!@INCLUDE{}%, or even

=%%!INCLUDE{}%= -where a double % in a format param is the same as =$percent=

but I really wonder if you've also noticed the fact that if you want %@ or %%

%SEARCH{"Web*"
   format="%%INCLUDE{"SomeAppTopic" section="showstuff" thetoic="$topic"}%"
}%

then you will also need to re-work the parser to not fall over with the embedded quotes that I also don't want to escape

(whereas with the $include() syntax, its pretty much a given that double quotes are not used)

/me thinks we might consider %% = =$percent plus magic to escape quotes...?

-- SvenDowideit - 01 Mar 2010

No, we hadn't overlooked that - the embedded macros defined by %@ must be well formed, that's understood. However note that thanks to the inside-out rule, they can still be composed somewhat:

   * Set BAN = %@BUG
%SEARCH{... format=" oh %BAN%GER{"ouch"}%"}%

will work because the %BAN will be expanded to %@BUG first before the closing }% is seen. At least I think that should work %-/

-- CrawfordCurrie - 01 Mar 2010

I have the feeling we are discussing two things in parallel.

• A way to delay macros - in general
• A way to make a delayed type of include using $include

While I have no objections against having a $include feature in SEARCH I would not want having to define what is inside a nested search in separate sections somewhere else in the many cases where what I want to do is something simple. What can be simple should always be kept simple.

I am thinking about a woman in our office who is writing documentation and is making a simple application for her little 5 person department so they can keep track of which manuals are completed and which are translated into which languages. And her greatest challenge will probably be a nested search where one loop finds manuals and another languages or similar. And all I want for her is something a little simpler than $percnt$quote hell. If we replace $percnt and $quot with something that is not at all simpler to use then we have won nothing and may as well keep the $percnt$quot hell.

The $include would be a great help for the very advanced and very experienced application builders because with parameters and everything you get real power.

• So why not see the two ways as two different proposals instead of chosing one or the other? Ie. raise a new proposal for the $include feature.

What I like about the %@ idea is that it is simple to explain and document. I would not like to try and complicate it too much with assuming setting a variable somewhere else. I would just let the Macro parser rip off a @ each time it sees one and when the final @ is gone the macro is expanded like normally.

The challenge and open question in my mind is how to avoid that the outer macro reacts to the inner because it will still see all the parameters and the }%.

So the parser has to use some take-out-block method so the next level macro does not do unwanted stuff.

The DelayMacroPlugin I did deals with this in a primitive way. It changes % to $percnt and " to $quot. It only changes the quotes around the parameters of the DELAY macro. Not any escaped quotes inside the parameters.

This limits the use of DelayMacroPlugin to be used inside macros that use the escape tokens $percnt and $quot which again limits the use of DelayMacroPlugin to be used for ANY macro which is inside a SEARH or other macro that uses the tokens. In practical life this probably covers 99% of the trouble with inside-out vs outside-in. But it would be sad to introduce a new syntax/feature and from day one have this hard to understand limitation.

There is the issue with commonTagsHandler. But I believe most handler code I have seen does a regex that looks for %TAGNAME\{(.*?)\}%. This also means that these handler are not very good at handling macros within macros. %CALC is a commonly known one we all fight with because of its load order and other issues. And there are many that does the same.

For a %@ to work inside commonTagsHandler the syntax we are need to take out the entire %MACRO{blabala}% including leading and traling %{ and }%.

Could it be as simple as changing the syntax to %@MACRO{trala parem="trala"}}@%? Then we only need the "take-out" code to handle the quotes. The take out representation of quotes does not have to be something we present to the user at all. %@@MACRO{bla}@@% would be 2-level delay.

• It is relatively simple
• commonTagsHandler will not see the syntax - not even in old plugins that know nothing about this new syntax
• Once the parser removes the last @ - the commonTagsHandler can also process these now normal macros.
• The @ part of the implementation is simple. The same parser that expands macros remove a @ from each pair of "%@ and @%" each time it sees them.
• There is still a unresolved problem how to deal with the parameters inside the delayed macro. My proposal is some coded representation of " which is not $quot because that would trigger an unwanted translation back to " inside a SEARCH.
• I would not translate tokens into $dollartokens because if I have a SEARCH within a SEARCH no macro parser can know if the user intended the $topic to belong to the inner or outer SEARCH. Here we will have to rely on the user to delay his tokens with $dollartopic, $dollarweb etc etc. Unless we also introduce a $@topic syntax that the user can use. But I think this does not simpify anything for the user. $@topic is as geek as $dollartopic. I'd rather keep ONE way of doing this part.

-- KennethLavrsen - 03 Mar 2010

Kenneth, I fear you are drifting off into too much implementation detail. The intention (not explicitly stated, but understood by at least Sven, MichaelT and myself) is to change the macro parser to support the embedding. commonTagsHandler is presented with raw text, which most plugins do mysterious and impossible to predict things with. The %@ will hopefully help avoid tripping them up too much, but some will undoubtedly break (several are not even smart enough to support the existing syntax fully). My half-joking $INCLUDE() proposal was only half joking, because it has the positive benefit that it avoids a number of "sensitive" characters (% and {}). You are right the quotes would still remain a problem for commonTagsHandler implementations of macros, however.

$include is a viable alternative proposal that also addresses delayed evaluation; it's not a parallel proposal, it's an alternative approach. The pros and cons have been discussed in detail above.

Note that this proposal still has no committed developer. It would be really great to get delayed eval into the next release - it's something that has been missing for an awfully long time.

-- CrawfordCurrie - 03 Mar 2010

The macro parser would have to (somehow) escape the parameters to %@MACRO{bla}% (I'm using Crawford's syntax here, but the specific syntax used is besides the point) and not use a take-out-block method, because in an example like %SEARCH{... format="%@MACRO{"$topic"}%" }%, the outer macro must be able to "pass values to" the inner macro.

I would avoid $@topic, as that is just as terrible as $dollartopic. I think we should recommend section-includes when implementing nested searches because that provides a way of giving different names to the different SEARCH-levels' $topics. Yes, that means more typing - but it is so much easier to get right.

-- MichaelTempest - 03 Mar 2010

You may have noticed that I just released the DelayMacroPlugin this morning. It does the job for anything inside SEARCH and it is dead easy to use.

I do not find it very user friendly that I have to not only write a simple SEARCH within a SEARCH. I also have to now define a section somewhere in the same topic or another topic. That is ZERO improvement in usability from the current $dollar hell.

Let is look at some real examples. Let us take the one I use in my DelayMacroPlugin. It is a typical example of a nested search with one additional macro inside. Simple thing to do a simple job.

%SEARCH{ "culture"  nonoise="on"
    format="   * $topic is referenced by:$n      * $percntSEARCH{$quot$topic$quot format=$quot$dollarpercntENCODE{$dollarquot$dollartopic$dollarquot type=$dollarquoturl$dollarquot}$dollarpercnt$quot nonoise=$quoton$quot separator=$quot, $quot }$percnt"
}%

With my plugin the syntax is much easier to cope with.

%SEARCH{ "culture"  nonoise="on"
    format="   * $topic is referenced by:$n      * %DELAYSEARCH{"$topic" format="%DELAY{macro="ENCODE" delay="2" "$dollartopic" type="url"}%" nonoise="on" separator=", " }%"
}%

Note that all % signs and quotes are normal unescaped. Only the $dollartopic needs to be escaped because the plugin cannot ever know if the $topic should come from the inner or outer search loop.

You must admit that this is very very simple.

In a %@ @% syntax the same would be even more simple

%SEARCH{  "culture"   nonoise="on"
    format="   * $topic is referenced by:$n      * %@SEARCH{"$topic" format="%@@ENCODE{ "$dollartopic" type="url"}@@%" nonoise="on" separator=", " }@%"
}%

I tried to write the same example with the $include inside the format but I end up with still having to use $quot and I probably need to define two sections in the topic.

Please show with an actual complete example how you would solve the same simple problem with $include. Full eample please with the sections you would have to write. And when done give yourself a NerdoMeter Score. My @@syntax example is probably a 4.

-- KennethLavrsen - 03 Mar 2010

OK.

%SEARCH{ "culture"  nonoise="on"
    format="   * $topic is referenced by:$n      * $percentINCLUDE{\"%TOPIC%\" section=\"inner\" topic=\"$topic\"}$percent"
}%
%STARTSECTION{"inner" render="hidden"}%
%SEARCH{"%topic%" format="$percentINCLUDE{\"%TOPIC%\" section=\"innerinner\" topic=\"$topic\"}$percent" nonoise="on" separator=", "}%
%STOPSECTION{"inner"}%
%STARTSECTION{"innerinner" render="hidden"}%
%ENCODE{"%topic%" type="url"}%
%STOPSECTION{"innerinner"}%

Of course that's the current implementation, without adding any syntactic sugar. $include is just $percentINCLUDE{}$percent recoded as something a bit more palatable.

With a friendler, more "heredocsy" syntax, we might have:

%SEARCH{ "culture"  nonoise="on"
    format="   * $topic is referenced by:$n      * $inc(inner)"
}%
%INC{"inner"}%%SEARCH{"%topic%" format="$inc(innerinner)" nonoise="on" separator=", "}%%ENDINC%
%INC{"innerinner"}%%ENCODE{"%topic%" type="url"}%%ENDINC%

On a scale of 0..10 (where 10 is ubernerd) I'd give this a 6.5. But at the same time I'd give your %@@ example a 6.

-- CrawfordCurrie - 03 Mar 2010

Just like I feared. FRIENDLY IRONY START If you guys think your solution is simpler than adding the $percnt and $quot then you need to get your heads examined. FRIENDLY IRONY END.

I think this syntax is great for when I want to

• Do something even more complex
• Reuse something many places. The $inc/%INC syntax enables the more advanced users to make a small library of reuseable TML pieces. And it enables to do thing much more dynanic and conditional. I would use it for this.

But again - I think we make a mistake if we make it a discussion of one or the other.

The @@ syntax is simple and great for the simple cases where you make a small nested search or put an ENCODE inside a SEARCH. Something that is normally a single line thing. The only purpose of this syntax is to avoid the fights with the $percent$quot hell.

And the $inc/%INC syntax enables very advanced constructions and even more important - TML code reuse.

Please consider both.

-- KennethLavrsen - 03 Mar 2010

so long as

%INC{"innerinner"}%%ENCODE{"%topic%" type="url"}%%ENDINC%

is a synonym for STARTSECTION, which we can then deprecate as too much typing, I'm happy to call use $inc() for delayed sectional include, %!INCLUDE for 'normal' sectional include and INC and ENDINC for denoting sections.

, and then %!@MACRO@{}% for when I want to quickly delay the evaluation of any random macro - which admittedly I can't think of a time when I would want to, but I'm sure others will find a place for it

and thus, we have a set of syntax that is in my mine a suitable syntax for both the delay and here docs feature request, which is imo much more inline with the pre-existing.

oh, and if someone wants to nest the INC sections, they presumably can add a name to the ENDINC, just as in ENDSECTION..

-- SvenDowideit - 04 Mar 2010

An attempt to draw this to closure. Here's where I think we are. A number of methods for expressing late evaluation have been proposed. Of these, a number don't work for technical reasons (a simple assignment operator has been kicked out as useless, a brand new embeddable macro language as just far too much work, and heredocs don't provide enough nesting power). Of the ones that might work, we have:

1. Quoting - e.g. using backticks `%DONTEVALTHIS%` and string concatenation.
2. A special escape for macros e.g. %@MACRO{...}@%
3. A single custom "available everywhere" operator $inc that can be used to include a section from elsewhere

All of these have pros and cons, and we have been unable to agree on the best approach. To try an break the deadlock, I have tried to illustrate each of the approaches given a common requirement - derived from Kenneth's example of a nested search with a format containing a macro call.

Approach*	Pros	Cons
Quoting (1)	Flexible, obvious once you understand it	may not be obvious to non-programmers
		Syntax clashes with most commonTagsHandler regexes
		assumes that the viewer's font sufficiently differentiates between quotes (Sven has watched this be a problem more than once in other languages)
Escape (2)	Familiar for those used to %MACROS	Visually messy
	Immediately obvious what's going on
	Consistent with format	New syntax more likely to clash with plugins
	New syntax NOT likely to clash with plugins
$inc (3)	Easy to implement	Difficult, non-obvious syntax
		Relies on sections, which are horrible to implement
		That macro language strikes again
		Messy to mix early/late eval
		Still doesn't entirely eliminate $dollar
$anyregisteredmacro(4)	Easy to implement, and encourages the adoption of a common formatting infrastructure	requires enough plugins to use a common formatting evaluation (um, expandMacros but done pluggably) CDot believes this is just (2) with a different syntax ($name instead of %@NAME)

I have resisted the temptation to assign nerd/usability factors, as one man's nerd is another man's newbie i.e. "nerd" is always a relative term, and our users are not just total beginners. We have to consider all levels of experience. Also, I'm trying not to consider implementation cost too much, though it's not easy to ignore.

Please add any additional pros and cons you see, and add your opinion to the table below.

Name	Approaches, in order of preference
CrawfordCurrie	1,3,2
KennethLavrsen	2,3
SvenDowideit	4,3,2
PaulHarvey	2,1, 4.1

-- CrawfordCurrie - 13 Mar 2010

Crawford writes "New syntax more likely to clash with plugins" for the Escape syntax. That statement is undocumented and I believe wrong. I assume Crawford things about the commonTagsHandler.

All plugins I have seen use regex like

$_[0] =~ /%CALC\{.*?\}%/ )

$cell =~ s/%CALC\{(.*?)\}%/&doCalc($1)/geo;

 $_[0] =~ /%TABLE{.*?}%/; 

my $PATTERN_TABLE = qr/%TABLE(?:{(.*?)})?%/;

return unless ( $_[0] =~ m/%ACTION.*{.*}%/o );

s/%ACTIONNOTIFICATIONS{(.*?)}%/_handleActionNotify($web, $1)/geo;

$_[0] =~ s/\%RENDERFORM{(.*?)}\%/Foswiki::Plugins::RenderFormPlugin::Core::render($1,$_[1],$_[2])/ge;

Every time I look at a plugin it always requires that the Macro use starts with %MACRONAME{ and ends with }%

The minute we add a character between the first % and the macro name and the same between the last } and the % I cannot see any macros catching he delayed macro string with its regexes. So I believe the syntax is plugin safe.

Now look at the same with the quoting proposal. PLONK. It fails all the example regexes above. I cannot see how it will work. I do not yet understand the feature at all. How is it supposed to work? I look at the syntax example and it makes absolutely no sense to me. Please explain how it works and how I would use it.

I have used my own DelayMacroPlugin for a couple of nested searches lately and I must say that even this plugin makes life much much easier when you do not have to escape all the % and ". We need a solution to this

-- KennethLavrsen - 13 Mar 2010

I don't think I agree with your pro's and con's :/, but what you said on ENCODEnlsToBR strikes a chord with me

Why exactly not continue with what we already have, and extend the $formfield() syntax to be the magical delayed eval?

what I mean is %ANYREGISTEREDMACRO{}% automatically gets its twin $anyregisteredmacro(), which is evaluated in a grand unified Foswiki::Render::format() function. If you look at what I've done with Foswiki::Search::formatCommon() and its more horrid mate formatResult(), you'll see the beginings of this work - though extracted from the pre-existing Search code.

-- SvenDowideit - 13 Mar 2010

But sven, what about quotes in attributes? Or will the parser magically differentiate between inner " inside a $foo() and the real " that ends a format string?

Would be interesting for back-ticks if it extended to the following - render without <p> and random creation of <dt> -

`{
   "foo": "a",
   "bar": 1,
   "car": 2
}`

-- PaulHarvey - 13 Mar 2010

We need delayed macro evaluation to work other places than SEARCH. I am looking for a SIMPLE GENERIC solution working anywhere.

-- KennethLavrsen - 13 Mar 2010

Kenneth, the reason I highlighted the %@ syntax as likely to clash with plugins is that it is obvious for any code to try and split on % in the way the core macro parser does. So it is better to avoid this character. I also want a generic solution that as far as possible minimises the risk of tripping up existing extension code.

The reason I dropped (4) from my original list is there's two ways of looking at it. The first way is what Sven describes above, where $macro does duty for %MACRO inside strings. This is just an alternate syntax for (2), as discussed earlier, so I didn't see value in classifying it separately - it has identical issues, just a slightly different syntax. The second way is to invent a new calling convention for macros (or a subset thereof) such as $formfield(Blah,some,more,special,params). I don't personally consider that as sensible, for the reasons I gave above, and at the end of ENCODEnlsToBR. As long as we are not considering implementation detail, I think Sven means the former and Paul means the latter (4.1) but it's unclear, nor is there an example, so I have put "votes for 4" (and 4.1) in red, above, until you can clarify what you meant.

Paul, interesting idea about the backticks. This vision I had was of single-quote in Perl, which simply suppresses variable interpolation at point of definition. Extending it to render-time semantics is, I fear, a step too far, as it would require the code to retain the "backtickedness" of the string throughout the rendering process (kinda like tainting).

-- CrawfordCurrie - 14 Mar 2010

For the record. I am not at all hung up at %@ being the syntax. It is the principle I am after. And I still haven't seen an explanation how the back-tick syntax is supposed to work. The example shows the backtick before the % meaning that commonTagsHandler plugins will ignore it and still see the %TAG{.*?}%

With the backtick between % and the name it would be hidden. %`SOMEMACRO{blabla}% or safer %`SOMEMACRO{blabla}`%. That would be OK syntax as well... IMHO.

Still the parser needs to also protect the quotes used inside. My plugin turns them into $quot. But that also means my plugin is limited to working inside the very few macros that take tokens. My plugin is not the generic solution either.

-- KennethLavrsen - 14 Mar 2010

Apologies, the explanation was terse and embedded in other discussion at UseSyntaxToChangeEvaluationOrder#BackTickExplanation

Anyway, how backtick would work. The explanation is in two parts (neither is currently implemented).

1. backtick is a quote character, and is an equal peer of " (double quote) with the difference that "special characters" within the backtick are automatically escaped (you can think of them being converted to their $dollar equivalent, so `%BLAH%` is equivalent to "$percentBLAH$percent") Backtick supports \` escapes.
2. any pair of quoted strings are concatenated when they are placed next to each other (separated only by whitespace) in a macro parameter list. So, "blah" "blah" is equivalent to "blahblah". Since ` is an equal peer of ", that means we can write "blah" `%BLAH%` "blah" to get the equivalent of "blah$percentBLAH$percentblah". This string concatenation technique is fairly common in programming languages where it is used, for example, to break long strings over newlines.

The advantages of this approach are

1. it can be handled entirely by the parser, no other code is affected
2. It is highly readable
3. it gives us the added (minor) benefit of splitting param strings over newlines

-- CrawfordCurrie - 14 Mar 2010

I am very strongly against using something as ambiguous as a backtick. I've had issues both in teaching a language that used them (when some students have a font where its indistinguishable from a single quote) and in watching pro's trying to quickly read the same code, and missing the subtle difference between ` and ' on their font - or how about ital`ic and ital'ic

simplest way to put it - it is NOT highly readable

on the other hand - @SOMEMACRO{}@ or pretty much any single char is readable - I don't know why you'd want the additional '%'.

and yes, to me the syntax is mostly there for the reader of the code - so long as it doesn't cause the parser to be totally invalidated thats the least important thing.

personally, I'd be happiest if there was a limit on the depth of the delay - its damned hard to read code like

(eg from Paul on irc)

• %!SEARCH{format="@SEARCH{format="$topic: @FOO{"$@topic"}@"}@"}% and it can be re-written much more readably using the rather ancient ParametricIncludes feature.

-- SvenDowideit - 14 Mar 2010

I sat down with my supervisor today, a software engineer who does use SEARCH and nested SEARCH. I explained @MACRO@ and back-tick approaches to him. Feedback was: backtick concept is better, find a different character/combo to do it.

He suggested something like %SEARCH{format="A macro that is ["%DELAYED%"]"}% - "brackets are an important concept to signify that something is encapsulated in a special way".

-- PaulHarvey - 15 Mar 2010

And that, unfortunately, illustrates Sven's point; you haven't understood how backticks work. The point of backticks is that they operate as quotes, i.e. as equal peers to double quotes, and strings are composed by concatenating them. So the example you gave above doesn't work. If we replaced backtick with [] (and I'm not wedded to ` in any way) then %SEARCH{format="A macro that is "[%DELAYED%]}% is how it would be written.

I'm still trying to work out in my own mind how right if Sven is. Nobody I have asked (admittedly not a representative sample) has any problem with the backticks concept. Sometimes the inability of a class to understand a concept is down to the teacher, rather than the class.

If you feel the urge to be more perlish, then mentally substitute forward single quote for backtick. The only reason I didn't use it is that it's already supported in "friendly" mode by the attrs parser.

-- CrawfordCurrie - 15 Mar 2010

I am afraid it is not the teacher or the student that is the problem. It is the book. To be it is getting obvious that the backtick feature is not intuitive and too geeky to use. What people need is a more readable way to delay macros than using the $percnt and $quot combination. That is all. And there has been several good proposals for exactly that. %@MACRO{..}@% or @MACRO{...}@ are the two best I have seen until now. It is not logical that "string"`other string` is causing a concatenation. It is not intuitive. And "string"\`other string\` gets even worse.

-- KennethLavrsen - 15 Mar 2010

I cannot get enough mindshare to get agreement on this proposal, and it's too important to proceed with the limited input received so far. I'm removing my name as committed developer, and parking the proposal.

-- CrawfordCurrie - 24 Mar 2010

Probably a good idea to park it - not to forget it - but to give time to get better ideas. But for sure we should get this set for 2.0.

Meanwhile people can have fun with my DelayMacroPlugin

Probably a good idea to start on a clean sheet of "paper" with a fully spec'ed proposal next time.

-- KennethLavrsen - 24 Mar 2010

Crawford indirectly pointed me here with a comment he made on RegisterSectionTagHandler. Even though this proposal has been parked, I thought I'd take a moment to weigh in and mention that I agree with Kenneth on this one. Using something like %@MACRO{}@% to delay macro expansion - and multiple @s for multiple levels of delay - seems a lot more readable and understandable for an ordinary user than the other presented alternatives. I also think it is a sorely needed feature, since the current method of delaying macro expansion is really hard to read and understand (though I guess DelayMacroPlugin is available in the meantime).

-- LeilaPearson - 02 Oct 2010

The main problem is format $token escaping. Sometimes it's wanted, sometimes it isn't.

The fix is to formally register that a param supports format tokens. So I have some sub-proposals:

Change duty of $syntax arg in Foswiki::Func::registerTagHandler($var, \&fn, $syntax)

Overload $syntax to take a hashref (at the moment it takes string 'classic' or 'context-free'. We can keep supporting that.).

Foswiki::Func::registerTagHandler( 'FOO', \&fn, 'classic' ) becomes

Foswiki::Func::registerTagHandler( 'FOO', \&fn, { paramSyntax => 'classic' } )

And Foswiki::Func::registerTagHandler( 'FOO', \&fn, 'context-free' ) becomes

Foswiki::Func::registerTagHandler( 'FOO', \&fn, { paramSyntax => 'context-free' } )

Now, indicate with the hashref which params require Foswiki::expandStandardEscapes (should probably be cloned into Foswiki::Func, ie. make public):

Foswiki::Func::registerTagHandler(
  'SEARCH',
  \&fn,
  {
    paramSyntax => 'classic',
    params => {
      header =>    { applyStandardEscapes => 1 },
      format =>    { applyStandardEscapes => 1 },
      footer =>    { applyStandardEscapes => 1 },
      separator => { applyStandardEscapes => 1 },
      newline =>   { applyStandardEscapes => 1 }
    }
  } 
)

This delegates the responsibility of applying the standard escapes back to Foswiki. Not sure if delegation is necessary, probably would make plugin authors unhappy (Micahel?). Probably just need to indicate that a param has Foswiki::Func::expandStandardEscapes applied to it.

Centralising standard escapes to a Foswiki::Func method is desirable because we want to teach Foswiki new formatting tokens like $include() or $query(); and plugin (macro) specific $tokens should be made known to Foswiki to enable better WYSIWYG macro editing, among other things

Anyway, the reason I've gone for attributes-per-param is for my desire to FormaliseMacroSpecification to better enable WYSIWYG macro editing, among other things (like useful user-feedback such as "I don't understand that param" or "expected a bool value but got a banana").

Assume delayed macros only appear in params which support standard escapes

Let Foswiki assume that %@MACRO@% (or %MACRO{ delay="1"}% ) are only used in parameter strings which are processed with Foswiki::expandStandardEscapes as outlined above.

%MACRO1{
  format="%@MACRO2{
    query="text=~'$notAtoken'"
    format="bar - $debar"
  }@%"
}%

Should be functionally equivalent to:

%MACRO1{
  format="$percntMACRO2{
    query=\"text=~'$notAtoken'\"
    format=\"bar - $dollardebar\"
  }$percnt"
}%

Note that we didn't convert $ to $dollar in the query param of MACRO2 (assuming only the format param was registered as requiring standard escapes).

Notes

• Support for the new delayed macro syntax would be gradual, IE. only for those macros which use the new registerTagHandler syntax
• I assume the macro parser wouldn't actually have to expand things as $percnt but rather use some internal mechanism to delay things?
• Exact mechanics of teaching macros new format $tokens and formal macro syntax specification to be dealt with in other proposals, but might be something like:

    Foswiki::Func::registerTagHandler(
      'SEARCH',
      \&fn,
      {
        paramSyntax => 'classic',
        tokens => { topic => \&handleToken, nhits => \&handleToken, ... }
        params => {
          header =>    {
            applyStandardEscapes => 1
            nhits => \&handleHeaderToken
          },
          format =>    { applyStandardEscapes => 1 },
          footer =>    { applyStandardEscapes => 1 },
          separator => { applyStandardEscapes => 1 },
          newline =>   { applyStandardEscapes => 1 }
        }
      } 
    )

-- PaulHarvey - 02 Oct 2010

Actually, to be honest I might be asking too much of Foswiki::Func::registerTagHandler. Somehow I wonder if a proper OO Macro class would be a better approach (each MACRO would be based on Foswiki::Macro at some level, and override methods, expose properties as required?)

-- PaulHarvey - 02 Oct 2010

gads, talk about loading together a pile of outstanding proposals :/ OO Macro Class - should be moved into a separate proposal really - and if we use that (by implementing more of Meredith's lightweight tags proposal), and FormaliseMacroSpecification as the new way to define tags, we might have something really useful - or something that breaks the camel's slowness back.

2.0 - but well worth doing imo - I really would like to get some more ground work done so that ComponentEdit becomes a normal part of the creation of MACRO applications.

-- SvenDowideit - 03 Oct 2010

I know this talks about too many separate proposals. But these are interconnected issues. The choices we make here affect the ability to make macros efficiently WYSIWYG editable.

-- PaulHarvey - 03 Oct 2010

A related discussion came up at SyntaxForDelayingMacrosAndFormatTokens.

-- MichaelDaum - 30 Dec 2011

Commenting is disabled while not logged in