Feature Proposal: Eliminate $dollar etc hell. Also see older UseSyntaxToChangeEvaluationOrder

Motivation

Easier to read write and maintain delayed macros and format tokens in nested macros

Description and Documentation

Incorporate %5MACRO{p1="$6topic" p2="..."}% as a delay syntax into the core.

I believe that currently that a MACRO name must not begin with a digit, similarly a format token cannot either. Therefore extending these to begin with one or two digits (as nesting is limited to 16 levels) and possibly followed by some other character.

When I actually look at example of the new coding I find that my eyes pick up %5MACRO% as a logical whole, whereas %5:MACRO% seems somewhat disconnected (especially with a fixed font). Therefore, I would suggest not having a character after the number.

As I understand the macro processing which splits the topic text on the % and the macro is not 'encountered' until the final %. This would mean that when the final % is reached then instead of invoking the macro immediately, test to see if the name has a numeric prefix. If so, reduce count by 1 and pass this back without calling the macro. So, %5MACRO% would become %4MACRO%, %4MACRO% would become %3MACRO% until %1MACRO% would become %MACRO% thus evaluating the next time thru the pipeline.

The format token can follow the same scheme I believe, indeed should for consistency.

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Plugin upgraders/writers: if this is implemented be careful not to assume this new syntax if you need to pass back MACROs with delayed evaluation. If someone upgrades the plugin to a site using the old core it will break things. A similar thing happened when $percent was allowed as well as $percnt: $percent was used in an upgraded plugin and ...

Implementation

See also:

• UseSyntaxToChangeEvaluationOrder
• HereDocumentSyntaxForMacros
• ProblemsWithHereDocuments

-- Contributors: JulianLevens - 29 Dec 2011

Discussion

And the current DelayMacroPlugin does not enough? Asking because i never used it - but the plugin is here for this purpose.

-- JozefMojzis - 29 Dec 2011

DelayMacroPlugin is very good I'm sure, indeed it was part of the inspiration. However this proposal:

1. There will be some performance benefit (not much, but every little helps)
2. Easier on the eye IMNSHO (others may find it ugly )
3. Simpler to follow (no extra macro in the way)
4. Co-ordinates with format tokens
5. Orthogonal with existing syntax

-- JulianLevens - 29 Dec 2011

I know it seems obvious, but with proposals such as these I'd really like to see some examples.

I somehow wonder if there should be some formalisation via Foswiki::Func::registerTagHandler() as to which parameters support delayed macros.

Also, I wonder if we can completely brainstorm something other than a new syntax for convenience... I know some think that a proper DOM for Foswiki is unrealistic, and that discussion of it is a distraction, but it's something I'm very interested in

-- PaulHarvey - 30 Dec 2011

Paul I'm not sure why you refer parameters that support delayed MACROs in registerTagHandler. All macros would automatically support this syntax (even commonTagsHandler ones as the basic % .. % parsing is done in the core isn't it?) Format tokens are another story and I'm not aware of any central and comprehensive management of this.

Remember that this is about a standard macro delay syntax. The second example has 3 levels of delay, and is relatively easy to follow. Of course this could be re-worked into sections, but some delays are still required.

Granted that this is a lot about a better syntax for convenience. But that's not to be sneezed at, death by a thousand inconveniences and all that. Remember all this code will need maintenance and you will be glad of the convenience.

I hear what you say about a proper DOM, but that OT here. I'll create another topic (some other time) about that ...

OK, a couple of examples follow:

From FormattedSearch

%SEARCH{
   "culture"
   nonoise="on"
   limit="5"
   format="\
   * $topic is referenced by:
      * %1SEARCH{
         \"$topic\"
         nonoise=\"on\"
         format=\"$1topic\"
         separator=\", \"
      }%"
}%

Taken from EasierNestedSearch and quickly hacked, so it's probably not quite right but good enough for illustration.

%~~ SEARCH{%DO%%URLPARAM{"start" default="WebHome"}%%END%
~~~     format="   * %FORMAT%
*~~     $n%1SEARCH{%1DO%$topic%1END%
~~~        format=$quot      * %1FORMAT%
*~~        $n%2SEARCH{%2DO%$2topic%2END%
~~~           format=$2quot         * %2FORMAT%
*~~        $n%3SEARCH{%3DO%$3topic%3END%
~~~           format=$3quot            * %3FORMAT%
*~~        $3quot
*~~        }%
*~~        $2quot
*~~        }%
*~~        $quot
*~~     }%
*~~     "
~~~  }%

Also from EasierNestedSearch, where Sven suggested refactoring using includes so we ended up with a format parm like:

format="$percntINCLUDE{"%BASEWEB%.%BASETOPIC%" TNAME="$topic"}$percnt"

Which simplifies to:

format="%1INCLUDE{"%BASEWEB%.%BASETOPIC%" TNAME="$topic"}%"

-- JulianLevens - 30 Dec 2011

IMHO, the proposed syntax sugar on this proposal -- as well as these on UseSyntaxToChangeEvaluationOrder -- is too complicated and only trying to mend symptoms. The real issue still is evaluation order. Adding more trickery to delay evaluation is only a work-around.

In real wiki apps there are larger blocks of tml that need a different evaluation order, not just one makro here and there. So instead of dumping more special symbols into the TML soup I'd prefer some block-level marker that flags the contained TML to be parsed differently, like this:

<parser mode="outside-in-left-to-right">
%SEARCH{
   ...
   format="%INNERMACRO{}%"
}%
</parser>

Note, that mediawiki markup is parsed outside-in-left-to-right by default. So when translating mediawiki content to TML, this content should be parsed in that mode in Foswiki as well to render it correctly, i.e. when mapping the mediawiki "template" concept to parametrized includes in Foswiki. Maybe the parser mode could be switched to a different mode using a preference variable or some TOPICINFO meta information. Not sure which level of configuration makes more sense.

What I am quite certain about is that additional special symbols to block normal parsing for a while is too fragile and hard to maintain from a wiki app writer pov.

-- MichaelDaum - 30 Dec 2011

Michael, I may misunderstand, but the $dollar etc stuff does provide us with a way to control evaluation order, yes by carefully delaying at the right point by the right number of levels, so it's syntactically yuk and fragile to boot. Now writing that sentence helped me to appreciate that this proposal (and others mentioned in related proposals) only improve the syntax and do not provide a fundamentally better way to handle this.

Conversely, your syntax:

<parser mode="outside-in-left-to-right">
%SEARCH{
   ...
   format="%INNERMACRO{}%"
}%
</parser>

While it does solve the problem to completely change the evaluation order when necessary. I can imagine that this will still need fine grained control to revert the evaluation order back to normal.

As for TML soup, well I grant you that I'm suggesting a bit more syntax, but I believed that this would be low hanging fruit to enable me to eliminate $dollar, It will therefore make my soup a lot clearer overall.

Nonetheless, I will ponder a more complete evaluation order solution as well.

-- JulianLevens - 31 Dec 2011

This may be harder to implement but how about turning my idea around a little. Still using %1MACRO% etc, but now %1MACRO% would mean evaluate me first, %2MACRO% would be processed second etc. A slight variation would allow the smallest number to go first, then then next largest etc. This would allow %10MACRO% %20MACRO% and allow a %15MACRO% to be squeezed in between and reducing fragility.

Formatting tokens will be quite tricky.

-- JulianLevens - 31 Dec 2011

imo we need to avoid numbers or counting 'delay' syntax, as its not going to help for code re-use - If i INCLUDE a section that contains delays, will it use the count that you wrote into it for that code to be used inline, or does the INCLUDE magically change the count (people will expect both, depending on what they're trying to acheive). Worse is the issue that someone copying your delayed code into another SEARCH would then have to edit each one to increment it once more.

tbh, that is one of the advantages of $$$dollardollardollar, its so horrible, that you are (er, ok, should be) looking for doing it another way - and that way should lead towards extracting the code - syntax smells exist in part to signal a need to refactor.

One of the thngs I wanted to explore in UseSyntaxToChangeEvaluationOrder is to stop us from inlining delays in the syntax using $dollar mess, and instead to use funtion call like mechanisims (sectional includes with parameters for eg), specifically so that we re-use blocks of Macro code.

When I write complex Macro code, I tend to use

%SEARCH{
   ...
   format="%percentINCLUDE{topic section=other param1=$topic param2=formfield(ReportedBy)}$percent"
}%

Which otherwise we were propsing as

%SEARCH{
   ...
   format=@INCLUDE{topic section=other param1=$topic param2=formfield(ReportedBy)}@"
}%

which is why I was originally asking for just adding a $include()

-- SvenDowideit - 31 Dec 2011

Julian, switching evaluation order back to "normal" is some magnitudes easier by explicitly calling some macro %EVAL{}% to expand the contained TML fragment before inserting it into the surrounding TML.

<parser mode="outside-in-left-to-right">
%SEARCH{
   ... 
   format="%EVAL{"%INNERMACRO%"}%"
}%
</parser>

Sven, an $include() is all fine but the problem is more about parsing TML in general and not only about %SEARCH .

-- MichaelDaum - 31 Dec 2011

Sven, I had not considered the %INCLUDE% dynamic, which if not fatal is going to require some amazing surgery to save the patient.

Michael, I wasn't sure that %EVAL{}% would actually work like you suggest, but then realised it would just need some lookahead in the parser to do the EVAL blocks first and then normal processing.

Does that suggest outside-in would have been a better default?

I then ponder how programming languages do this, or at least imperative scripting languages, and that is they parse outside in, but evaluate inside out and that's roughly the Foswiki way.

Then again, this is not procedural code. We are building n-layers of expression, with interdependencies between layers. The $include approach will work for nested layers which therefore have consistent dependencies between them. I'm not sure its a generic solution.

I am left with a better understanding of the situation:

1. We are doing n-layered expressions
2. I cannot see a way to eliminate them
3. Therefore we need a generic solution to controlling evaluation order
   • This must in turn work logically whether inline or via an include

I can see a way to eliminate many n-layered expressions. That's by adding joins to the QUERY language. I'm writing my own DBI based back end store, using Crawford's DBIStoreContrib to crib ideas and code. However, I intend to add in some of the things Crawford mentioned as missing: flexibility and indexes. I will also add in versioning (and fast querying of) plus the regex cache idea. Now, joins are becoming an important item on the to do list

-- JulianLevens - 31 Dec 2011

Extra cool to see somebody picks up working on DBIStoreContrib seriously. This is really an important missing enterprise feature. A lot of enterprise customers will jump on that wagon while shying away from more exotic store implementations like mongodb.

With regards to nesting TML in argument position: 2 layers are very frequent, 3 very rare already, 4 layers - never seen them. 3 layers of escape tokens probably are a hint to think about restructuring TML code in a more modular (and reusable) way. 4 layers of escape tokens are most certainly a hint to very badly structured code.

With regards to parsing and evaluation order in other languages. These are happening at the same time in TML, as Foswiki doesn't build up a parse tree nor caches it for later reference. So talking about parsing order in TML actually means evaluation order when comparing it to other languages. And yes, these evaluate inside-out-left-to-right as Foswiki does by default.

However, other programming languages introduce "functions" as first-order objects. So functions can be (a) defined in place anonymously and (b) be used in argument position of other function calls. The concept "function" and "function call" more or less maps to topics holding reusable blocks of TML in sectional includes. However there's no such thing as "anonymous functions" in TML. All have a unique physical location and address, which sort of makes it impossible to anonymize and use them in argument position to delay evaluation.

In contrast, when changing evaluation order to outside-in-left-to-right, delayed evaluation is paralleled by prematured evaluation to switch back and forth between both modes in programming. Premature evaluation is very common in programming languages as well, for instance perl's eval block construct. The block argument to eval sort of resembles anonymous functions being defined and evaluated in place. Note, that they aren't stored as a first-order object in place however; they simply return their result right away, which