Feature Proposal: Support all macros in template topics

Motivation

Clean up the way templates are defined, make them more understandable, plus get access to all macros in template topic definitions

Description and Documentation

In Tasks.Item13909, JohnRouillard pointed out that TWiki had added a "feature" to expand any macro during template topic expansion.

The TWiki implementation hasn't been thought through, though it's actually quite a good idea if done properly. It has the potential to help clean up the way templates are defined, make them more understandable, plus get access to all macros in template topic definitions.

One of the most annoying problems I have come across is the divide between "skin" templates and "topic" templates. This is a frequent source of confusion for the unwary user, and a source of major confusion in the code. Currently there are three "types" of macro:

1. Skin template macros / pragmas (TMPL:P, TMPL:INCLUDE, TMPL:DEF..TMPL:END, TMPL:P, TMPL:PREV)
2. Topic template macros (the subset of normal macros that always expand on topic template instantiation, and the control mechanisms like expandvariables and templateonly)
3. "Normal" macros.

In several places developers have tried to shoehorn the "normal" macro functionality into "skin" and "topic" template expansion

• selective macro expansion on topic template instantiation e.g. DATE, WIKINAME
• the TMPL:P{condition} mechanism (which I deeply regret)
• the expandvariables and templateonly sections

This has left us with a bit of a mess, but it isn't irredeemable. Let's say we accept that there are two "expansion domains" *. The first domain applies when a template - "skin" or "topic" - is being instantiated. The second applies when a topic is being viewed. If we think of "skin" and "topic" templates being essentially the same thing, we can see that these expansion domains already exist, i.e.

• TMPL: at the start of a macro puts it in the "template" domain e.g %TMPL:INCLUDE.
• No TMPL: means the macro is expanded in the topic domain e.g. %INCLUDE.

This is something that has been in the back of my mind for a long time, perhaps it's time has come. There are clearly limitations - a condition using URL params during a "skin" template expansion probably wouldn't be a good idea - but these can be resolved by identifying those macros that can be expanded during skin template expansion, and those that have to be left unexpanded in the output. For example, a macro might fire a NotAllowedDuringTemplateExpansion exception. I can see this leading to greater things:

• A macro that instantiates a %DEF{...}%
• Rationalisation of TMPL:P, IF, TMPL:INCLUDE and INCLUDE to eliminate duplication / overlap
• Template precompilation

Examples

Let's say we have a template topic that contains:

Today is %DATE%

at present this expands to the date that the template topic is instantiated. This is fine, but it's rather confusing as %DATE% is normally today's date.

• Only an unmemorable subset of macros expands in template topics, other macros are left untouched
• Complex control mechanisms using SECTION are required to control what is, and what is not, expanded

The proposal is in three stages.

1. The first step is to introduce a TMPL: prefix that can be attached to all macros (except sections, which are handled out-of-band) to expand them at template instantiation time.
2. Deprecate (under configuration control), and eventually remove, automatic expansion of the subset of macros in template topics i.e. require the use of TMPL:, allowing us to support both TMPL:DATE (expand when template is instantiated) and DATE (same thing, but expand when generated topic is viewed)
3. Rationalise and remove overlaps between skin template pragmas and macros

Note: because of a pre-existing hack in Foswiki.pm, TMPL:TMPL:P will expand a skin template in the templated topic. So an additional change may be required to "correctly" support TMPL:P

Implementation

Checked in to branch "Item13909"

-- Contributors: CrawfordCurrie - 05 Jan 2016

Discussion

+1, this looks like a very well though out solution. Much better than the alternate implemented by the other project. If you've already got the patch written on master, please pop it into an item branch so we can all easily test

• git checkout -b Item13909 Switch to a new branch with the pending changes
• git commit [the modified files] Commit the changes into the new branch
• git push -u origin Item13909 push the new branch to github, and automatically make it a tracking branch

-- GeorgeClark - 06 Jan 2016

Knowing how many template topics we have spread around made in over a decade now - how do I take advantage of the new feature while still being able to maintain compatibility? If it is a configure only setting to enable or disable legacy then it is a plaig vs colera decision. I'd like to enable the new feature and use them in new applications while still being able to leave the old alone. The issue is only related to a few macros that get expanded like DATE and WIKINAME. Actually 99.99 % of the cases are those exact two macros.

-- KennethLavrsen - 06 Jan 2016

That's a good question. Topic templates written to use the %<nopTMPL:RANDOMMACRO% support described in step 1 above will obviously only work with a Foswiki that supports the extended syntax. Templates which do not use this feature will continue to work as they always did.

Step 2 is not something we'd undertake lightly. As you correctly point out, it would shake existing templates to their core. The start point would be to fix all existing topic templates. I'll do that in the branch.

-- Main.CrawfordCurrie - 08 Jan 2016 - 13:02

This is long past any objection date, but Jast brings up an interesting point. There is one collision between templates and macros:

TMPL:INCLUDE could be requesting either of:

• Inclusion of a skin template file, or
• Early inclusion of a Topic.

How does this get resolved.

-- GeorgeClark - 29 Jun 2016

Documentation of Skin vs. Topic template behavior

	When expanded in:
Token/Macro	Skin Template	Topic Template
TMPL:P	Includes a previously defined template block. (TMPL:DEF{})	Same
TMPL:INCLUDE	Includes a template file	Includes a topic.

So TMPL:INCLUDE has a context sensitivity.

• When used in a Skin Template (file or topic) during template expansion it's used to include other template files.
• When used in a Topic Template, then it requests an early expansion of another topic/section.

One concern is that for TMPL:P to be useful, the TMPL:DEF Needs to have been created. As there is no include for a Skin Template during Topic Template creation, then TMPL:P is probably not useful in the Topic context. In my testing of a modified NewUserTemplate, TMPL:P and TMPL:TMPL:P appear to be not used at all. They just disappear. I tried expanding blocks defined in both registermessages template, and in the foswiki.tmpl to make sure that a block would have been defined.

-- GeorgeClark - 29 Jun 2016

TMPL:P is very useful in a topic context. Example: define how a user profile image is fetched using a TMPL:DEF{"user::photo"}, then create a SEARCH to list all responsible persons (however that is defined) and call $percntTMPL:P{\"user::photo\" user=\"...\"}$percnt for each found. The advantage of defining user::photo that way is that you can change the wiki app using a skin overlay. By default a user photo would be fetched from the list of attachments to a user profile page. A customization using a skin overlay could then redefine user::photo to fetch it from an LDAP directory as a thumbnail blob (via %LDAP). The initial wiki app doesn't need to be changed, only the SKIN path needs a prefix that fetches user::photo from another template. The same user::photo definition would be used in meta comments as well, or any other people search.

-- MichaelDaum - 30 Jun 2016

Using the prefix TMPL:... is particularly misleading for the intended feature as it is mainly used for "skin templates". What we are talking about here is actually more "blueprints" for topic content inserted while it is created the first time. In this sense a TMPL:... prefix is the wrong clue.

I'd like to suggest EXPAND:... as that reveils the intention more clearly.

Use cases:

   * Set TOPICCREATOR = %EXPAND::WIKIUSERNAME%

%META:PREFERENCE{name="usd-to-euro" value="%25EXPAND:XCHANGERATES{...}%25"}!%

Adding them to a topic template will expand the macro once and add it to the new content being created.

WARNING: these snippets are expanded in the context of the user creating the new topic. This might leak information that later visitors might have no access right to. Example:

%EXPAND:SEARCH{"find some sensitive info"}%

-- MichaelDaum - 30 Jun 2016

I've flipped this back to "Under Construction". It probably should go back to under investigation due to the major change from TMPL: to EXPAND:. +1 for that change. I think it's much clearer.

-- GeorgeClark - 02 Jul 2016

This next comment is certainly off topic. I'll move it to its own proposal if it gains any legs. I think a root cause of some of the confusion is themultiple uses of "template". Michael used the word "blueprint" to describe the topic templates. I did some digging into a thesaurus. How about:

Prototype

prototype 1603, from Fr. prototype, from M.L. prototypon, from Gk. prototypon "a first or primitive form," prop. neut. sing. of prototypos "original, primitive," from protos "first" + typos "impression." Definition from thesaurus.com

If there is any support for making this a terminology change in the upcoming Foswiki 2.2 or 3.0, I'll pull this out into a separate proposal. After reviewing a lot of synonyms and definitions, I believe that a "Prototype Topic" seems to most closely match the actual use of our existing template topics.

-- GeorgeClark - 02 Jul 2016

I like "prototype" too.

However, what I've seen in other systems is content "snippets" that can be inserted using a menu in the editor. These are no full-fledged topics but useful bits to add to a page while editing it. This somewhat blends over to "prototypes".

Anyway, +1 for "prototype" instead of "template".

-- MichaelDaum - 02 Jul 2016

I'd be quite happy with a change to EXPAND is it keeps the namespaces apart (though explaining why we have "TMPL:INCLUDE" and "EXPAND:INCLUDE" might be interesting. This would probably be best done on the branch before merging.

-- Main.CrawfordCurrie - 19 Feb 2017 - 08:55

In the end I used "CREATE:" because when I was writing the doc it made so much more sense. For example, %CREATE:WIKIUSERNAME% is the name of the user who created the topic, %CREATE:DATE% is the date it was created.

I used a configure setting to disable the expansion of the normal macros. This setting can be disabled for old installs where CREATE was not available when the templates were designed. Messy, but something had to be done.

The work is on branch Item13905

-- Main.CrawfordCurrie - 21 Feb 2017 - 08:24

Commenting is disabled while not logged in