Feature Proposal: Setting Variables Using Macros

Motivation

The standard syntax for setting a variable is

* Set FOO = bar

However this prevents them to be set as part of a wiki application. A

%SET{"FOO" value="bar"}%

could be the natural Foswiki way to do the same.

The big difference between Set FOO and =

This macro is used to set a temporary value for the named preference. This temporary value is only set when the topic is prepared for viewing, and exists from the time the %SET macro is expanded through to the end of the current scope. This is either:

1. The end of the topic being viewed, or
2. The end of the topic in which %SET is used, when the %INCLUDE macro is used to include it.

The %SET macro doesn't generate any text; its only effect is to set the preference, which can then be used like any other preference.

Tests

%IF{"defined foo" then="FAIL" else="SUCCESS"}%

%IF{"isempty foo" then="SUCCESS" else="FAIL"}%

%SET{"foo" value="bar"}%

foo=%foo% (should be bar)

%IF{"defined foo" then="SUCCESS" else="FAIL"}%

%IF{"isempty foo" then="FAIL" else="SUCCESS"}%

%SET{"foo" value="baz"}%

foo=%foo% (should be baz)

%SET{"foo"}%

foo=%foo% (should be "empty")

%IF{"defined foo" then="SUCCESS" else="FAIL"}%

%IF{"isempty foo" then="SUCCESS" else="FAIL"}%


<verbatim>
%STARTINCLUDE%%SET{"foo2" value="bar2"}%foo2=%foo2%%STOPINCLUDE%
</verbatim>

%INCLUDE{"%TOPIC%"}% ... (should be bar2 ... local variable)

%foo2%=%foo2% ... (should be undefined)

%IF{"defined foo2" then="FAIL" else="SUCCESS"}%

%IF{"isempty foo2" then="SUCCESS" else="FAIL"}%

tests from skin templates (for eg)

%TMPL:DEF{"simpleheader"}%%SET{"foo2" value="bar2"}%foo2=%foo2%%TMPL:END%

Community Vote

This proposal has been discussed for almost 3 years now. Concern has been raised. There is no sign of a consensus being reached.

Time for a community vote again.

The vote runs for 7 days starting 10 Aug 2012 and ending 17 Aug 2012 at midnight GMT. The code to be contributed is available as a patch part of this proposal. Unit tests have been outlined as well.

The question is: Do you agree that %SET should be added to the core?

Name	Vote (Yes/No)	Reason
MichaelDaum	Yes	a simple frontend to a feature available via Foswiki::Func on perl level already, that will be of a lot of value to wiki application programmers
PadraigLennon	Yes	Much simpler approach to this common task. Have had to work-around on many occasions
GeorgeClark	Yes	However it is imperative that "fix Foswiki::Func::setPreferencesValue() not to be able to change an already finalized preference variable" is done, otherwise it opens a huge hole
LynnwoodBrown	Yes	Very useful feature for application development. I'm assuming security matters have been worked out since Macros are used in setting permissions.
MichaelLorenzen	Yes	Seems to be an useful addition
OliverKrueger	Yes	I have no concerns.
ColasNahaboo	Yes	Seems quite useful. I guess wide scale use is needed at this stage to really detect the implications anyways. I share GeorgeClark concern
CrawfordCurrie	Yes	I still feel it is geeky and not adequately explained. However a TML interface to Foswiki::Func::setPreferencesValue= does make sense.
SvenDowideit	Yes	I do want this feature, my concerns have never been addressed (I've added an overly simplistic eg above), so presumably we'll just hit them during testing / several years later
MichaelTempest	Yes	I could have used this three years ago
KennethLavrsen	Yes	More an expert user a la app dev feature than a plain end user feature so some geekiness is OK. And we can help each other making the user docu clear. So OK with me. I personally see good use of it
KipLubliner	Yes
AndreLichtsteiner	Yes	I see fantastic use for this feature. Document types (data forms) can make type specific macros available through view- and edit-templates. I just hope it gets not to messy in regard of expansion sequence!

-- Contributors: MichaelDaum - 13 Nov 2009

Discussion

After re-reading the discussion here it has wandered so far from the original proposal that I think the "accepted" status needs to be withdrawn and the proposal revisited. Set back to investigation state until the outstanding concern can be removed (it can't, from my reading of it)

-- CrawfordCurrie - 24 Feb 2012

Implementing %SET as specified is as trivial as interfacing Foswiki::Func::setPreferencesValue().

Stripped down the proposal to that.

-- MichaelDaum - 20 Apr 2012

I never thought it would be hard to implement - things like FormaliseTheProgrammingLanguage just make me wonder if its the right syntax to implement.

its pretty scary that given a few minutes hacking, I was able to implement the same result using TML only.

So - the proposal is missing the motivation - Why do we want to confuse users with preferences that look the same as the static pre-rendering extracted values, which change randomly during the render phase? (its worth answering in detail)

I'm leaning towards %SET at the moment - but don't expect me to jump for joy until i play some more with the other options.

so - in the context of Crawford's scope proposal - what scope is %SET? do we add a scope param to %SET to push up to a topic's scope (dangerous, but I thought a large part of what you wanted was a global %SET, not a local one)

-- SvenDowideit - 20 Apr 2012

The scope question is key; and it's that that makes me wonder if SET or LOCAL are the right name. Let's see, we're trying to say "here's a temporary preference that will stay active until it drifts out of scope". So maybe %TEMPORARY (or %TMP for brevity) would be more apt? Or - given that the world is driven by perl - %MY{"foo" value="bah"}%

A feature proposal like this needs user documentation, so here's my stab at the doc (moved to top of proposal)

-- CrawfordCurrie - 20 Apr 2012

%MY makes sense coming from perl. Doesn't make sense otherwise cus: why the f* is this called "my", this is not "my" variable. Fankly the "my" and "our" labels don't make sense in perl either.

The "my" meme is all over the (social) internet, like in "my contributions", "my settings". So a "%MY{"color" value="blue"}%=" might be the wrong message here as users might read this as "my color is blue" rather than "set this color to blue in the current scope".

Imho, %SET is okay as it does what it says. It even is called like this in Foswiki::Func.

Scope is an interesting - and tricky - thing. For now we only have local scope which is nice for a lot of use cases. However there are lots of useful things to have global scope for, i.e. to have sideeffects in totally decoupled places. For instance: use %SET to change things in the sidebar navigation. There are some tricky parts, most of them have to do with evaluation order in the skin itself: when is the sidebar computed and when is the topic text rendered, which could become very confusing.

-- MichaelDaum - 20 Apr 2012

The problem with SET is that it implies permanance. A * Set doesn't just apply from the position of the set onwards; it applies throughout the topic. A *Set also has different semantics in included topic. So *Set and SET are not the same, so should not share the same name. As an exercise, try to explain it to a noob on IRC.

-- CrawfordCurrie - 20 Apr 2012

I did

(16:19:58) total_noob: MichaelDaum: can you explain to me the difference between %SET and * Set please? I'm confused......
(16:25:30) MichaelDaum: total_noob, the normal * Set isn't processed as part of parsing the TML language. it is done on a separate stage _before_
(16:25:51) MichaelDaum: by reading WebPreferences, and SitePreferences and wherever they hide themselves
(16:26:14) MichaelDaum: this then forms the current context within which all TML is then evaluated
(16:26:43) MichaelDaum: now, %SET is a macro as any other %MACRO being parsed as part of the TML language
(16:27:12) total_noob: why do they have the same name, then?
(16:27:23) MichaelDaum: so there comes the big differences: the preference variables can be in the linear parsing order of any other TML macro
(16:27:52) MichaelDaum: they have the same name as they set the same variable in the same namespace of variables
(16:28:13) ***total_noob has a brain meltdown when he reads "linear parsing order"
(16:28:31) ***total_noob doesn't know what a namespace is
(16:29:12) MichaelDaum: each variable is known by its name.
(16:29:22) MichaelDaum: this name is the key to the value of the variable
(16:30:41) MichaelDaum: so I guess you see the difference now between * Set and %SET don't you?
(16:31:42) total_noob: I do, but total_noob is utterly confused; he doesn't know what "linear parsing order" and "namespace" mean.
(16:31:47) MichaelDaum: linear parsing order means: there is a deterministic order in which program code is evaluated. there is no guessing in that.
(16:32:34) MichaelDaum: all deterministic things can be linearized. thats why I said that.
(16:32:37) ***total_noob thinks that *Set should be *Constant, and %SET is the correct name for that
(16:33:25) total_noob: as in, it's not the new macro %SET that is mis-named; it's the original *Set that is wrong.
(16:33:28) MichaelDaum: total_noob, dont try to split hair. sometimes you'll be a happier noob letting all hair - that you still have - grow out of your ears.
(16:34:01) ***total_noob is careful not to sit too close to the fire, in case his ear-hair catches fire

-- MichaelDaum - 20 Apr 2012

Yup, that was a fail.

• %MY won't work - its Perl-ish
• %SET won't work - users are already confused by the word Set - and we really don't want to make another mess like we have with the word template

what we need, is something that makes the ephemeral-ness of this name-value association.

I'm presuming that %ELPH will work throughout the entire (single) request - including templates - but that for jqLoader delays, it'd have to be passed as a param

-- SvenDowideit - 22 Apr 2012

%ELPH? makes as much sense as %TROLL or %WYVERN, I suppose.

I suggested %TEMPORARY / %TMP / %TEMP above. Though I am (slowly) coming round to the view that * Set X and %SET{"X"}%%EXPORT{"X"}% should be the same thing, and thus %SET would be correct.

-- CrawfordCurrie - 22 Apr 2012

Wouldn't it be totally confusing not to call it %SET ?

-- MichaelDaum - 23 Apr 2012

Sven, can you elaborate more your concerns and what we can do to address it?

-- MichaelDaum - 26 Apr 2012

I think I've decided that I need to see a grouping of unit tests - usage of SET in weird places like tmpl files, ADDTOZONE's etc, especially for values that are set in SitePrefs.

If you get to them before I do, great.

secondly, I'm still contemplating the different implications (and confusions) of *Local, *Set META:PREFERENCE and then adding a new one (or is that two?)

%SET{"CM" value="$percntDEFAULT{default=$quotebanana$quote}$percnt cm"}%

would mean %CM% would be 'banana' and %CM{"two"}% would be 'two'

-- SvenDowideit - 26 Apr 2012

Other things I'm liking is that we can do away with the 'copy these 12 settings to your SitePrefs ' for skin themes - we just make the topic a viewTemplate and provide the functionality / skin settings via an EXPORT

So this feature will allow us to convert skins into WikiApps, and begins to merge skin and app tmpl's

on the other hand, I'm not overjoyed about EXPORT - throwing more values into the GLOBAL namespace means we will get app developers clashing and un-debuggably over-writing other's values. (At least in the *Set and META:PREF case, it is possible to SEARCH / grep for where the value is set.

(as an eg, imagine several people build wikiapps that %EXPORT{"AGE, and then someone shows all of them in a dashboard page - the globally exported values will clash, and everyone has to needlessly re-write.)

Instead, I think we need to become more structured in how we intend to RETURN name-value pairs, and allow the caller to choose new names

As far as I can see, EXPORT=/=RETURN only has meaning for transclusion - INCLUDE/SEARCH/QUERY/**** do we have a canonical list of transclusion operators?,

%SET{"AGE" value="21"}%
%SET{"CM" value="$percntSET{$quotHEIGHT$quot value=$quot$percntDEFAULT$percnt$quot}$percnt$percntDEFAULT$percnt cm"}%

%STARTSECTION{"age_info"}%
At age %SET{"YOUNG" value="13"}%%YOUNG%, my height was %CM{"165"}%. At age %AGE% it is now %CM{"185"}%
%RETURN{"YOUNG"}%%RETURN{"HEIGHT"}%
%ENDSECTION{"age_info"}%

%INCLUDE{"%TOPIC" section="" import="HEIGHT,RETURNED_AGE=YOUNG"}%

   * %AGE% is 21
   * %RETURNED_AGE% is 13
   * %YOUNG% is undef
   * %HEIGHT% is 185

(as you can see import matches better with EXPORT - so )

• additionally - I'd rather we implement CM as a SECTION to avoid the escaping hell
  • %DEFAULT% cm

So as you can see, one of my concerns is what your cartoon references. Another is that I'm not sure what happens to SET's and EXPORT's that happen in tmpl files and Template Topics.

-- SvenDowideit - 27 Apr 2012

You mean something like this?

%SET{"foo" section="bar"}%

%STARTSECTION{name="bar" type="section"}% a complex value being computed here %ENDSECTION{name="bar" type="section"}%

Could be done using HERE documents (never expedrimented with them / did we actually implement them?)

And yes, neither am I overexcited with %EXPORT. The alternative would be to flag scope as part of the %SET macro directly.

-- MichaelDaum - 27 Apr 2012

Michael - are you going to do some work on my concern wrt the global-ness, or will this wait until I do? (I'm distracted atm)

-- SvenDowideit - 10 May 2012

As an total_noob , really the original Set should be "Setenv" and the %SET would be ok.

Unfortunately legacy. So, now: what about the %SETVAR{"foo", value="bar"}%? and its counterpart %GETVAR{foo}%? And would be helful in the SpreadsheetPlugin make $SETVAR / $GETVAR too and marking $SET as deprecated in few years..

-- JozefMojzis - 10 May 2012

@Jozef: %SETVAR and %GETVAR would clash with SetVariablePlugin

@Sven, let's chat about it as soon as you are available again.

-- MichaelDaum - 10 May 2012

oh sht... so, need something other, but, don't call it SET. Its confusing. Ideas: ASSIGN, LET

-- JozefMojzis - 10 May 2012

Try to answer this: "Why the f* isn't it called the same?"

There definitely is a relation to the * Set way of setting variables / constants: this is storing values into the same namespace with variable names competing with each other in terms of priority.

That's why I think that we should have very very good reasons normal users understand not to call it %SET.

As I already said above already: anything else would rather be more confusing than helping for users lining out the differences between * Set and %SET, that definitely exist.

-- MichaelDaum - 11 May 2012

Set and Local - and both might be useful - SET being the =LOCAL=&Export version.

Michael - what I need is unit tests that show how the current implementation idea reacts in the plausibly difficult places - tmpl files, and weirdly nested situations - and how they then interact with FINALISED / ACL things.

The worst thing is that your cartoon is exactly what i'm concerned about. using %SET (with export) enables users to randomly change Global settings to random things - whereas at the moment, preferences in a context are immutable.

-- SvenDowideit - 02 Jul 2012

Finalize is a good point. It actually might be a hint that Foswiki::Func::setPreferencesValue shouldn't be allowed to change a finalized value on perl level either, and as a consequence nor would %SET being a pure interface to Foswiki::Func::setPreferencesValue.

Sven, which weirdness do you expect in %TMPL + %SET mixups? I can't think of any abnormal case. If you have a good idea of what this might be, I'd be able to carve that into a unit test.

Changing global values is already possible using the other means of adding stuff to the prefs stack, e.g. in parametrized %INCLUDEs., isnt it. Everything happening inside that %INCLUDE is in danger of suffering from strange globals.

-- MichaelDaum - 02 Jul 2012

I didn't say I expected weirdness, I expect unit tests that show what happens in places were we are not (yet) expecting SET to be used.

no, parametrised INCLUDEs only flow on to SECTIONs that are called from within the preceeding INCLUDE.

this SET proposal proposes to do something more complicated that we have before - to allow adding and changing preferences values in the middle of a section

And then it talks about an EXPORT macro that changes the preference tree in a completely new way, in that it allows affecting the calling section, again, in the middle of the rendering of that section.

the core thus far has shied away from this kind of thing - we've been treating preferences as immutable within their context.

so the long & short is, write unit tests, see what the code does before and after the proposed change - I might be concerned about nothing, but the unit tests need to be written for the feature either way.

-- SvenDowideit - 03 Jul 2012

Okay, no problem. The unit tests will be pretty straight forward juggling a few constellations of reading and writing variables using %SET. Behavior will be guided by standard evaluation orders.

I won't implement an EXPORT macro. That's the idea of somebody else, not me, can't even remember who proposed that. So it will require a feature request of its own.

What I will implement is given in the #Implementation section above, plus some unit tests hammering it. See the #Tests above for some.

-- MichaelDaum - 03 Jul 2012

Sven, what else can I do to address your concerns?

-- MichaelDaum - 04 Jul 2012

So what I need, is to see how the adhoc evaluation changes the precedence of, and how non-topic evaluations can affect preferences.

You've not updated the docco to reflect what you're going to do, and how this will change preferences from 'evaluated before rendering' (essentially when a topic is loaded), and what happens when %SET is used in a template - both tmpl and topic Template.

I also wonder what fun things will be possible using SET in a topic creation or form definition topic

My concern is that the proposal does not cover these complicated things which may well require changing the proposal.

-- SvenDowideit - 09 Jul 2012

What is a non-topic evaluation? Whatever it is, why is it important? All %SET does is call Foswiki::Func::setPreferencesValue(). These values are set during parsing and thus take precedence over values set anywhere else. That's basically all.

Sorry, but I don't get what you mean, Sven.

The only issue I see is expressed by my #finalizeQuestion above that nobody seems to have an opinion on.

-- MichaelDaum - 09 Jul 2012

Maybe that's because immutual is somewhat difficult to understand. Does it mean "not mutual" or "distinct"? In any case I'm not nearly qualified enough to have an opinion on this matter. From the users point of view I'd like to be able to set/get variable values differently in applications via macro calls.

-- FranzJosefGigler - 09 Jul 2012

Basically, "finalize" means the variable can't be changed anymore. As things are right now, plugins still can change values of finalized preferences under the hood using a standard call to Foswiki::Func::setPreferencesValue() ... which %SET would be a frontend to.

There are two options to resolve this issue:

1. fix Foswiki::Func::setPreferencesValue() not to be able to change an already finalized preference variable
2. let %SET do the necessary things to respect finalization. this might duplicate the same logic done elswhere for plain * Set.

I'd personally strongly opt for (1). It shows that there actually is a deeper problem in the concept of finalization as the logic doesn't seem to be implemented at a point in the code path where it belongs to.

-- MichaelDaum - 09 Jul 2012

I removed the %EXPORT macro from the documentation. I was never intending to implement it and the person who added it is not committed to implement it, as far as I see.

To clarify: this proposal is only about implementing %SET as provided in the patch above. Any further extension to this needs a feature proposal of its own, and a different CommittedDeveloper, i.e. not me.

Note also that this rather simple and pretty much self contained feature proposal (implementation and tests provided) will soon become 3 (three) years old constantly being vetoed against it for unclear reasons, at least to me. I have steam to leave it in discussion mode for say another month before I will step back from this initiative and park this proposal.

-- MichaelDaum - 22 Jul 2012

12 days left before I wil step back from this work and park the proposal. Sven, you are still blocking this even though I addressed all of your concerns as far as I see.

-- MichaelDaum - 10 Aug 2012

Michael, why not flip this to CommunityVote? And above the word immutual is used. I suspect it meant to be immutable?

-- GeorgeClark - 10 Aug 2012

The CommunityVote has been set up.

-- MichaelDaum - 10 Aug 2012

How do we vote?

-- RuiProcopio - 10 Aug 2012

@RuiProcopio: Add your vote in a new line into the voting table above. -- MichaelLorenzen - 10 Aug 2012

My concern is basically that the effects of using SET in places like tmpl has not been addressed. I asked for unit tests to show what the currently proposed implementation would do so we could discuss those facts, but you have not done so.

Clearly, no one else cares about this.

So vote away.

-- SvenDowideit - 11 Aug 2012

I do not envision the security implications. But my gut feeling is that we will only find them by using this feature "in the real world", beta testing is not sufficient for this. Maybe the best approach would be to implement this feature with 2 safegards: - detailed logging to trace how it is used - a simple to activate "kill switch" to disable it in case a hole is discovered. (temporarily until a proper fix is designed)

-- ColasNahaboo - 11 Aug 2012

Hey. New guy here. As a Software Engineer, I fully understand the concepts of scope and order of operations. For those who have not read http://shop.oreilly.com/product/9781593274245.do I can see how those concepts can be confusing.

As a Board member of several mostly-volunteer non-profits, and sometimes Manager at work, I would really like to see a separation between the conversation about "will this confuse the user" and "is it a good idea". They are very different conversations that need very different processes. "Will this confuse the user" mostly about finding a differentiating name, like "setLocal" or "setInScope", but it's not a technology problem. You know in your heart that someone will come up with the right word and description.

"Is it a good idea" and "how should it be implemented so it doesn't break stuff" are technology questions which, as SvenDowideit said, is best proposed, tested, and discussed over a nice hot bowl of unit tests, where hypotheses on how the sky will fall can be factually disproved.

I feel mixing the two conversations is making consensus harder. I'm hoping here to help the decision process.

-- DavidKramer - 12 Aug 2012

Having said the above, I will squirt just a bit of lighter fluid on the fire How will this work differently than $percentCALC{"$SET(my_total, $SUM($ABOVE()))"}$percent from SpreadSheetPlugin? On the one hand, if it does the same thing, maybe this new feature is not needed, because SpreadSheetPlugin can be used. On the other hand, maybe it can be used to test those aforementioned hypotheses with no coding at all.

-- DavidKramer - 12 Aug 2012

AFAIK are macros set by SpreadSheetPlugin only available to SpreadSheetPlugin via %CALC("$GET ...}% and therefore not as 'valuable' as the here proposed %SET{ ... }%.

-- AndreLichtsteiner - 13 Aug 2012

I would propose some tests along the following idea:

1. %SET("FINALPREFERENCES")%or %SET("DENYTOPICVIEW")% in a template
2. add the template to the skin by ?cover=myskin
3. see whether Foswiki honours all ACL and final preferences as it supposed to.

Along above test, I would be interested, what would be the sequence, in which the Macros get set? Topic before Template?

-- AndreLichtsteiner - 13 Aug 2012

(1) Yes, it should not be possible to %SET some of these special variables.

(3) %SET must not allow to alter variables listed in FINALPREFERENCES. Thats probably a flaw of the underlying perl API already as has been noticed in the above discussion.

I am not sure in how far (2) leads to any test of a %SET. Templates are expanded well before anything else with none of the macros being expanded at all, including %SET. So variables can't affect template expansion that way.

-- MichaelDaum - 17 Aug 2012

Commenting is disabled while not logged in