Feature Proposal: Set any addressable topic element from save

Motivation

(Also known as SetTopicValuesPlugin )

The TopicObjectModel allows us to query on addressible sub-elements of a topic. This feature adds the ability to change those values.

Initially, I want to add setting topic preferences via save urlparams, but longer term this should be expanded to reuse the IfStatements TOM addressing syntax.

• http://somewhere.com/save/Sandbox/TestTopic5?set+VIEW_TEMPLATE=SomeValue
• http://somewhere.com/save?set+Sandbox.TestTopic5/preferences[VIEW_TEMPLATE]=SomeValue
  • adds a META PREFERENCE (or potentially edits an existing in-text Setting?)
• http://somewhere.com/save?unset+Sandbox.TestTopic5/preferences[VIEW_TEMPLATE]
  • removes the META PREFERENCE (or comments out the in-text Setting)

Additionally, add texttemplate and settingstopic similar to templatetopic and formtemplate, but used to over-ride the text and META:PREFERENCES of the target topic.

• TODO: this was the original seed that started this feature and should really be moved and discussed on a separate FeatureRequest.
• Some templates already have a SETTINGSTEMPLATE html form input, but it is not actually used - this proposal would enable us to over-ride the settings in a topic.
• turns out I also had code in that old plugin to copyattachmentsfrom topic specifier - make another feature req in case its interesting to people.

Description and Documentation

Would also need to differentiate between setting the variable to and empty string, and unsetting it entirely - eg save/Web/Topic?unset=BANANANA

an alternative I'm exploring would use QuerySearch like statements, allowing (or attempting to) a single save to modify several topics

   * ?set+Sandbox.TestTopic5/preferences[VIEW_TEMPLATE]=SomeValue
   * ?set+Sandbox.TestTopic5/fields[EditDocumentState]=SomeValue
   * ?set+Sandbox.TestTopic5/fields[EditDocumentState]=SomeValue;set+Sandbox.AnotherTopic/preferences[VIEW_TEMPLATE]=SomeValue

This also means we can remove the set meta preferences from Manage and put it into Save where it belongs.

important hard parts

1. sufficient unit tests.
2. sufficient transactionality and limits
   • GET & POST
   • number of topics we can affect at one time
   • number of fields we can change
   • size of request
   • what do we do when there is a lease or save clash - do we send a datetime to attempt to merge with or warn on inter-user edits and info loss
   • do we force new rev due to possible info loss
   • do we test for ability to change (permissions, lease etc), then lock for edit, then modify all topics in memory, and then write all topics once (risk of out of memory etc)
   • do we forbid things (like changing the META:TOPICINFO) - and should this code restrict them, or should the Foswiki::Meta object protect me from foot holes.
3. responses to error issues
   • if any of permission, lease, lock, or filesystem have problems midway through a change req what do we do.
4. other scripts and chaining UI's
   • What if these parameters are set in edit, should they be passed along to preview and save?
   • its a slightly different UI from the manage based set Meta - we should merge that over to save too.

Security..

I use this feature with a GET operation so I can make records during normal user clicks - this is actually bad, but an overused foswiki idiom :/ that I should probably get out of the habit of - maybe we should contemplate making this work for POST only?

• POST only with a timestamp and a magic number would give us the beginings to insert link to the security topic
• worry about XSS - make sure it does not make things worse.
• URLENCODE!

Examples

Impact

WhatDoesItAffect:

API, Documentation, Prefs, Usability, Vars %WHATDOESITAFFECT%

edit

-- Contributors: SvenDowideit - 23 May 2008

Discussion

removed lots od discussion moved into the proposal

From Tasks:Item5701

I vaguely remember asking for clarification on this one, because I couldn't understand why on earth you would ever want it to work the way it did. To me, templatetopic means "a topic to be used as a template when creating a new topic", rather than "a means to blow away any text parameter you might pass to the save". I agree with your conclusions, though long term I might have preferred to see a mechanism by which I could template any part of a topic that is indexable under the terms of the QuerySearch syntax.

-- SvenDowideit - 25 Jun 2008

Yes adding ?Set+MYVAR=value to a save url and make it save a PREFERENCE value is pretty easy to implement. Two additions:

1. How do we delete a preference value?
2. Let's extend the url param syntax to store not only META:PREFERENCE. We could extend it to store any meta data, e.g.

   META:COMMENT{name="..." value="" author="..." date="..." ref="..." }

   That way we could easily implement a commenting system storing comments in meta data.

Other applications:

• setting access rights from within edit
• switching on/off components of the view, similar to wordpress' "show on screen" select.

Another problem that comes into play when implementing wiki applications ontop of this mechanism: while this proposal outlines how to store meta data, it does not cover how to read them in a controlled way to initialize the GUI that shows which preference values are currently set. Why is this a problem?

Well, preference values can possibly come from everywhere : site prefs, web prefs, web pref of a matrix web, user perfs, session values or the current topic prefs etc. This is in a way related to ExtendedVariableHandling. While Wolf's proposal allows to read variables from an arbitrary topic, it does not cover the use case that you want to know where this preference value actually comes from. That's important not to accidentally propagate a preference value that comes from web preferences or site preferences into each topic on every save as a consequence of a pre-initialized GUI.

For example you might design a GUI to toggle the display of the data form or the attachment list on a topic within edit. Then a preference variable SHOWDATAFORM can be set to "on", "off" or "default". Now, you need to know which of these to mark as checked and during save wether it is necessary to set or delete the preference variable with default settings coming from some other web prefs....

Was I clear?

-- MichaelDaum - 21 Jan 2009

Clear, y, i think so :). Presumably a TOM will be defined for it, and thus it'll be supportable.

Arthur also has a use case in Tasks.Item1084 for me to test

-- SvenDowideit - 15 Feb 2009

Here's some not-yet released code from BlogPlugin from its beforeSaveHandler(). Maybe that helps. It also deals with unsetting pref variables when they are set to the empty string.

  # store Set+VARNAME and Local+VARNAME urlparams
  my $request = Foswiki::Func::getCgiQuery();
  foreach my $key ($request->param()) {
    next unless $key =~ /^(Local|Set)\+(.*)$/;
    my $type = $1;
    my $name = $2;
    my $value = $request->param($key);
    if ($value) {
      $value = Foswiki::entityEncode($value);
      writeDebug("key=$key, value=$value");
      $_[4]->putKeyed('PREFERENCE', {name=>$name, title => $name, value => $value, type => $type});
    } else {
      $_[4]->remove('PREFERENCE', $name);
    }
  }

-- MichaelDaum

---

What about other forms of META? I want to be able to address META:FILEATTACHMENT etc. as well. I liked Sven's proposal for a syntax above, though I'd like a more formal specification. The Set+VARNAME syntax tends to drag away from that, into the world of one-drip-at-a-time hacks.

-- CrawfordCurrie - 24 Feb 2009

True. I'd also like to be able to set a non-standard META:COMMENT variable for a topic-meta data based commenting system. Anyway, that's code that I currently use and might be of use in this proposal...

-- MichaelDaum - 24 Feb 2009

ok, to begin with, I do not want to hamstring us with 'empty string' == remove setting, as that is not a true representation of how Foswiki works. Which is why I have an 'unset' param.

To the others, my intent is that this feature is primarily to be used as ?Set+Web.Topic/fieldtype[fieldname]=value as the proposal has always specified.

The ?Set+VARNAME syntax is only a shortcut that happens to work due to the way I wrote the defaulting, and because its consistent with Query.

Crawford, I don't understand what part of reuse the IfStatements TOM addressing syntax (I'd rather reuse the code too) is not well defined.

Michael - thanks for the code, but to be honest, I've had similar code for over 12 months now, and its 2% of the work we need - 80% is the unit tests for all the odd cases involving leases, conflicts, merges and other surprises. That is what we need to work together to make happen, cos I doubt any of us have the imagination to find all the failure cases.

The other work, is to re-work it to use the TOM node address parsing system to get the Meta field that will be changed and to ensure we only write to the writeable parts (re-writing the fieldname is probably unintended?

I just looked at the code in http://svn.nextwiki.org/trunk/RestPlugin - from 2007 - it does something excruciatingly similar to what I'm intending, but at the time I invented the : separator, which Michael and Crawford subsequently implemented in DBQuery and QuerySearch as the \ operator - yoink.

Discussion question: should we require the Set+ part at all? or should we do this as a POST only (ie change request by definition) and thus just have direct field specifiers? (and an unset selection list)

Please note that this work is proceeding in an Extension in svn.. - http://svn.nextwiki.org/trunk/SetTopicValuesPlugin - once its done we can decide if the code makes sense there or moved to core.

-- SvenDowideit - 25 Feb 2009

reuse the IfStatements TOM addressing syntax is not well defined because the query language used by %IF and %SEARCH is defined to return a boolean result (or an array of boolean results). It is not defined to return lvalues (node addresses). It probably works if you do return lvalues, I'm just not sure, and think it needs to be thought through in terms of a more formal spec that covers these points.

Here are some questions that have to be answered (not a complete list by any means)

1. If I have an expression that returns an array of lvalues, how do I assign to them? e.g. ?Set+META:FILEATTACHMENT[size>100].name?
2. Or do we always require a scalar lvalue for a Set+?
3. What happens if two Set+'s set the same lvalue? Which takes precedence?
4. If I unset all the fields in a META:TOPICINFO, what happens?
5. What happens if I set something that the system defines, and give it a bad value?
6. What happens if I unset something the system relies on (e.g. fields in TOPICINFO)?

A more future-looking question is:

1. How would the syntax extend to address components in a topic that are not currently part of the TOM e.g. tables, paragraphs, * Set statements?

These questions, and others, were not answered in RestPlugin as well. I don't have any answers yet, I just think these issues have to be thought through before diving into code. As you say, none of us has the imagination to find all the failure cases, but there are a lot we can find.

FWIW I quite like the ?Set+ part, because it is intention revealing. It also leaves other parameters (such as the standard ?topic=;username=;password=) out of the problem.

-- CrawfordCurrie - 25 Feb 2009

IMO you are wrong.

The TOM in SEARCH does NOT return a bool - Web.Topic\FIELD[Something].value is the TOM part, and importantly, that can be both an rvalue and an lvalue. Initially I am proposing that the rvalue is a constant, and the lvalue is a leaf. Best thing is I know it works, been there coded that :).

Precedence of duplicates, or worse, things that replace each other need are interesting :/ the rest of your issues, i agree - most of my approach right now is to write unit tests in the Plugin to find out what the existing codebase will allow us to do. Once I have a feel for that I'll expand on what we can discuss, vs what we don't have a choice about (without changing existing functionality)

after walking to the shops - Set+ is needed - else working our what the field is is more work than we want.

Still - I wonder if this should be restricted to POST

-- SvenDowideit - 25 Feb 2009

Been looking at the code of your new plugin, Sven. Bit shaggy still. Anyway, let's not make it too baroque. Try to create a real-world UI right away, just a simple one, and see how difficult it becomes to set and unset preference variables as well as initialize the UI reflecting the current value in existence, i.e. by a variable already set in a different non-topic context. I say that because the proposed interface might very well be extremely flexible however hard to actually use.

-- MichaelDaum - 27 Feb 2009

taste baroque has nothing to do with it. Due to the realities of our Preferences setting, we have no choice but to differentiate between setting to an empty string, and unsetting a variable. Take for example the ALLOWTOPICCHANGE setting - set to empty string means one thing, not set at all means a completely different one.

We cannot allow 'trivial to use' to force our API's to be crippled in such a way that we cannot use them.

-- SvenDowideit - 20 Mar 2009

I am trying to interpret if Michael and Crawford still have reservations about the syntax and spec.

I do not sense any opinion against the features as such. Only the syntax and spec.

So I now put Michael and Crawford in the ConcernRaisedBy fields. And when they remove their names I assume this one accepted by consensus. In fact the one that removes the last name can change the state to accepted. I just do this exercise because I am not 100% sure I interpret the signals above correctly. I have no comments about the proposal myself. If the 3 of you agree I am just fine with it.

-- KennethLavrsen - 21 Mar 2009

My concerns are: provide a real world UI (a usable html form of some sort) that uses the proposed mechanisms. Try to make it without any javascript. If that is impossible, explain why and probably think of a different way the mechanisms underneath could support a javascript-less interface.

-- MichaelDaum - 21 Mar 2009

I've talked this through with Sven, and I'm pretty sure I understand what he intends. I'm not anti- the feature, never have been. My remaining concerns relate to the lack of documentation in this topic. There are still no answers to my previous questions:

1. If I have an expression that returns an array of lvalues, how do I assign to them? e.g. ?Set+META:FILEATTACHMENT[size>100].name?
   • Initially, we require a scalar lvalue for a Set+
2. What happens if two Set+'s set the same lvalue? Which takes precedence?
   • Order - left to right - SD
   • So the query string has to be parsed? What happens if the Set statements are fields in a form? - CC
3. If I unset all the fields in a META:TOPICINFO, what happens?
   • Some things are read only. - TOPICINFO is not user settable
   • Which things are read only? For example, can I change the form? - CC
4. What happens if I set something that the system defines, and give it a bad value?
   • What happens if you do that now - no change.
   • You mean, what happens if you manually edit the topic cache and change a %META statement? Sorry, I don't consider that appropriate. This sort of interface needs much stronger protections than that - after all, it is accessible through the browser. - CC
5. What happens if I unset something the system relies on (e.g. fields in TOPICINFO)?
   • see above

A more future-looking question is:

1. How would the syntax extend to address components in a topic that are not currently part of the TOM e.g. tables, paragraphs, * Set statements?
   • Set statements are in the spec above (in fact, it seems to be the first example), and imo are in TOM now - the rest, wait until they are in TOM

-- CrawfordCurrie - 21 Mar 2009

answered inline, though really, these are not the hard questions imo - the hard one is what happens if a use sets several, and one fails. or merge issues.

Michael - ah, ok, will do - I've been using FormPlugin using my Plugin's syntax, and am quite happy with it, but yes, adding a checkbox for delete or a second button - i guess I'll make a screne.

-- SvenDowideit - 21 Mar 2009

Added some further questions above because they impinge on the TOM.

-- CrawfordCurrie - 28 Jun 2009

An alternative has been implemented in SetVariablePlugin, including some extensions necessary to make it work on real world forms, e.g. the one displayed on QueryAccessControls.

-- MichaelDaum - 28 Jun 2009

Interestingly enough, AddRequestSetToCore has added the same POST param syntax as I proposed and implemented - Micha and I look to have copied each other in 2007

So I'm going to point out that this proposal extends the now existing feature to any Query addressable element....

-- SvenDowideit - 29 Jul 2012