Feature Proposal: Continue extending the canonical form of the SCRIPT / PUB URL macros

Motivation

Currently it cannot easily handle the REST script. And at the same time, it duplicates the web= and topic= parameters into the path.

Actually it's been pointed out that the canonicalization of SCRIPTURL and PUBURL macros was "slipped in" So this is a bit retroactive to some changes that made it in through the unicode revisions.

Description and Documentation

These changes apply to: VarSCRIPTURL, VarSCRIPTURLPATH, VarPUBURL, and VarPUBURLPATH

• With current usage, the web and topic parameters should be used to build the URL Path, and be eliminated from the query parameters. See Tasks.Item13795.
• Since rest requires a non-standard path, add path="subject/verb", or consider separate subject= and verb= options with similar syntax to the web and topic parameters.

Note, as mentioned in Item13795,

The enhancements to %SCRIPTURL has got nothing to do with adding web to CGI scripts.

See the discussions at DeprecateContextlessURLConstructs and EnableCloudStorageForAttachments. None of them are accepted.

I've started this new because the other two proposals don't really align with what was actually implemented.

Examples

...{"script"}	Generate conventional script URL compatible with current "concatenation" of the URL
...{"view" web="web" topic="topic"}	Generate ...view/Web/Topic changed: omit web= and topic= query params
...{"rest" web="web" topic="topic"}	error - rest will require the subject & verb or path parameters
...{"rest" path="CommentPlugin/comment" web=web ...}	Generate ...rest/Subject/Verb?web;topic="
...{"rest" subject="CommentPlugin" verb="comment" web=web ...}	Alternative syntax for rest

Impact

The intention is to support full canonicalization of macros, so any random parm="value" will be mapped into the query parameters. This potentially could result in collisions from existing parameters. web= and topic= are well established reserved parameters. But new parameters like verb, subject, path, ... could collide with existing scripts / URLPARAM macros. Given that, we might want to chose more unique parameters. restVerb, restSubject vs simple verb / subject. scriptPath vs. path. urlAnchor="anchorstring".

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: GeorgeClark - 04 Feb 2016

Discussion

rest needs a topic url param as otherwise the core and a couple of plugins don't know where the action is taking place. This is a known design flaw in the rest interface, meaning a call like this

%SCRIPTURL{"rest" subject="SomePlugin" verb="action"}%

results in an internal problem

$session->{webName} = "SomePlugin";
$session->{topicName} = "action";

This location does not exist of course and the system is in an unknown state.

Therefore,

%SCRIPTURL{"rest" web="web" topic="topic" subject="SomePlugin" verb="action"}%

should result in

http://.../rest/SomePlugin/action?topic=web.topic

When no web and topic are specified they should default to web=%USERSWEB%%, topic=%HOMETOPIC%. That

%SCRIPTURL{"rest" subject="SomePlugin" verb="action"}%

should result in

http://.../rest/SomePlugin/action?topic=%USERSWEB%.%HOMETOPIC%

thus mitigating the rest design flaw.

Minor remark on web and topic. The latter should suffice, i.e. writing topic="web.topic" is just fine. When both are provided normalizeWebTopicName() should be applied, meaning when topic specifies a web it takes precedence over web

%SCRIPTURL{"view" web="web1" topic="web2.topic"}%

expands to

http://.../view/web2/topic

To prevent any confusion the web parameter should be removed from the API ... if not already too late.

-- MichaelDaum - 05 Feb 2016

+1 to Michael's comments.I'm making no excuses for the canonicalisation of PUBURL. I think we adopted the web param as a standard many years ago, so it probably is too late.

-- Main.CrawfordCurrie - 05 Feb 2016 - 09:32

Another important point missing is url parameters. Right now, these are simply concatenated to whatever %SCRIPTURL and friends return. However the main reason to make them proper macros is to provide context to their implementation, i.e. which web and topic they are to. Alas, we forgot about url parameters being part of the game. Let me explain.

The deeper idea behind is that in the future some of those links might return quite different results based on the store used for the topic or attachment (DeprecateContextlessURLConstructs, EnableCloudStorageForAttachments). For instance imagine some %PUBURL expanding to a link to another system running a document management server:

%PUBURL{topic="Foo.Bar" attachment="somemovie.mp4"}%

Expands to

http://some.steaming.server?container=foo&location=Foo.Bar&user=wikiuser

Problem is that %PUBURL already creates url parameter, however the markup tries to append yet another url parameter such as in

%PUBURL{topic="Foo.Bar" attachment="somemovie.mp4"}%?t=12345

... and so would create an invalid uri all together.

http://some.steaming.server?container=foo&location=Foo.Bar&user=wikiuser?t=12345

I'd therefore recommend to extend the original proposal accordingly by converting all other "unknown" params of the macro to url parameters.

Example

%PUBURL{topic="Foo.Bar" attachment="somemovie.mp4" hello="world" t="123"}%

expands to

http://.../pub/Foo/Bar/somemovie.mp4?hello=world&t=123

Only then would have enough of a context to plug in a different implementation pointing to some DMS with a totally different uri schema and return:

http://some.steaming.server?container=foo&location=Foo.Bar&user=wikiuser&hello=world&t=123

-- MichaelDaum - 10 Feb 2016

SCRIPTURL macro already works this way. Unknown parameters are automatically added to the query string. However there are a couple of enhancements we could consider:

• Configurable use of ; or & as the query param separator
• Allow # as a parameter, so that anchor can be specified. ("anchor" could conflict with other parameters, and technically it's a fragment identifier, not the anchor.) However this would be a change to the macro parser.

-- GeorgeClark - 22 Feb 2016

Also, for SCRIPTURL, what's the preferred way to determine the construction of the URL. For ex. rest needs subject/verb, jsonrpc needs Namespace. Initially above I proposed adding subject= and verb= parameters, however that doesn't apply to jsonrpc, and would not necessarily apply to other non-standard scripts. Other possible solutions:

Add path="..." for use with rest or jsonrpc

Example %SCRIPTURL{"rest" path="subject/verb" topic="...}% or %SCRIPTURL{"jsonrpc" path="MyNameSpace"}%. Possible issue, conflicts with query parameter name of "path".

Embed the path into the script name.

Ex %SCRIPTURL{"rest/CommentPlugin/comment" topic= ...}% Macro processor would have to split at the first /, to insert any script suffix. For other standard scripts, the web/topic would automatically be added to the path and omitted from the query string.

-- GeorgeClark - 22 Feb 2016

I wonder if there is some intersection between this proposal and MoveQueryPathParsingIntoFoswikiRequest. There are pieces that are complementary operations:

• Construction: Assembling $web and $topic or other parameters into action/Some/Path
• Deconstruction: Parsing action/Some/Path into $web and $topic

If this is true, maybe the type of request Foswiki::Request::<type> should also have a "construction" method which reverses the parse operation and takes a collection of macro parameters and assembles them into the appropriate URL.

-- GeorgeClark - 20 Mar 2016

+1 to Michael's comments.I'm making no excuses for the canonicalisation of PUBURL. I think we adopted the web param as a standard many years ago, so it probably is too late.

Actually there is no web= query param. At least it is not documented. And core does not use it. The parameter is "defaultweb" which provides a web only if topic= does not provide a web, AND the path doesn't have a web.

-- GeorgeClark - 05 May 2016

I wanted to finish this off for 2.0. There are two design issues that I need to resolve.

1. Handling of rest / jsonrpc. I plan to add urlpath to support the subject/verb or Namespace/action for json and rest.
2. Should subject/verb/namespace/method also be supported for json / rest requests
3. Just discovered another one. The print button provides a complete QUERYSTRING so that the print view has all the same parameters as the current view. My thought there is to add a querystring parameter for passing arbitrary formed query strings to be appended to the expanded macro. (duplicates need to be filtered though.)

Revised SCRIPTURL and SCRIPTURLPATH documentation:

Parameters

Parameter	Description	Default
"$script"	Name of script
web	Web name to add to URL
topic	Topic (or Web.Topic) to add to URL
#	Specify the anchor string (New in 2.2)
urlpath	Arbitrary string appended to the script. Intended for jsonrpc and rest requests that do not use Web/Topic in the URL. (New in 2.2)
querystring	Arbitrary string appended to the query. Intended to pass the %QUERYSTRING% macro. Leading ? or ; will be provided as required (New in 2.2)
Alternative specification for jsonrpc/rest
subject	Alternative to urlpath for rest
verb	Alternative to urlpath for rest
namespace	jsonrpc Name Space
method	(optional) jsonrpc method if not provded in the post request
Any other parameters to the macro will be added as parameters to the URL

Examples

• %SCRIPTURLPATH{"view" topic="Cartoons.EvilMonkey"}% expands to /Cartoons/EvilMonkey
• %SCRIPTURLPATH{"view" topic="Cartoons.EvilMonkey" #="MyAnchor"}% expands to /Cartoons/EvilMonkey#MyAnchor
• %SCRIPTURLPATH{"view" web="Cartoons"}% expands to /Cartoons?web=Cartoons
• %SCRIPTURLPATH{"view" topic="Cartoons.EvilMonkey" rev="1"}% will expand to /Cartoons/EvilMonkey?rev=1
• %SCRIPTURLPATH{"edit" web="Cartoons" topic="EvilMonkey" t="%GMTIME{"$epoch"}%"}% expands to /bin/edit/Cartoons/EvilMonkey?t=1733012710
• %SCRIPTURLPATH% expands to /bin
• %SCRIPTURLPATH{script}% expands to /bin/script
• %SCRIPTURLPATH{"rest" subject="CommentPugin" verb="comment" topic="Cartoons.EvilMonkey"}% expands to rest/CommentPlugin/comment?topic=Cartoons.EvilMonkey
• %SCRIPTURLPATH{"rest" urlpath="CommentPugin/comment" topic="Cartoons.EvilMonkey"}% expands to rest/CommentPlugin/comment?topic=Cartoons.EvilMonkey (Alternative syntax)
• %SCRIPTURLPATH{"jsonrpc" namespace="Configure" }% expands to =jsonrpc/Configure
• =%SCRIPTURLPATH{"view" querystring="%QUERYSTRING%"}% expands to view/Cartoons.EvilMonkey?%QUERYSTRING%

-- GeorgeClark - 10 Mar 2017

Maybe I'm just being thick, but I don't understand the problem with subject and verb. Only rest and jsonrpc need them. Isn't a defined precedence order sufficient to resolve any potential conflicts? For example,

"The rest script should always be called with subject and verb parameters. However for compatibility with previous releases, If (and only if) these parameters are not given, Foswiki will fall back to interpreting the topic parameter as subject/verb. However this usage is deprecated and you are strongly recommended to always call rest with subject and verb parameters."

I think that provides compatibility for rest, and I think jsonrpc should be compatible. I really don't like urlpath, it feels like a fudge.

• %SCRIPTURLPATH{"rest" subject="CommentPugin" verb="comment" topic="Cartoons.EvilMonkey"}% expands to rest/CommentPlugin/comment?topic=Cartoons.EvilMonkey
• %SCRIPTURLPATH{"rest" topic="CommentPlugin.comment"}% expands to rest/CommentPlugin/comment works but generates a warning

It might be preferred to adopt the jsonrpc namespace and method parameters for rest, and deprecate subject and verb. Either way, the goal should be to develop consistent signatures for rest and jsonrpc

"The rest script should always be called with namespace and method parameters. However for compatibility with previous releases, If (and only if) these parameters are not given, Foswiki will fall back to interpreting the topic parameter as namespace/method. However this usage is deprecated and you are strongly recommended to always call rest with namespace and method parameters. For compatibility with previous releases, the rest script also supports subject and verb parameters, which are synonymous with namespace and method, respectively."

• %SCRIPTURLPATH{"rest" namespace="CommentPugin" method="comment" topic="Cartoons.EvilMonkey"}% expands to rest/CommentPlugin/comment?topic=Cartoons.EvilMonkey
• %SCRIPTURLPATH{"rest" subject="CommentPugin" verb="comment" topic="Cartoons.EvilMonkey"}% expands to rest/CommentPlugin/comment?topic=Cartoons.EvilMonkey works but generates a warning
• %SCRIPTURLPATH{"rest" subject="CommentPugin" method="comment" topic="Cartoons.EvilMonkey"}% expands to rest/CommentPlugin/comment?topic=Cartoons.EvilMonkey works but generates a warning
• %SCRIPTURLPATH{"rest" topic="CommentPlugin.comment"}% expands to rest/CommentPlugin/comment works but generates a warning
• %SCRIPTURLPATH{"rest" subject="CommentPlugin" topic="Cartoons/RogerRabbit"}% does not work - "no method given"
• %SCRIPTURLPATH{"rest" subject="CommentPlugin" method="comment" topic="Cartoons/RogerRabbit"}% does not work - "no namespace given"
• %SCRIPTURLPATH{"rest" namespace="CommentPlugin" verb="comment" topic="Cartoons/RogerRabbit"}% does not work - "no method given"
• %SCRIPTURLPATH{"rest" namespace="CommentPlugin" subject="CommentPlugin" topic="Cartoons/RogerRabbit"}% does not work - "conflicting parameters subject and =namespace="
• %SCRIPTURLPATH{"rest" namespace="CommentPlugin" method="comment"}% defaults the topic to "Main/WebHome"

Obviously this validation is quite complex and script-dependent. It would be worth considering an extension to tag handler registration to support the declaration of a validator callback, as this is a broader requirement than just rest and jsonrpc.

-- Main.CrawfordCurrie - 11 Mar 2017 - 09:22

I'm not worried about backwards compatibility with macros, but with the much larger issue of compatibility with any urlparams used by any arbitrary wiki applications. And that's a huge unknown. If an old wiki application uses the now newly reserved variables are they forced to change?

For example, but just guessing here: An Email application which uses URLPARAM of "subject" in a mailto form processed by a rest handler? I have no idea if such a situation exists, but subject, verb, method and namespace seem to be such common words, that collisions might really exist out in the wild.

Maybe I'm worrying about nothing. I suppose that if such a collision exists, then the solution is to continue to use "concatenation" in building the URLs for that particular application. And as this is a new feature, that's probably what's in use anyway. I'm convinced. urlpath is out.

The things I don't like out of your proposal are:

1. the overloading of topic= to provide the subject & verb. That's going to confuse the core parser which asserts the topic name must begin with upper case.
2. The mixing of subject/verb, namespace/method or topic in the URL path. I really think for URL "path" construction:
   • subject/verb should only processed into the path for rest, and are just arbitrary query params otherwise.
   • namespace/method - Similarly only processed into path for jsonrpc (and method is optional as it is typically supplied in the json structure.)

Trying to have all four params active for both jsonrpc and rest just going to cause confusion and will add to the possibility of collisions with other url parameters. I suppose it could be a way to work around a collision, but then if both subject and namespace are provided, as you point out it's a conflict. So there is no benefit there.

For now I'm not going to add a tag validator callback. None of these are "registered" tags, and in addition, the processing is also needed in the core Foswiki::getScriptURL call. So that just expands the scope and can't really be used for this situation.

-- GeorgeClark - 11 Mar 2017

Ugh. Foswiki::getScriptUrl is going to be difficult to extend. Calling convention is: getScriptUrl( $absolute, $script, $web, $topic, @params ) -> $scriptURL with optional parameters passed in an array.

I'm leaning towards making a minor change in the "definitions" of the calling conventions: getScriptUrl( $absolute, $path1, $path2, $topic, @params ) -> $scriptURL, where the Web and Topic are replaced with two path components. Here are some examples:

• getScriptUrl( 0,'rest', 'CommentPlugin', 'comment', 'topic', 'SomeTopic' ...)
• getScriptUrl( 0,'jsonrpc', 'Configure', undef, 'topic', 'SomeTopic' ...)
• getScriptUrl( 0,'jsonrpc', 'MyNamespace', 'weeble' )

It would be up to the SCRIPTURL macro to properly deal with the mapping of Subject/verb, Namespace/method to the correct positional parameters.

Foswiki::Func would have the same definition change.

• getScriptUrl( $path1, $path2, $script, ... ) -> $url
• getScriptUrl( 'CommentPlugin', 'comment', 'rest' ...)
• getScriptUrl('NameSpace', undef, 'jsonrpc', ... )

It would be the responsibility of the caller to ensure that the topic=, and other parameters were supplied as necessary.

-- Main.GeorgeClark - 13 Mar 2017

Sorry George, I was confusing things, sorry, I did a bit more reading. The documentation of jsonrpc states that you can pass namespace/method in any of three ways:

1. Using namespace and method params in the JSON payload (NOT in the URL!)
2. Using namespace in the URL path and method in the JSON payload
3. Using namespace/method in the URL path

The only way you can call a jsonrpc using %SCRIPTURL% is to specify the namespace/method in the URL path, e.g.

The jsonrpc SCRIPTURL signature therefore looks pretty much identical to the rest URL signature. So I agree with your proposed change above - that's all that's needed. However your checkin suggests that jsonrpc responds to URL parameters, which is wrong - it only accepts parameters in the JSON payload.

-- Main.CrawfordCurrie - 14 Mar 2017 - 07:51

Note that these changes cannot be made to the default extensions, since they need to be compatible with older Foswiki releases.

-- Main.GeorgeClark - 05 Dec 2017

Commenting is disabled while not logged in