Feature Proposal: Support a cleaner syntax for quizzing meta-data

Motivation

(I could have sworn I created a feature proposal for this before, but I can't find it. For clarification, this is not a new idea. It was proposed many times by many people over the years, and something similar was even implemented as a plugin by SvenDowideit. We always intended it to be in 1.1, I just slipped up by not raising the proposal earlier)

The only ways to get the values of meta-data into a topic are rather unwieldy, especially when further processing is required. We need a single clean macro that fetches meta-data values.

Description and Documentation

QUERY -- get the value returned by a search query

• Uses the query syntax described in QuerySearch (and also used for %IF) to get information about meta-data. All the operators of %IF are supported.
• Syntax: %QUERY{ "query" [ style="perl|json" ] }%
• See QuerySearch for more details of how to write queries
• Examples:
  • %QUERY{"form.name"}% gets the name of the form in the current topic
  • %QUERY{"fields[name='Firstname'].value"}% gets the value of the 'Firstname' form field in the current topic
  • %QUERY{"Firstname"}% gets the value of the 'Firstname' form field in the current topic (shorthand version)
  • %QUERY{"'System.DocumentGraphics'/attachments[1].name"}% gets a list of all the names of attachments on the topic 'System.DocumentGraphics'

Plain strings (such as field values) are returned without quotes. Simple arrays of scalars are also returned without quotes, in a comma-separated list (though beware of values that contain commas, whcih are *not*escaped).

More complex data structures (e.g. array of hashes) will be returned as Perl code strings generated by running through CPAN:Data::Dumper.

You can also change this default behaviour to generate valid perl code strings for all values by giving the style="perl" parameter, and as JSON strings by passing style="json".

Impact

Because this proposal re-uses the query parser used in %IF statements, the risk is very low, for a great feature improvement.

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

if you want to try it, apply the attached patch against trunk. It's cool!

-- Contributors: CrawfordCurrie (and all the other people who contributed to this discussion over the years)

Discussion

CDot, very very cool. I hope with this we can finally standardise and extend ContentAccessSyntax.

For EVAL to be an effective content accessor, it should be able to provide a plain comma separated list when it comes to dumping an array.

hash/nested structures are a new problem altogether, so I appreciate the JSON which could be useful for javascript clients, but let's just keep this one simple.

-- PaulHarvey - 24 Feb 2010

I cannot see that this syntax supports getting META from other topics than current.

• There is an example above. Here's how you'd use it to get the parent of another topic (called BlahBlah) %EVAL{"'BlahBlah'/parent.name")% (nothing new in this; it's been supported in %IF for a long time)

It is only 2 days ago I tried to help someone on IRC that needed to know the parent topic of another topic. And there was no way to do that except doing a regex search on meta data which is bad for future compatibility. And not efficient. If we do add a new feature for getting META it should at least support this need.

I think the name EVAL is poor choice. What does it EVALUATE? Normally something called EVAL executes some code or matematical formula inside. It lacks the intuitive element.

• Quite happy to consider proposals of other names.

I added you to the "Concern" list

-- KennethLavrsen - 24 Feb 2010

I like the intended feature. It definitely is missing. There are plugins that allow to query the meta data already, e.g. DBCachePlugin where you query for metadata of a specific topic using its %DBQUERY{topic="..." format="$expand(...)"}% ... which by no means is as clean and flexible as the query syntax that we have in %IF and %SEARCH{type="query"}%.

So something like %EVAL makes total sense.

Some questions:

1. in which context (topic) is it evaluated...it might need a topic parameter...is it a list of topics to eval the expression each ... would be handy?
2. how is the result formatted ... maybe using our fantastic four (head, format, separator, footer)

When answering (1) and (2) in which way is %EVAL still different to %SEARCH other than being designed with a cleaner syntax?

How about calling it %QUERY ... as from what I see this is a shortcut to search-type-query plus result sets done right?

-- MichaelDaum - 24 Feb 2010

Kenneth, I embedded answers for you above.

It's evaluated in the context of the current topic. I started to add a topic parameter, but then I realised it wasn't needed because you can use the ref operator to access other topics already (see my example for Kenneth above). Remember this is not like EXPAND, the values of meta-data don't change depending on the context.

The result formatting is controlled by the style parameter; Paul is arguing for a simplification of the array presentation (currently it's shown as a perl array). I am not in favour of adding a format/header/footer/etc because of the complexity of expressing a format for values that might be scalars, arrays or hashes.

You might be able use SEARCH for some of the above examples. Let's see:

Using EVAL	Using SEARCH
%EVAL{"form.name"}%	%SEARCH{"name='%TOPIC%'" type="query" nonoise="on" format=" ... damn, no way to get the form name
%EVAL{"Firstname"}%	%SEARCH{"name='%TOPIC%'" type="query" nonoise="on" format="$formfield(Firstname)"}%
%EVAL{"'System.DocumentGraphics'/attachments[1].name"}%	%SEARCH{"name='DocumentGraphics'" web="System" type="query" noonoise="on" format=" ... damn, no way to present an array of attachment names

So, you can do some of what EVAL can do using SEARCH, albeit in a rather wordy (and in implementation terms, inefficient) way. If you cast your mind back to when we were designing the query syntax, I was being pushed to use $formfield in the queries themselves; the limiting effect is pretty obvious (if anyone can fill in the %SEARCH examples ahead with something that works, please go ahead).

%QUERY is a good name, fine by me. Any other suggestions, anyone?

-- CrawfordCurrie - 24 Feb 2010

There are two natural candidates for a macro to be evaluated (a) %TOPIC% or (b) %BASETOPIC%. Both have important use cases. So a topic parameter does make sense for lots of macros, i.e. those that extract data from topics.

Limiting it to a fixed style of output seems limiting although I agree that using the FFs makes formatting recursive structures a challenge. For instance FlexWebListPlugin adds another set of FFs, subheader/subformat/subseparator/subfooter for child nodes as there are use cases where they have to be different from the top level formatting...

-- MichaelDaum - 24 Feb 2010

No, I don't think it does. %EVAL{"'%TOPIC%'/parent.name"}% and %EVAL{"'%BASETOPIC%'/parent.name"}% should suffice.

It may be limiting, but at this stage I'm struggling to think of a use case for generic formatting of recursive data structures. If you want pretty formatting, you can always post-process with PerlPlugin.

-- CrawfordCurrie - 24 Feb 2010

Suggestion for a different macro name: GET:

%GET{"'System.DocumentGraphics'/attachments[1].name"}%

-- ArthurClemens - 24 Feb 2010

GET is a good name. QUERY is OK too. METAQUERY could be a proposal also. METAGET. And the explanation from Crawford that I can fetch from any topic (I did not realize that from the example above) is fine for me. All better than EVAL which triggers the wrong expectations in my little brain.

-- KennethLavrsen - 24 Feb 2010

@Crawford: got it.

GET is good. How about merging this proposal with SettingAndGettingVariablesUsingMacros?

-- MichaelDaum - 24 Feb 2010

GET is good for me too. I updated the proposal, and with Paul's request as well.

This proposal relates to SettingAndGettingVariablesUsingMacros only because of the nmacro name clash. We need to disambiguate between getting/setting meta-data and getting/setting preferences. Though... it occurs to me that the $ operator in %IF is supported here too, so, %GET{"$ TOPIC"}% should work.

-- CrawfordCurrie - 24 Feb 2010

We really need to sort out if the two GETs fit, and what this means for SET.

-- MichaelDaum - 24 Feb 2010

I obviously want this allot - I havn't had time to find the older proposal - it was made on the other project, and intended the same implementation.

Michael - wrt the GET in SettingAndGettingVariablesUsingMacros - what that proposes is way more complex than needed - for the same reason as this proposal does not need a format, sort or limit ....

trunk implements FOREACH, which isa format operation.

so you can already do part of:

%FOREACH{"%EVAL{'System.DocumentGraphics'/attachments}%" format="$this[1].name" separator="" sort="" page="2" pagesize="3"}%

the portion that is missing is that the format and sort params need to be extended to have a $this and $address that can be evaled.

basically - you should not need to do anything to implement GET, as %!GET{"$SOMEVAR"}% does the trick - instead, we'll need to work on adding the filters and $this to the FOREACH impl.

As I'm working thinking about (ie, no code work done yet) moving the Foreach code out of Search.pm and making it an Iterator, making an iterator pipeline using FilterIterator will make the filter / include, exclude portion pretty simple too.

-- SvenDowideit - 24 Feb 2010

We should not mix Michael's GET and this feature.

And we need to respect that Michael's proposal was passed first and also that SET/GET are natual complements.

We just seem to have all forgotten about that proposal (they are piling up too much at the moment)

So how about QUERY? Crawford liked that and noone said no.

-- KennethLavrsen - 25 Feb 2010

I'm going to switch to QUERY, aside from anything else to keep people's eye on the ball in SettingAndGettingVariablesUsingMacros

-- CrawfordCurrie - 26 Feb 2010

Given that this is a gateway to our ContentAccessSyntax, another idea did cross my mind: ACCESS. But QUERY is good too.

-- PaulHarvey - 28 Feb 2010

Michael, you still have a concern against this proposal, despite the rename; is this a reflection of the GET/SET question? Or something else? I'd really like to call a vote on this proposal, to try and get it into 1.1 (I have it implemented already).

-- CrawfordCurrie - 28 Feb 2010

QUERY is okay.

-- MichaelDaum - 27 Feb 2010

Committed under Tasks.Item8638 - I'm quite happy to revert it if it's vetoed for any reason.

-- CrawfordCurrie - 28 Feb 2010

Is there any role for formfield default values to be accessible via QUERY or a parallel macro?

MacroForRetreivingFieldDefaultValues

-- PaulHarvey - 07 Mar 2010

How about METADATA{} ?

Query confuses with SEARCH query.

-- MartinCleaver - 08 Mar 2010

I would say on the contrary Martin. METADATA can be confused with META. There is no macro called anything with QUERY* today. And on top of it the QUERY uses the same syntax as was introduced with SEARCH type=query.

I think we should stick to QUERY which is also what is checked in now. There seemed to be a broad acceptance for that.

Much more interesting is the question that has been raised. Can this QUERY feature list the default values (single or list) for a given field in a form?

It would be a natural place to have it using existing or slightly extended query syntax.

Something like %QUERY{"fields[name='Firstname'].options"}% or %QUERY{"fields[name='Firstname'].default"}%

-- KennethLavrsen - 09 Mar 2010

I agree with Kenneth that Martin's Query confuses with SEARCH query is incorrect. Specifically because the QUERY syntax is the SEARCH query syntax.

and additionally, QUERY, and SEARCH/IF are expected to cover all possible topic datums - the only reason they don't yet, is because we haven't had time (either to implement it (as for QUERY{context}), or to work out the best syntax - as for attachments).

-- SvenDowideit - 09 Mar 2010

Can this QUERY feature list the default values (single or list) for a given field in a form? - No. Default values are held in form definition topics. A form definition topic is a schema for the data that is stored in META. Only the stored data is accessible via QUERY, not the schema.

This question does raise an interesting spectre, however. Preferences can be defined two ways (* Set and META:PREFERENCE). Why not support form schemas defined two ways as well? For example,

%META:FIELDDEF{name="FavouriteSausage" type="select" size="3" value="Pork and Leek,Cumberland,Beef and Stilton" attrs="M"}%

In this case, the default values would be accessible via %QUERY{"META:FIELDDEF[name='FavouriteSausage'].value"}%. In the same way as * Set is synonymous with META:PREFERENCE, so the tabular schema definition would be synonymous with the META:FIELDDEF spec. The form table would be automatically parsed out if the topic is listed in WEBFORMS.

-- CrawfordCurrie - 09 Mar 2010

Good debate that can continue how to later extend the query syntax in general.

But the original proposal was to implement a macro that can access meta data using the query syntax we know from SEARCH and IF.

And after a discussion on the name chosen and concern rasied and lifted again - and with the 14-days passed - I declare the proposal using the QUERY macro as accepted by consensus.

-- KennethLavrsen - 09 Mar 2010

And since I already implemented it, i declare it "merged".

-- CrawfordCurrie - 10 Mar 2010