Formalise a specification for a macro's arguments, among other things

Prompted by WIBNIF thoughts over the last 6 months and then seeing that XWiki seems to have already very successfully made TinyMCE such a nice experience for its users, we need to do something about this.

Goals

• (semi?) automatic generation of test coverage for registered macros
• BuildContrib could be modified to replace an occurance of %$ARGUMENTSPECIFICATION% in the system topic to replace it with documentation automatically compiled from the spec. Some of this could leverage substantial existing work in CPAN, but I'm not able to judge which libs would be the best choices.
• Both of the above items require some formal way of expressing permissible values for each parameter: enumerate possible values; list ranges; detail any complex logic/inter-dependency with other args. This is not just about a standard way of doing free-text descriptions for macro parameters.

• WYSIWYG editor might be able to do more than just display bareword macro. Actually the possibilities are fun to think about:
  • At a minimum, should be able to automatically provide useful autocomplete or context sensitive help/doc links/dropdowns similar to common programming IDEs
  • Desirable (ordered roughly by feasibility):
    • Display a list of available (documented) macros and variables to choose from.
    • %INCLUDE{...}% on mouseover might offer an edit link to the included topic, or a button to expand out with the included topic as a read-only div
    • %STARTSECTION{"foo"}% on mouseover might offer a link to jump to the matching %ENDSECTION%
    • BibliographyPlugin might want to auto-complete on %CITE{"ref"}%, not just a drop-down of what refs are already defined but also what they look like, and provide a quick-link to edit the bibligraphy table or automagically add to it
    • %TOC% might want to render the TOC as it looked in the current topic version and also become a complex div/ajax object with user friendly dialogues instead of macro arguments
    • ImagePlugin might want to be a complex ajax div/object in the editor instead of (as well as?) just presenting itself as %IMAGE{...}%, on which somebody can interact with it via useful dialogues
    • %SEARCH{...}% might offer a nice real-time interactive ajax gui which renders either as a plain list of matching topics when editing the search expression or a preview of the formatted output when editing the format strings

See also WysiwygEditorAndMacros

-- PaulHarvey - 03 Jan 2010

after working on a demo javascript editing tool in 2005-6 called InlineEditPlugin, I also worked on a popup wizard editor for macros - who's code was extracted as ComponentEditPlugin. In this, I built a perl hash of Macro, parameter types and documentation which was later pulled in part into the lightweight tags proposal, and another proposal who's name escapes me.

as an example of this quick demo - svn contains a partial definition of the SEARCH macro at the time:

my %syntax = ( SEARCH => {
      DOCUMENTATION => {type=>'DOCCO', DOCCO=>'Inline search, shows a search result embedded in a topic'},
      search => {type=>'text', defaultparameter=>1, default=>'', DOCCO=>'Search term. Is a keyword search, literal search or regular expression search, depending on the type parameter. SearchHelp has more'},
      web => {type=>'text', default=>'', DOCCO=>'Comma-separated list of webs to search. The special word all means all webs that doe not have the NOSEARCHALL variable set to on in their WebPreferences. You can specifically exclude webs from an all search using a minus sign - for example, web="all,-Secretweb".'},
      topic => {type=>'text', default=>'', DOCCO=>'Limit search to topics: A topic, a topic with asterisk wildcards, or a list of topics separated by comma.'},
      excludetopic => {type=>'text', default=>'', DOCCO=>'Exclude topics from search: A topic, a topic with asterisk wildcards, or a list of topics separated by comma.'},
        header => {type=>'text', default=>'', DOCCO=>'Custom format results: see FormattedSearch for usage, variables & examples'},
        format => {type=>'text', default=>'', DOCCO=>'Expand variables before applying a FormattedSearch on a search hit. Useful to show the expanded text, e.g. to show the result of a SpreadSheetPlugin %CALC{}% instead of the formula'},
        seperator => {type=>'text', default=>'', DOCCO=>'Line separator between hits'},
        type => {type=>'options', option=> ['keyword', 'literal', 'regex'], default=>'', DOCCO=>'Do a keyword search like soap "web service" -shampoo; a literal search like web service; or RegularExpression search like soap;web service;!shampoo'},
      scope => {type=>'options', option=> ['topic', 'text', 'all'], default=>'text', DOCCO=>'Search topic name (title); the text (body) of topic; or all (both)'},
      order => {type=>'text', default=>'', DOCCO=>'Sort the results of search by the topic names, topic creation time, last modified time, last editor, or named field of DataForms. The sorting is done web by web; if you want to sort across webs, create a formatted table and sort it with TablePlugin\'s initsort. Note that dates are sorted most recent date last (i.e at the bottom of the table).'},
      limit => {type=>'text', default=>'', DOCCO=>'Limit the number of results returned. This is done after sorting if order is specified'},
      date => {type=>'text', default=>'', DOCCO=>'limits the results to those pages with latest edit time in the given TimeInterval.'},
      reverse => {type=>'onoff', default=>'off', DOCCO=>'Reverse the direction of the search'},
      casesensitive => {type=>'onoff', default=>'off', DOCCO=>'Case sensitive search'},
      bookview => {type=>'onoff', default=>'off', DOCCO=>'show complete topic text'},
      nosummary => {type=>'onoff', default=>'off', DOCCO=>'Show topic title only'},
      nosearch => {type=>'onoff', default=>'off', DOCCO=>'Suppress search string'},
      noheader => {type=>'onoff', default=>'off', DOCCO=>'Suppress search header '},
      nototal => {type=>'onoff', default=>'off', DOCCO=>'Do not show number of topics found'},
      zeroresults => {type=>'onoff', default=>'off', DOCCO=>'Suppress all output if there are no hits'},
      noempty => {type=>'onoff', default=>'off', DOCCO=>'Suppress results for webs that have no hits.'},
      expandvariables => {type=>'onoff', default=>'off', DOCCO=>'Expand variables before applying a FormattedSearch on a search hit. Useful to show the expanded text, e.g. to show the result of a SpreadSheetPlugin %CALC{}% instead of the formula'},
      multiple => {type=>'onoff', default=>'off', DOCCO=>'Multiple hits per topic. Each hit can be formatted. The last token is used in case of a regular expression ";" and search'},
      nofinalnewline => {type=>'onoff', default=>'off', DOCCO=>'If on, the search variable does not end in a line by itself. Any text continuing immediately after the search variable on the same line will be rendered as part of the table generated by the search, if appropriate.'},
        recurse => {type=>'onoff', default=>'on', DOCCO=>'Recurse into subwebs, if subwebs are enabled.'},
      }
   );

What I hoped, was that this idea was obvious ( sigh sorry ), and that we would add such a definition structure as a mandatory parameter to registerTagHandler.

I still think that the specification should be part of the tag registration - that way foswiki can reformat it to:

1. generate the currently installed tags documentation (rather than there being VarTopics who's code has been disabled)
2. generate parameter range and perturbation tests
3. generate editing tools
4. who knows - web service endpoint definitions..?

-- SvenDowideit - 19 Jan 2010

Of course it's theoretically possible to generate these definitions from doc pages, with a search....

-- CrawfordCurrie - 01 Feb 2010

It would be nice if doc pages were built from %$DOCUMENTATION{"key"}% macros which pulled from well-formed perl code comments and/or structures. BuildContrib could warn about unused key in the System topic.

More than this though, it should then be possible to do some kind of standard UI widget against each macro automagically in the WYSIWYG editor - either programming IDE type autocomplete/param suggestions, or a params builder UI similar to what Sven has in ComponentEditPlugin.

WIBNIF you could measure test coverage from the specs too, and warn about it somewhere.

-- PaulHarvey - 01 Feb 2010

yes, I was going to write a rest handler to pull out the marco definitions from the docco - however, the formatting or the docco is way too irregular for it to be anything but a starting point.

I'm contemplating some dataform like magic, or maybe microformats, but in the end I suspect it'll be more useful to generate the docco from the code definition - rather than trying to parse hand written stuff. See VarSEARCH and VarINCLUDE for a loverly example of weirdness.

-- SvenDowideit - 02 Feb 2010

Yes, I wrote that search just to see if it could be done. It can't

I don't have a problem with drawing documentation from a stored definition. The obvious place is a combination of a hash defined in the Macros/*.pm, and POD doc in that same file (I'm against always-loading the doc because of crappy perl performance). The key question in my mind is what has to be stored, and how to deal with exceptions. In terms of standard macros, there probably has to be some way of expressing:

• _DEFAULT
• named parameters
• whether it allows extended (friendly) syntax or not
• the types of named parameters (what is "type" in this context? Is it the same as the type of a %META:FIELD?).
• Whether the param supports standard $format identifiers.
• how to handle params that are present, but not in the macro schema (macro params to %INCLUDE, for e.g.)
• whether this is one end of an open-close macro, and if so, which end (%STARTINCLUDE...%STOPINCLUDE)
• other macros, that don;t have "proper" implementations (such as %TOC and those processed by commonTagsHandler in plugins)

-- CrawfordCurrie - 02 Feb 2010

I suppose this calls for a taxonomy of existing macros. TaxonomyOfMacros

-- PaulHarvey - 02 Feb 2010