Feature Proposal: SEARCH separator as newline after header and before footer ONLY when separator is not specified

Motivation

There has been a long and heavy discussion on Tasks.Item1773 and I would like to see a decision. It involves a change in default TML behavior so it requires a feature proposal.

Description and Documentation

Examples

The problem we have today is that SEARCH always adds a newline after the header and an additional one before the footer.

This is very commonly used when people create SEARCH that returns a TML table. If we change this behavior we will break 10000s of normal simple web applications.

There is no way we can change this default behavior.

But we have a problem. We are not able to easily create a SEARCH that returns for example an HTML table, or a list of options for a pulldown menu using header and footer. We have to omit the header and footer and place the prefix and postfix text before and after the search and define the separator as for example ", "

This proposal is a compromise that

• Enables advanced developers to use separator, and header and footer in a what is equivalent to perl: return $header.join($sep, @result).footer;
• Maintains full compatibility with the most common use of SEARCH where the user has defined a header table row and a footer table row and relies on SEARCH to add newlines after header, after each search result including the one before the footer.
• Breaks compatibility only in the situation where a user has defined separator and relies on having this separator after the header and before the footer.

It is very important that reviewers of this proposal evaluates the impact of this breaking compatibility.

I claim that it is not very likely that people will have specified separator="$n" when this is the default already.

People still wanting the $n after the header and before the footer will need to add the $n to the header and footer parameters.

I assume concern raised in advanced from MichaelDaum. I assume Sven approving as this compromise proposal was his. I assume CrawfordCurrie having raised concern as he suggested a fully compatible but more complex and maybe also messy proposal.

This proposal should be kept as for or against the compromise proposal. If we start a new war with counter proposals we will not get a resolution in 1.1.

The proposal is best described by writing the complete spec for VarSEARCH with the modification

The text in red is the new text

SEARCH{"text"} -- search content

• Inline search, shows a search result embedded in a topic
• Syntax: %SEARCH{"text" ...}%
• Supported parameters:

  Parameter:	Description:	Default:
  "text"	Search term. Is a keyword search, literal search, regular expression search, or query, depending on the type parameter. SearchHelp has more	required
  search="text"	(Alternative to above)	N/A
  web="Name" web="Main, Know" web="all"	Comma-separated list of webs to search. You can specifically exclude webs from an all search using a minus sign - for example, web="all,-Secretweb". The special word all means all webs that do not have the NOSEARCHALL preference set to on in their WebPreferences. Note that AccessControls are respected when searching webs; it is much better to use them than NOSEARCHALL.	Current web
  topic="WebPreferences" topic="*Bug"	Limit search to topics: A topic, a topic with asterisk wildcards, or a list of topics separated by comma. Note this is a list of topic names and must not include web names.	All topics in a web
  excludetopic="Web*" excludetopic="WebHome, WebChanges"	Exclude topics from search: A topic, a topic with asterisk wildcards, or a list of topics separated by comma. Note this is a list of topic names and must not include web names.	None
  scope="topic" scope="text" scope="all"	Search topic name (title); the text (body) of topic; or all (title and body)	"text"
  type="keyword" type="word" type="literal" type="regex" type="query"	Control how the search is performed when scope="text" or scope="all" keyword: use Google-like controls as in soap "web service" -shampoo; searches word parts: using the example, topics with "soapsuds" will be found as well, but topics with "shampoos" will be excluded word: identical to keyword but searches whole words: topics with "soapsuds" will not be found, and topics with "shampoos" will not be excluded literal: search for the exact string, like web service regex: use a RegularExpression search like soap;web service;!shampoo; to search on whole words use \bsoap\b query: query search of form fields and other meta-data, like (Firstname='Emma' OR Firstname='John') AND Lastname='Peel'	%SEARCHVAR- DEFAULTTYPE% preferences setting (literal)
  order="topic" order="created" order="modified" order="editby" order=  "formfield(name)"	Sort the results of search by the topic names, topic creation time, last modified time, last editor's WikiName, or named field of DataForms. The sorting is done web by web; if you want to sort across webs, create a formatted table and sort it with TablePlugin's initsort. Note that dates are sorted most recent date last (i.e at the bottom of the table).	Sort by topic name
  limit="all" limit="16"	Limit the number of topics from which results will be returned. This is done after sorting if order is specified. Note that this does not limit the number of hits from the same topic when you have multiple="on".	All results
  date="..."	limits the results to those pages with latest edit time in the given time interval.	All results
  reverse="on"	Reverse the direction of the search	Ascending search
  casesensitive="on"	Case sensitive search	Ignore case
  bookview="on"	BookView search, e.g. show complete topic text. Very resource demanding. Use only with small result sets	Show entire topic content.
  nonoise="on"	Shorthand for nosummary="on" nosearch="on" nototal="on" zeroresults="off" noheader="on" noempty="on"	Off
  nosummary="on"	Show topic title only	Show topic summary
  nosearch="on"	Suppress search string	Show search string
  noheader="on"	Suppress default search header Topics: Changed: By: , unless a header is explicitly specified	Show default search header, unless search is inline and a format is specified (Cairo compatibility)
  nototal="on"	Do not show number of topics found	Show number
  zeroresults="off"	Suppress all output if there are no hits	zeroresults="on", displays: "Number of topics: 0"
  noempty="on"	Suppress results for webs that have no hits.	Show webs with no hits
  header="..." format="..." footer="..."	Custom format results: see FormattedSearch for usage & examples	Results in table
  expandvariables="on"	Expand embedded macros before applying a FormattedSearch on a search hit. Useful to show the expanded text, e.g. to show the result of a SpreadSheetPlugin %CALC{}% instead of the formula	Raw text
  multiple="on"	Multiple hits per topic. Each hit can be formatted. The last token is used in case of a regular expression ";" and search	Only one hit per topic
  nofinalnewline="on"	If on, the search variable does not end in a line by itself. Any text continuing immediately after the SEARCH macro on the same line will be rendered as part of the table generated by the search, if appropriate.	off
  recurse="on"	Recurse into subwebs, if subwebs are enabled.	off
  separator="..."	Line separator between search hits (only used when format= is set) If separator is not defined, the default is "$n" (newline). Not defining the separator will additionally cause a newline to be added after a header and before a footer.	"$n" (Newline)
  newline="%BR%"	Line separator within a search hit. Useful if you want to put multi-line content into a table cell, for example if the format="" parameter contains a $pattern() that captures more than one line, or contains a $formfield() that returns a multi-line textfield.	"$n" (Newline)

• Example: %SEARCH{"wiki" web="Main" scope="topic"}%
• Example with format: %SEARCH{"FAQ" scope="topic" nosearch="on" nototal="on" header="| *Topic: * | *Summary: * |" format="| $topic | $summary |"}% (displays results in a table with header - details)
• If the Foswiki:Extensions.TablePlugin is installed, you may put a %TABLE{}% macro just before the %SEARCH{}% to alter the output of a search. Example: %TABLE{ tablewidth="90%" }%
• Related: METASEARCH, TOPICLIST, WEBLIST, FormattedSearch, QuerySearch, SearchHelp, SearchPatternCookbook, RegularExpression

Additional Documentation updates

in FormattedSearch:

• In the section for the header after the table add the note:
  • Note that if the separator parameter for SEARCH is not defined a newline is added after the header.
• In the footer section after the table add the note
  • Note that if the separator parameter for SEARCH is not defined a newline is added after the last search result.

Impact

There will be users that have defined separator="$n" and will find that the header and the first row are merged to a very wide table row. They will be annoyed. They will need to add a $n after the header OR remove the separator definition.

There will also be people that have used the header to define a list of constant items followed by a search that uses the separator. They will need to add a comma after the header.

Implementation

It takes 2-3 code lines to add this in trunk

And then some unit tests that need repair. Probably 1-2 to be added to existing suite.

-- Contributors: KennethLavrsen - 09 Mar 2010

Discussion

yes, I still prefer this compromise option that I created I think I implemented most of the unit tests needed already - but more are always a happy thing, and it might be that (as i'm still moving code around in there) it'll be more like 5 lines of code, and 20 to be removed..

-- SvenDowideit - 09 Mar 2010

I think Kenneth's proposal above is fine. Concern removed.

-- CrawfordCurrie - 09 Mar 2010

My first reaction was apprehensive: "so header/footer are not equal citizens to the format hits?". Because I always thought of them as first/last items in a list, with each format hit in between as equal citizens in that list.

So I read the task which looks like the initial problem is that the final result /always/ has the separator appended. Isn't that the bug?

Then I got thinking about why we even need special treatment for header/footer - if your preceding/following text are not equal citizens in a list with the format hits, shouldn't we just place that text before/after the SEARCH?

Then I realised it's probably because we want to take advantage of header/footer options that make them hidden when there's no results?

I don't have a concern; but it is weird that I had a completely different reaction to everybody else... maybe because I am thinking about the searches mathematically.

Please let me know if the reason for the special treatment is because it would be nice to exploit zeroresults="off" with header/footer in more diverse situations.

-- PaulHarvey - 13 Mar 2010

Paul, bloody brilliant.

Note to Sven: don't forget to consider header&footer in the zeroresults as format feature request (please add link) - it might just mean that the zeroresults param needs to expand $header and $footer - I am not sure about the existing code (which i suspect has been broken

as to the rest of your thoughts - yes, the problem is entirely due to a decade of legacy. footer was only just added in the last year, but header has been naf for too long to just break everyone's topics.

-- SvenDowideit - 13 Mar 2010

Header and footer aren't considered part of the (hit) list, i.e. they don't extend the list by 2 elements. Normally, they are suppressed when nothing has been found. Think of it in terms of this code

return '' unless @list;
return $header . join($sep, @list) . $footer;

Obviously, SEARCH diverges from this logic ... for historical reasons and that's what we are trying to work around now.

-- MichaelDaum - 13 Mar 2010

I am glad we all agree on this compromize. We may add an expert setting to turn it off again if the 1.1.0 later creates trouble and release that in 1.1.1. For now I implement as suggested.

I will make this before the 1st April deadline.

Accepted by 14-day rule. The proposal additionally was announced because it is breaking backwards compatibility and will in rare cases create a need for admins to fix topics.

But the community has decided the pain is lower than the gain.

-- KennethLavrsen - 24 Mar 2010