My SEARCHes are a mess. How can I write them more clearly?

Before you read this you should be familiar with formatted search and especially how the $percnt etc. standard formatting tokens are used to delay evaluation of macros in search results.

A common requirement when building Wiki applications is a "nested search" - that is, a search that returns a list of results that need to be passed to another search. This is usually done using Foswiki formatted search. These searches can get very complex and unreadable unless they are crafted carefully. This FAQ describes some techniques for keeping your searches readable, and also re-usable.

For a real world example, consider this: we have a set of topics, each of which describes a technical standard. Each of these topics has a TML table that contains a list of cross-references to other relevant technical standards. So one of our standards topics (BritishStandard5751) looks like this:

---+ BS5751
Description: Specification for dimensions of housings for hydraulic seals for reciprocating applications
---+ Cross references
| BritishStandard1134|
| BritishStandard4500 |
| BritishStandard5242 |

Other standards topics will cross-reference this standard, for example:

---+ BS5242
Description: Tubes for fluid power cylinder barrels. Specification for steel tubes with specially finished surfaces
---+ Cross references
| BritishStandard5751 |

We want to craft a search that will give us the names and descriptions of each British standards and a bulleted list of the other standards that refer to this standard; i.e. we want to generate a list with entries like this:

BS5751

Referring standards:

• BS1134: Assessment of surface texture. Guidance and general information
• BS4500: ISO limits and fits. General, tolerances and deviations
• BS5242: Tubes for fluid power cylinder barrels. Specification for steel tubes with specially finished surfaces

To do this, we need to:

1. search for each standards topic,
2. for each topic we find, search for other standards that refer to it,
3. extract and present the Description of that referring standard.

Foswiki formatted search makes this possible, by nesting other searched in the formatted results of the first search. If we were to write out the search required, it would look something like this:

%SEARCH{ "^BritishStandard[0-9][0-9]*$" type="regex" scope="topic" nonoise="on"
    format="
---+ $topic
Referring standards
$percntSEARCH{$quot\| *$topic* *\|$quot
 type=$quotregex$quot multiple=$quoton$quot nonoise=$quoton$quot
 format=$quot   * $dollarpercntSEARCH{$dollarquotDescription:$dollarquot topic=$dollarquot$dollartopic$dollarquot nonoise=$dollarquoton$dollarquot
format=$dollarquot [[$dollartopic][$dollardollarpattern(.*?---\+ (BS[0-9]*).*)]]: $dollardollarpattern(.*?Description: (.*?)\n.*)$dollarquot}$dollarpercnt$quot}$percnt
"}%

Not only is this unreadable, it also makes it difficult to re-use bits of the search in other places.

How can we organise this complex search more clearly? By making use of a powerful Foswiki feature called "sectional include". Sectional include uses the INCLUDE macro, which allows us to include just a small section of a topic - even the same topic. By using sectional include to include the format of search results, we can produce something much more readable. The same example written using sectional include would look like this:

%SEARCH{ "^BritishStandard[0-9][0-9]*$" type="regex" scope="topic" nonoise="on"
    format="
---+ $topic
Referring standards
$percntINCLUDE{\"%TOPIC%\" section=\"getstds\" refstd=\"$topic\"}$percnt
"}%

<!--<literal>
The HTML comment and the literal tag prevent this section from being formatted and displayed in the browser.

Here's the section that is included once for each result from the top-level search.
%STARTSECTION{"getstds"}%
%SEARCH{"\| *%refstd%* *\|"
 type="regex"
 multiple="on"
 nonoise="on"
 format="   * $percntINCLUDE{\"%TOPIC%\" section=\"getdesc\" std=\"$topic\"}$percnt"}%
%ENDSECTION{"getstds"}%

And here's the section that is included once for each *second* level search.
%STARTSECTION{"getdesc"}%
%SEARCH{"Description:" topic="%std%" nonoise="on"
format=" [[%std%][$pattern(.*?---\+ (BS[0-9]*).*)]]: $pattern(.*?Description: (.*?)\n.*)"}%
%ENDSECTION{"getdesc"}%

</literal>-->

Breaking this down, we have a top-level search for standards topics. Each result of this search then INCLUDEs the getstds section, which contains a search for other topics that refer to the containing standard. Finally for each topic that is found, another section with a search is used to format the information from the topic. This last search isn't strictly needed - you could put the format statement into the calling search - but it's shown here to demonstrate how a complex search can be broken down into re-usable sections. For example, the last search can be called with:

%INCLUDE{"%TOPIC%" section="getdesc" std="BritishStandard5751"}%

to render the information about that standard as a bulleted list item.

As you can see we've minimised the $horror of the flat search, and created re-usable sections. For even greater re-usability you can put the re-usable sections in a topic of their own, as a kind of "library of searches", and then just include them as you need them.

There are other topics on this subject but I can't find them; anyone who knows where they are, please add cross-references here.

-- CrawfordCurrie - 24 Mar 2010

last night, i was also looking for examples in the docs and couldn't find anything either! i have a local version of System.FormattedSearch demonstrating the use of sectional includes with search, but now i'm thinking of grabbing crawford's text. perhaps examples should go in System.IncludeTopicsAndWebPages as well. in any case, i would sure like to know if we have any existing documentation on sectional includes?

-- WillNorris - 24 Mar 2010