MultiSearchPlugin

Perform multiple searches with one single efficient macro

This plugin is a supplement to the normally powerful built-in SEARCH feature in Foswiki.

SEARCH can perform very advanced searches with a large number of Foswiki topics and return the results in a very flexible way. You can create nested searches where a search depends on the result of a previous search.

But there are searches that the normal SEARCH only can do by repeating the same search over and over again. Each time you execute a new SEARCH all the same topics are being searched again and again.

MultiSearchPlugin enables you to perform only a few searches and display the result formatted by common index values, or by presenting the search results devided in intervals (numbers or dates)

The performance result can be dramatic. An example of searching 1300 records by 2 search criteria by 80 intervals may take 30 seconds using a FORMAT which again contains two SEARCH macros. Same can be done in 4 seconds using one single MULTISEARCH macro.

The MultiSearchPlugin always assumes searches to be query type searches and the search syntax is the same as in the normal SEARCH macro.

MultiSearchPlugin has 2 presentation modes

• index - where all the search results are grouped by the existing values of the chosen index fields
• interval - where all the search results are groups by defined intervals between a start and an end value. The feature is mainly meant for dates but can also be used for pure number fields using text fields

An index field is a named field in a form. You can define the index field as either text, multi, or date.

• text - the results are groups by the raw text content of the field
• multi - this is used when a field can have multiple values and you want the same search result to be counted for each value in the field
• date - the date is treated as a date field

Index mode

Let us say that we have a bug reporting system. Each bug is assigned to an engineer. We would like to create a list of all the engineers that are current having at least one unresolved bug assigned to them and we want the list of bugs listed as a comma separated list of topic name links.

Here is how we can do it

%MULTISEARCH{
    search1="State='Assessed' AND Release='Project X'"
    web="Track"
    indexfield1="Assignee"
    indextype="text"
    indexmode="index"
    listseparator1=", "
    listformat1="[[$web.$topic][$topic]]"
    format="   * $indexfield - $list1$n"
}%

Another example is we have a customer complaint system. We would like see how many complaints each business area has and we would like to see how many are still not resolved. We want this in a table with one row for each business and one column for all products and for not yet resolved issues.

The records have a Business and a State field. The business field can have multiple values as the same product may cover several businesses. In a 3rd column we want the number of open severity High complaints. In a 4th column we want a list of all the open issues.

And finally a footer with the totals.

%MULTISEARCH{
    search1="State != 'Rejected'"
    search2="State = 'Open'"
    search3="State = 'Open' AND Severity = 'High'"
    web="Complaints"
    indexfield1="Business"
    indexfield2="Business"
    indexfield3="Business"
    indextype="multi"
    indexmode="index"
    listseparator2=", "
    listformat2="[[$web.$topic][$topic]]"
    header="| *Business* | *Total* | *Open* | *Open A-level* | *List of Opens* |$n"  
    format="| $indexfield | $nhits1 | $nhits2 | $nhits3 | $list2 |$n"
    footer="| | $ntopics1 | $ntopics2 | $ntopics3 | |$n"
}%

Interval mode

In this example we have web where we register issues. We would like to produce a chart of the arrival and closure rate accumulated per week. We want it only for project X. And we want the metric from the project start on January 1st till and including this week we are in.

The ChartPlugin needs a table with 3 columns. The X axis text, the data for the first series and the seconds series. For our own need it would be nice if we could click directly to the individual bugs so we add two more columns with links to the topics.

We do not want to count rejected and duplicates in the metric.

We have a field DateOriginated that gets set when we create the record. And we have a field DatePerformed which is set the day the issue is resolved.

%TABLE{name="srrate" columnwidths="150,100,100,200,200"}%
%MULTISEARCH{
    search1="(State!='Duplicate' OR State!='Rejected') AND Release='Project X'"
    search2="(State='Closed' OR State='Performed')  AND Release='Project X'"
    web="Track"
    indexfield1="DateOriginated"
    indexfield2="DatePerformed"
    indextype="date"
    indexformat="$day $mon $year"
    indexmode="interval"
    indexstart="01 Jan 2014"
    indexend="next Sunday"
    indexstep="1 week"
    listseparator1=", "
    listseparator2=", "
    listformat1="[[$web.$topic]]"
    listformat2="[[$web.$topic]]"
    format="| $indexfield | $ntopics1 | $ntopics2 | $list1 | $list2 |$n"
    header="| *Date* | *Opened* | *Resolved* | *Topics* | *Topics* |$n"
}%

Syntax Rules

MULTISEARCH -- perform multiple formatted searches in one efficient macro

The %MULTISEARCH% variable is handled by the MultiSearchPlugin

Parameters

Parameter	Description	Default
"..."	Unnamed parameter is not used in this macro
web	Web in which the search is performed	current web
topic	Limit search to topics e.g. topic="WebPreferences" topic="*Bug" topic="MyTopic,YourTopic" A topic, a topic with asterisk wildcards, or a list of topics separated by comma. Note this is a list of topic names and must not include web names. Adding a topic restriction to a search can greatly improve the search performance.	All topics in a web
excludetopic	Exclude topics from search e.g. excludetopic="Web*" excludetopic="WebHome, WebChanges" A topic, a topic with asterisk wildcards, or a list of topics separated by comma. Note this is a list of topic names and must not include web names.	''
format	Format of the resulting search for each index value of interval. Typically defines a bullet point or a table row	''
header	The header is output just before the formatted search results. Typically used for table headers. See tokens used in headers below
footer	The footer is output just after the formatted search results. Typically used for table footers and for summary texts. See tokens used in headers below
indexmode	Defines the mode of the search. Valid values are index and interval	index
indextype	Defined how to treat the index fields. All index fields are treated as the same type. text is plain text multi means the field has multiple values date means the field is interpreted as a date. The plugin will understand dates in the format described in System/TimeSpecifications and any form field defined as a date field	text
indexformat	Only for date type fields. Defines how the intervals are displayed when showing the start date. Defined by using Time Format Tokens. See below. If this is not defined dates are shown as epoch and text just presented in the original format	undefined
indexstart	For interval searches this defines the start value. For dates this is the date of the first interval. See note 2 about dates below.	undefined
indexend	For interval searches this defines the end value. The actual end of the last interval will depends on the indexstart and indexstep because the plugin assumes you will always want the last interval to be a complete interval. See note 2 about dates below.	undefined
indexstep	The size of each interval. For dates it can be number of seconds or it can be any positive relative time. See note 3 below	1 or 1 week
search#	The # is a number starting from 1. The search syntax is 100% identical with query type searches using standard SEARCH. You cannot jump the number sequency. The number must start with 1 and increment by 1 for each new search option.	''
listformat#	The format that is used for each found topic. These formatted lists are then joined together using the listseparator and output with the $list# (# - the search number) token inside the format parameter	''
listseparator#	The number # must also be used in a search# option. This is the format used for the $list# token used in the format string. Typically this is set to ', ' (comma space) to create a list separated by commas.	''
indexfield#	The name of the form field used for search number #.	''
delay="#"	Number of times the MULTISEARCH will delay its own expansion by changing the first and last %-sign to $percnt and all double quotes (") to $quot. This is used when you use MULTISAVE inside the format parameter of a SEARCH or FORMAT macro and you do not want to expand MULTISEARCH until the outer SEARCH/FORMAT is completed. Set the delay to 1 if it is a single level SEARCH/FORMAT. If you have nested SEARCHes you can set delay to the level of nesting.	0

• Note1: The parameters formats, headers, and footers do not by default put a new line at the end. You must always specify this with a $n if you need a new line (you will need this in most cases)
• Note2: Dates given to indexstart and indexend can be entered in two different formats
  • It will first try the normal Foswiki date/time formats as described in TimeSpecifications.
  • If the above fails it will try and calculate the date based on relative time that the CPAN library Time::ParseDate can handle. E.g. "-3 months" , "last Monday", "+2 days", "next Wednesday", "2 weeks"
• Note3: indexstep relative times is always relative to now. You can use any time format that the CPAN library Time::ParseDate can handle. E.g. "1 month", "7 days", "1 week", "2 weeks"

Parameter Tokens

header="..." parameter, footer="..." parameter

Use the header and footer parameter to specify the header and footer of a search result. It should correspond to the format of the format parameter. Both parameters are optional.

Example header:

header="| *Business* | *Total ($ntotal* | *Open* | *Open A-level* | *List of Opens* |$n"

Example footer:

 footer="| | $ntopics1 | $ntopics2 | $ntopics3 | |$n"

Format tokens that can be used in the header and footer strings:

Name:	Expands To:
$ntopics#	Number of topics found in current web. The total is calculated AFTER all the searches are complete
$calc(...)	Perform simple calculations of numbers inside the ( ). Only numbers and +, -, *, and / is allowed. $calc is evaluated after $ntopics# so you can write $calc($ntopics1 - $ntopics2) to get the difference between the two results

• In addition you can use all the standard format tokens with the header and footer parameters. See below.

listformat#="..." parameter

Example:

listformat1="$formfield(Originator)" listformat2="[[$web.$topic][$topic]]"

Format tokens that can be used in the listformat# string:

Name:	Expands To:
$web	Name of the web
$topic	Name of the found topic
$formfield(name)	Content of any formfield of the given topic. No formatting is done with the formfield content. It may contain characters that needs to be dealt with. (This may be enhanced in later versions of the plugin)

listseparator#="..." parameter

Example:

listseparator2=", "

You can use all the standard format tokens with the listseparator# parameter. See below.

format="..." parameter

Example:

format="| $indexfield | $ntopics1 | $ntopics2 | $list1 | $list2 |$n"

Format tokens that can be used in the format string.

Name:	Expands To:
$indexfield	When indexmode is index $indexfield show the common value of the indexfields When indexmode is interval $indexfield shows the lower value of the interval
$list#	The # is the number of the corresponding search number. $list# displays all the values given by the listformat# parameter which are joined together by the string defined by the listseparator# parameter
$nhits#	The number of topics found by this search number # for this index value or interval.
$ntopics#	$ntopics# is the total of topics found in search# in the accumulated indexes or intervals shown until now. NOTE! For intervals the $ntopics value starts at the number of topics found having an indexfield value below/before the indexstart parameter value
$calc(...)	Perform simple calculations of numbers inside the ( ). Only numbers and +, -, *, and / is allowed. $calc is evaluated after all other tokens are done so you can write $calc($ntopics1 - $ntopics2) to get the difference between the two results

• In addition you can use all the standard format tokens with the format parameter. See below.

indexformat="..." parameter

indexformat is only used when the indextype is date and it accepts these special time format tokens

Token:	Unit:	Example
$seconds	seconds	59
$minutes	minutes	59
$hours	hours	23
$day	day of month	31
$wday	day of the Week (Sun, Mon, Tue, Wed, Thu, Fri, Sat)	Thu
$dow	day of the week (Sun = 0)	2
$week	number of week in year (ISO 8601)	34
$month	short name of month	Dec
$mo	2 digit month	12
$year	4 digit year	1999
$ye	2 digit year	99
$tz	either "GMT" (if set to gmtime), or "Local" (if set to servertime)	GMT
$iso	ISO format timestamp	2024-11-28T07:40:30Z
$rcs	RCS format timestamp	2024/11/28 07:40:30
$http	E-mail & http format timestamp	Thu, 28 Nov 2024 07:40:30 GMT
$epoch	Number of seconds since 00:00 on 1st January, 1970	1732779630

Tokens can be shortened to 3 characters

Standard Tokens for header, footer, listseparator#, and format parameters

Name:	Expands To:
$n or $n()	New line. Use $n() if followed by alphanumeric character, e.g. write Foo$n()Bar instead of Foo$nBar Most macros accept parameter strings which are split over multiple lines. This is usually more readable than using $n tokens. If you are familiar with sectional includes, you might also consider nested sectional includes to hold the newline content outside of the parameter string entirely. Note that newline is not a line break. The browser will wrap the lines together. If you require a line break, displaying the results on two lines, use %BR%. Or use two consecutive newlines to create a TML "Paragraph".
$nop or $nop()	Is a "no operation". This token gets removed; useful for nested search
$quot	Double quote (") (\" also works)
$percent	Percent sign (%) ($percnt also works)
$dollar	Dollar sign ($)
$lt	Less than sign (<)
$gt	Greater than sign (>)
$amp	Ampersand (&)
$comma	Comma (,)

Installation

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. Use "Find More Extensions" to get a list of available extensions. Select "Install".

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See http://foswiki.org/Support/ManuallyInstallingExtensions for more help.

The plugin requires the CPAN library Time::ParseDate. Redhat/Centos users can run 'yum install perl-Time-ParseDate-2013' with the 'epel' repo enabled. Debian based distributions can install libtime-modules-perl

Plugin Info

Change History:
1.3 (08 Dec 2015)	Fixed a crash of the plugin when a search did not return any results
1.2 (19 Aug 2015)	Added support for topic and excludetopic parameters to improve performance. Added support for the $calc() token to enable simple calculations within the plugin. Added more user input check of the field names
1.1 (18 Aug 2015)	Better handling when using month as relative time (same date in next month instead of fixed number of seconds). More checks on input values to avoid infinite loops Support of the delay parameter when using MULTISEARCH inside a nested SEARCH or FORMAT
1.0 (17 Aug 2015)	Initial release
Dependencies:	Name Version Description Time::ParseDate >0 Required