Feature Proposal: Improve performance of nested sectional includes

Motivation

INCLUDE statements are far more expensive than they have to be, especially nested sectional includes

1. The same topic(s) are read again and again from disk for no reason
2. Those same topic(s) are parsed for START/ENDSECTIONs again and again for no reason

Description and Documentation

At minimum, we should be able to easily avoid parsing START/ENDSECTION more than once per topic.

SvenDowideit is concerned that we keep using PageCache feature with this, Eg. maintain the parsed section info across multiple requests. Using PageCache is desirable (to avoid having to re-implement the entire topic dependency stuff, for a sections cache that spans multiple requests), but it may require re-architecting to create a reuseable cache infrastructure.

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: PaulHarvey - 12 Nov 2010

Discussion

Furthermore, if we could optimise this:

%SEARCH{
  format="$percntINCLUDE{\"%TOPIC%\" section=\"foo\" variable=\"value\"}$percnt"
}%

%STARTSECTION{"foo"}%This is %TOPIC% and variable is %variable%!%ENDSECTION{"foo"}%

So that the foo section's occurance of %TOPIC% could somehow only be expanded once instead of on every search hit, then sectional INCLUDEs could be used quite cheaply more thoroughly throughout our wiki app developing practices

-- PaulHarvey - 12 Nov 2010

mostly, I'd like to see us bring all the different caches into one codebase - with unrelated portions being in subclasses, so that we reduce the testing requirements to only needing to prove that the core cache is correct. given that sectional includes can have a similar dependency issue to full pages - even topic user mapping groups have this, there should be some gains..

-- SvenDowideit - 12 Nov 2010

I would suggest formatsection instead of section, since that also allows for headersection, pagerformatsection etc.

Also: "prove" is a very, very strong word. For a start, it would require formal semantics for the programming language in use. Perl, now, handy as it is, does not meet that criteria. Then, once you have a proof, you have to maintain it. That kinda raises the bar for the average Foswiki dev. I certainly wouldn't qualify

-- MichaelTempest - 12 Nov 2010

Our WikiGroups problems I suspect could have been avoided if we avoid re-parsing a given topic for sections over and over again, on every INCLUDE statement.

I will examine the cache code at some point.

-- PaulHarvey - 12 Nov 2010