Feature Proposal: Improve Page Load Time

Motivation

Currently, all CSS and JS content (inline and files) are added to a page using ADDTOHEAD. This however is not optimal given todays browsers and how they process a page while loading it.

Description and Documentation

The page load time can significantly be shortened by putting CSS into the HEAD element while JS files are appended to the BODY element. While a browser downloads a page, it starts parsing it incrementally and begins to request secondary content like CSS, JS and image data immediately. However, it fully stops this process when it comes accross Javascript code, hands over processing to the JS-compiler and continues afterwards. Therefor, it is best to first let the browser download other content before, i.e. CSS. Once the bowser has downloaded the CSS files the user will immediately see the effect of the CSS, a styled page. If the browser is blocked by a JS file and can't continue to download CSS files, the user will see uggly page flashing and the like.

This feature proposal does not cover preprocessing CSS and JS files, like compressing and merging files on the server for now. There won't be a benefit for pipelined requests anyway. (Default in Opera, Firefox - about:config - enable network.http.pilelining )

A more in-detail explanation can be found at http://www.die.net/musings/page_load_time/

For the same reason drupal provides two distinct API calls: add_css and add_js. These both differ in the extra options specific for each content type (media, type, preprocessing, ...).

For more deteails see the documentation of the ZonePlugin.

Examples

Instead of

%ADDTOHEAD{"PATTERNSKIN"
  text="<link ..href='....css' /><script src='...js'></script>"
}%

you'd separate this into

%ADDTOZONE{
  "head"
  tag="PATTERNSKIN::CSS"
  text="<link ..href='....css' />"
}%
%ADDTOZONE{
   "body"
   tag="PATTERNSKIN::JS"
  text="<script src='...js'></script>"
}%

Additionally, a Foswiki::Func::addToZone() is introduced.

Impact

This impacts all plugins that use %ADDTOHEAD to add JS code to the head and requires other JS code to be loaded before.

Example:

• TwistyPlugin
• EditChapterPlugin
• JQueryPlugin
• Skins loading System.JavascriptFiles

Just to name a few.

Maintaining plugins in a backwards compatible way is a problem. Some plugins that start using %ADDTOFOOTER and Foswiki::Func::addToFOOTER() will be incompatible with older Foswiki cores. Therefor, it seems reasonable to provide the new functionality in an extra AddToFooterPlugin to ease the transition.

The name "ADDTOFOOTER" basically does not work out quite as well as "ADDTOHEAD" as there is no FOOTER html element.

Alternatively, an ADDTOZONE{"zone-name" ... } could be implemented that superseeds both ADDTOHEAD and ADDTOFOOTER. The former would be deprecated and replaced by

%ADDTOZONE{"head"...}%

All files of a specific zone are then rendered by the skin using

%RENDERZONE{"zone-name"}%

Note, that RENDERZONE must be expanded at the very end of the rendering pipeline, i.e. not as part of the TML parsing process. For backwards compatibility reasons the "head" zone will be added to the HEAD element if it wasn't added by an explicit %ADDTOZONE{"head"} call. Similarly, the "footer" zone will be appended to the end of the BODY element if there wasn't an explicit %ADDTOZONE{"footer"}% already.

Finally note, that ADDTOZONE and RENDERZONE might have other interesting use cases for wiki apps that collect material at a specific named position of the page during the render process.

Implementation

-- Contributors: MichaelDaum - 04 Feb 2010

Discussion

As this removes my concerns about bundling JS and CSS, I would vote for it. Also, I don't understand why you don't go "all the way" and add some %ADDCSS% and %ADDJS% functions while you're at it? These would just be some wrappers around the function(s) you're proposing.

And with these, we could more easily migrate to something more advanced JS and CSS handling, without having to once again recode all the plugins.

Also, I find the title misleading. This will do nothing much to improve page load time, it can just lure the user into thinking it's faster, when it fact it's just taking the same overall time, or did I miss something?

• I did indeed miss something. As the HTML parser will stop and hand over to the JS processor when it encounters JS code, this way we will save context switching, and hand over to the JS processor only once, and will not prevent parallel downloading, as MichaelDaum pointed out on IRC.

-- OlivierRaginel - 04 Feb 2010

I like the idea, but I would also rather see ADDCSS and ADDJS. I don't see why anyone would need the flexibility of ADDTOZONE.

-- AndrewJones - 04 Feb 2010

How about adding stuff to the sidebar incrementally using some %ADDTOZONE{"sidebar" text="render facet"}%? Just an idea.

The dedicated ADDCSS and ADDJS do have the advantage that they could be leaner to use. Instead of

%ADDTOZONE{"header"
   tag="MYNEWPLUGIN"
   text="<link rel="stylesheet" href="%PUBURLPATH%/%SYSTEMWEB%/MyNewPlugin/style.css" type="text/css" media="all" />"
}%

we'd write

%ADDCSS{
   "MYNEWPLUGIN"
   href="%PUBURLPATH%/%SYSTEMWEB%/MyNewPlugin/style.css"
}%

default media="all"

Both paralleled with %RENDERCSS% and %RENDERJS% in the skin. But then we'd still need a %ADDTOHEAD% for things like <meta ...> or when an author decides to add JS to the head nevertheless.

-- MichaelDaum - 04 Feb 2010

or maybe ADDTOHEAD is redundant? If we allow recursive TMPL definitions, doesn't that mean that skins can put the top level TMPL:P{CSS} and TMPL:P{JAVASCRIPT} in the right place for that skin, and everyone else can add to them?

I admit that we still need the dependency stuff in ADDTOHEAD - but then my original proposal was for adding dependencies from TMPL sections..

-- SvenDowideit - 04 Feb 2010

That wont work afaics as RENDERCSS and RENDERJS are expanded at the very end of the rendering pipeline while TMPL:Ps are among the very first. The final ordering can only be computed when every item and its dependencies has been collected per zone.

Another problem: content rendered via an ajax call might contain their own ADDTOHEAD. However RENDERHEAD (or </head>) is never processed let alone code being added to the head incrementally.

-- MichaelDaum - 04 Feb 2010

Just came across another use case for ADDTOZONE: collect buttons in a toolbar using %ADDTOZONE{"toolbar" text="render button"}%

-- MichaelDaum - 04 Feb 2010

ah, yes, very good point.

I like the idea of ADDTOZONE and RENDERZONE as a replacement for ADDTOHEAD. It would solve some of the ideas that I've been wondering about wrt dynamically customiseable sidebars too.

makes me think about redoing a skin using mostly ZONES.

-- SvenDowideit - 05 Feb 2010

I like the proposal, it's simple, however %ADDTOHEAD includes topic and requires parameters. The proposal does not mention these, and they are almost certain to be required, so raising a concern.

-- CrawfordCurrie - 05 Feb 2010

Basically ADDTOZONE has got the very same parameters as ADDTOHEAD has got now ... plus one: zone. This will be the new _DEFAULT as well so that the tag being the _DEFAULT of ADDTOHEAD will become an explicit parameter of ADDTOZONE. Technically the zone name is used to parametrize the internal hash in the Foswiki object, which is _HTMLHEAD for ADDTOHEAD now. This will be replaced with a _HTMLZONE hash that keeps all named zones encoded like they have been before. The Foswiki::_genHeader will be reused to create a Foswiki::_genZone taking one extra parameter for the zone name. In Foswiki::writeCompletePage() the call to _genHeader will be replaced with a loop over all zones that have been named during an ADDTOZONE. Special zones are head and body. If there wasn't a respective RENDERZONE for them, their content will be appended to the named html element.

So basicaly all of the ADDTOHEAD code is reused with some modifications to makeit more generic. I hope this addresses your concerns Crawford.

So far for the core changes.

Now comes the plugins part.

First, to allow plugin authors to make use of ADDTOZONE and RENDERZONE immediately, a plugin is created for these. Let's call it ZonePlugin. This will only be required by those plugins that run on a Foswiki-1.0.x engine and want to participate in an optimized html page layout.

Second, the default plugins will be reviewed to spot their use of ADDTOHEAD and inline <script> and <link> tags. These are rewritten using ADDTOZONE.

Documentation, unit tests, deprecate ADDTOHEAD.

-- MichaelDaum - 05 Feb 2010

I have a concern with the name: "Zone" is too abstract to reveal its purpose (while "Head" is not for the average user either but is understandable for developers).

Suggestions:

• ADDTOHTML
• ADDTOHTMLSECTION
• ADDTOPAGE

-- ArthurClemens - 05 Feb 2010

The sheer quantity of .js and .css files on a pimped out Foswiki is quite extreme. Compiling/merging files is a very difficult problem that in practice - for wiki interaction (as opposed to CMS usage) - probably wouldn't be a lot more beneficial than simply providing a simple time-stamp as URL parameters on files so that admins can confidently set their expires headers to a decent time into the future (12 months).

JQREQUIRE has a way of doing this, but it's not really accessible to casual skin developers who aren't used to the Foswiki svn/buildcontrib ecosystem.

An %ADDTOPAGE{"identifier" target="zone" requires="someotheridentifier" css="/pub/System/Skin/layout.css"}% should append a URL param ?201002071423 (for example) based on a stat() of the file for the last modified time.

Until somebody can convince me otherwise, I feel this is more urgent - and easier to implement (ie.: lower hanging fruit) than targeting html zones.

---

On a related note, it would be nice if our JQREQUIRE could fetch the JS from google's CDN instead of locally (a lot of sites do this, so chances are the user will already have jquery cached from some other site visit).

Also, with most browsers apparently limiting a site to about 4-8 concurrent requests, it would be terrific if we had some way of more easily hosting our files across multiple vhosts.

But these are just observations. My concern on this proposal is lack of file versioning to enable sensible use of future expires headers.

-- PaulHarvey - 05 Feb 2010

A word of warning on names; don't use HTML in the name of the macro, we generate other formats too (e.g. rdf and xml). What we can generate using templates is only limited by our imaginations. ADDTOZONE is OK. ADDTOOUTPUT more mundane, but perhaps more revealing.

-- CrawfordCurrie - 06 Feb 2010

@Arthur, not sure your suggestions are better than "zone". PAGE is pretty much all of the output, however we want to arrange CSS and JS a specific positions within the page so that todays browsers are satisfied. HTML is bad as well for the same reason imho. SECTION is overloaded with other semantics already but comes closest to what the macro is intended to do.

@Paul, versioning JS files is not part of this proposal though it could be extended to cover it. ADDTOHEAD and so ADDTOZONE never were intended to deal with versioning of CSS and JS files. Todays JQREQUIRE sits on top of ADDTOHEAD adding version tags as url params already. Note, that the current solution there is not enough: many proxy caches refuse to cache files that have url parameters. A better scheme would be to encode a version tag into the url path instead. This however is not that simple.

For now this proposal does not cover preprocessing of JS and CSS files to lower the number of HTTP requests. That's because this effort is lapsed as soon as browsers enable pipelining by default. Preprocessing would mean to merge, minify and gzip static files and cache the result to avoid the effort on every call. Note, that any preprocessing+caching solution would then be able to store the result at a location that encodes versioning into the url path. Further note, that multiple foswiki topics need different sets of JS and CSS files depending on the number of active extensions on a page. For example, there might be a sensible subset of CSS files (like the skin theme) to be merged while other files should be served individually as needed. ... As you see this is far from low hanging fruits. The research results at http://www.die.net/musings/page_load_time/ show that this effort won't even reach the performance improvements of browsing with pipelining enabled. That's why I'd like to limit the scope of this proposal to the outlined "zone" concept to relocated JS files to the end of the page.

CDNs are an interesting aspect that I havent thought of before. Something we can deal with later. Again, this proposal is only about the anatomy of an HMTL page, nothing more.

-- MichaelDaum - 07 Feb 2010

I understand that ADDTOHEAD and ADDTOZONE were never designed to deal with specific types of js/css files, but given the broad, sweeping title of this proposal I still consider URI versioning the issue because its absence prevents us setting proper dates for our expires headers.

I am not considering proxy caches in this, simply the behaviour of the user's browser.

The users for which I'm trying to target for a "faster" experience are not the random one-hit wonders who stray in from Google. These are "regular" users (upwards from 3 hits per week) who shouldn't have to be constantly downloading the same .css and .js again and again.

If we feel my concerns are better addressed in a different feature proposal, perhaps we can do this, but it would make sense to have the title of this proposal be more specific.

-- PaulHarvey - 07 Feb 2010

Why are your regular foswiki users constantly downloading the same CSS and JS again and again?

I assume you are using something like ExpiresDefault access plus 2 month. Try to remove all ETags using FileETag = none I am also setting a Cache-Control public for all static files which seems to help https connections sometimes.

I agree that adding some kind of version tag will help to prevent users caching old files for too long. I see this happening very often.

Maybe the title of this proposal is too broad to be practical and it should have been called GeneratedMoreEfficientHTML. I really don't mind. What I want is to cure one central deficiency in the design of ADDTOHEAD that will directly ImprovePageLoadTime.

-- MichaelDaum - 08 Feb 2010

I can't set more than 3 days, because I am afraid of bad interactions with upgraded js/tweaked CSS, which I do on Fridays. Then hopefully by Monday their browsers will get the new version.

I am still concerned ADDTOHEAD and even this new iteration is still severely lacking for practical skin/plugin development. If we add file semantics in order to be able to apply URI versioning, we might also get a solution to this problem - for free:

• Support.BestPracticeTip12 can call %JQREQUIRE{"wikiword"}%. But how to do that to get the same functionality shipped with core in pub/System/JavascriptFiles/foswikiString.js? Today, I have to do an ugly ADDTOHEAD with <script> tags, being mindful of single/escaped quotes, and not to hard-code PUBURLPATH or System, and also lock myself into assuming that the function I need (makeWikiword) will always exist in that directory in that file.

On the other hand, although I dont' like the idea of changing ADDTOHEAD and its successors yet again at some point in the future, I don't want to overload this proposal too much. Open source is about incremental improvements, and I lack the time to invest in finding a solution to my own concerns here.

-- PaulHarvey - 08 Feb 2010

I see what you mean. We've got the same problems for images and rest handlers as well. So this problem occurs not only when adding stuff to the head or a special zone.

-- MichaelDaum - 08 Feb 2010

Some thoughts: Another way of thinking about ADDTOZONE is that at its core it should really behave like an INCLUDE: respecting STARTINCLUDE, STOPINCLUDE, have a section parameter in addition to topic. Instead of inserting the text into the original place of ADDTOZONE, it then adds the content to a named position somewhere else.

-- MichaelDaum - 08 Feb 2010

ZonePlugin has been implemented. Plugin docu has got more details.

-- MichaelDaum - 12 Feb 2010

Please let me know what I can do to address the raised concerns, CrawfordCurrie and ArthurClemens.

-- MichaelDaum - 15 Feb 2010

Activating ZonePlugin broke our Wiki more ore less completely. We are making heavily use of small enhancements with JQuery. But, after we installed ZonePlugin , all the run-in <script>-Code threw errors, because JQuery and needed Plugins where not initialized at that point.

Heaving countless topics of this type, I think we will not be able to move Javascript libraries to the end of the html document any time soon.

I would even say, before we would have cleaned all our Wiki Topics from run-in <script>-Code, Javascript Worker Threads will have made the whole moving obsolete. I wouldn't even be surprised, if others start to move there Javascript back to the top, to avoid flickering webpage rendering caused by late DOM manipulation.

We will add %RENDERZONE{"body"}% to our skin further up the page. (Tip by MichaelDaum)

-- AndreLichtsteiner - 17 Feb 2010

Why do you expect flickering webpages?

-- MichaelDaum - 17 Feb 2010

Yet another 10 days passed and no sign of Arthur and Crawford. Could you two please drop a note wrt your concerns? Thanks.

-- MichaelDaum - 17 Feb 2010

With respect to the flickering - as far as I know, there are several jquery plugins that manipulate the DOM, for such things as syntax highlighting and tabs. If the javascript is only loaded at the end of the page, and the browser already has any needed CSS, then the browser will render the page before the javascript completes running. The javascript modifies the page. If the javascript engine is slow, the time from initial rendering (sans javascript) to the completion of the onload/ready/whatever javascript processing could be long enough to make the page appear to flicker when it loads.

-- MichaelTempest - 18 Feb 2010 (who thinks he is stating the obvious, but since no-one has mentioned this, he thinks he's better stick his neck out )

I should chime in and say that it also broke out wiki, too. Although I broke it for out test host first, as I was aware that JQueryPlugin had these heavy changes.

Situation was this:

ADDTOHEAD{"OurFooPlugin" .... requires="JQUERYPLUGIN"}

Now that JQueryPlugin lives in the body, we have a conflict: our local plugin depends on something that lives in the body, but itself lives in the head.

As our little plugin doesn't care about being in the head or the body, I was able to "simply" change it to RENDERZONE{} (without an explicity target, lest I fall into this trap again, Eg. JQUERY goes back to living in the head!)...

I am concerned more about JQueryPlugin assuming it should live in the head, than with this actual proposal.

-- PaulHarvey - 18 Feb 2010

@Michael: this is an exact description of how things work out ... no matter if javascript is put into the head or the body. In general jQuery is very efficient starting all initializers on DOM-ready event so that flickering hardly visible '. So far I have not experienced any more or less flickering at all. However, that definitely needs a more closer inspection. What I definitely do have experienced is a much snappier overall page loading experience. Even firebug reports much faster times of completion. But that is also a result of the overall improvements to the plugins.

@Paul: did the net load of your ADDTOHEAD{"OurFooPlugin" .... requires="JQUERYPLUGIN"} carry CSS as well as JS in one block? Try to split it into two, one ADDTOHEAD{"OurFooPlugin::JS" .... requires="JQUERYPLUGIN"} and one ADDTOHEAD{"OurFooPlugin:CSS" .... requires="JQUERYPLUGIN"} That's backwards compatible but will participate nicely in ZonePlugin's optimization heuristics.

ZonePlugin tries to auto-detect if an ADDTOHEAD is

1. pure css,
2. pure js,
3. both js and css or
4. something else.

By default it puts everything into the head zone, except pure js (case 2). If it detects js but also css it will fall back to head zone. Maybe for case 3 it is better to put stuff into the body zone instead of the head zone as that's where most js is supposed to live. The extra css in there might hurt less than any missing jQuery library.

-- MichaelDaum - 18 Feb 2010

Right, we had both JS and CSS. I was not aware that ZonePlugin did this auto-detection. I did split it up into two different ADDTOZONEs as you suggest, but only because I looked at the HTML and realised that the CSS was now in the wrong place. I suppose I should have read the Perl API documentation.

I have a feeling that it would be "less broken" for js+css blocks to be moved to body. Maybe depending on requires=""? At the end of the day though, mixing the old usage and assumptions of ADDTOHEAD with the new ADDTOZONE is likely to result in broken sites.

I would even go as far as saying that if there are any uses of ADDTOHEAD at all which have a requires dependency that cannot be resolved (exists outside of head target), then ZonePlugin should "give up" and render everything into the head (revert to old behaviour).

-- PaulHarvey - 18 Feb 2010

Maybe the docu is not clear enough about auto-detection. It is mentioned here.

Wrt "giving up": there are a lot of plugins that add a required dependency for on other stuff that only occasionally shows up. Overspecifying dependencies is actually quite useful. It is not that the problem then gets overconstrained / unsolvable. It is more of a security network for later material that the current has got the right relation to already.

-- MichaelDaum - 18 Feb 2010

More comments at High Performance Web Sites: Rule 6 – Move Scripts to the Bottom. My pick:

This is a very useful method for making a site appear to load faster than it actually is. If you are using good, unobtrusive JavaScript techniques (and making JS enhance the site rather than create it) this is a beautiful technique.

Most of the negatives in these comments are stemmed from bad JS practices - body onload attributes and document.write (:x) should not be used in the new web world we live in.

All modern JS libraries make use of a document.ready event, which is key producing JavaScript that only gets activated when the site is done loading. In this scenario, the script tags are fine to have right before the closing body tag - it makes the visible parts of the screen load first, and once the js is loaded and the dom is ready - the javascript fun kicks in and it's a fun user experience for all.

Posted by: Jade Rauenzahn at August 22, 2008 8:04 AM

That's been 2008 .

-- MichaelDaum - 18 Feb 2010

What makes me nervous here is that Arthur and Crawford are being chased to give up their concern.

But at the same time I read reactions from several people like: "Activating ZonePlugin broke our Wiki more or less completely." and "I should chime in and say that it also broke our wiki, too".

I think the spec and the consequences to the 200+ plugins we have is the greatest concern here.

And on top of that. What about the concern about "flickering browser"? I do not want that.

Yes. I want "ImprovePageLoadTime". But what do we get if we say yes to this proposal? More rushed and unfinished features into trunk?

I add my name to the concern list. What I wish for is

• Refactored more clear spec. It is not too bad. I can ignore the wrong topic name. Nevermind that.
• More analysis of the consequences now that we know the kind of small disaster it was to people that use Michaels plugin suite.
• Some guarantee that adding such a feature to core will not break one single plugin that works today with 1.0.X.

But yes I want ImprovePageLoadTime so the concern can be lifted if the real issues are lifted. The already raised concern about the name of the ADDTOXXX is called is the least of the problems IMHO. ADDTOZONE is fine. The idea is fine. The intentions are fine. Problem is the rush. It will not be too difficult to get my name off the concern list.

-- KennethLavrsen - 18 Feb 2010

Here is another one for the concern pile.

[23:36] I wonder if I could get a sanity check from someone with latest NatSkin on 1.09. EditTablePlugin seems to have lost any javascript abilities, to re-order or delete rows, even is specified to be on in If I set the skin to patten in the topic prefs, the EditTable is fine.

-- KennethLavrsen - 18 Feb 2010

If we want to avoid breaking a single plugin - there is no other option but to "fall back" to putting everything into the head, if there is any plugin code or TML usage of ADDTOHEAD macro that specifies a dependency which cannot be resolved due to the dependency living outside the "head" zone.

Even doing this might still break some plugins, that fail to specify a dependency when they should have (ie. those plugins only work by accident and luck that their content is inserted after anything they depend on).

Also - if I ADDTOZONE{text="..." requires="JQUERYPLUGIN"}, will this always add to head zone? or will it automagically add to whatever zone the JQUERYPLUGIN has landed in?

-- PaulHarvey - 18 Feb 2010

CraigBowers' problem with EditTablePlugin is documented at Tasks.Item8571. Found a work-around.

Actually, this problem lies with NatSkin, not ZonePlugin.

But in the big picture, ZonePlugin is likely to provoke more similar breakage, simply because the way we deal with javascript in Foswiki is almost totally unmanaged - but ZonePlugin, or something that leverages it, could help us.

If EditTablePlugin requires foswikilib.js, it should be able to do something like a ADDTOZONE{text="...edittable.js'>" depends="Foswiki::lib"}.. where "depends" or some new keyword that implies that the "Foswiki:lib" dependency must be defined and included somewhere and generate errors if it can't be satisifed. Essentially I would really like to see something functionally equivalent to JQREQUIRE which could be useful beyond JQueryPlugin land.

-- PaulHarvey - 19 Feb 2010

Damn! We kiled TinyMCEPlugin for IE users. Tasks.Item8566

-- PaulHarvey - 19 Feb 2010

@Kenneth, thanks for the feedback will clear the specs above.

Meanwhile, I've added a BackwardsCompatibility switch to the latest release to ZonePlugin for those skins and extensions that bypass ADDTOHEAD or Foswiki::Func::addToHEAD() APIs to add their bit of JavaScript code. In fact, this all shows that we really f* messed up the way JavaScript has been added to the HTML page. See my comments on Tasks.Item8571. I.e, all libraries in JavascriptFiles are not properly added to the HTML page in most cases, even multiple times depending on the plugins currently being used on a topic. This can only be fixed by making proper use of ADDTOHEAD and by agreeing on tags for them, e.g. FOSWIKILIB for adding foswikilib.js to the page). Not having done so in the past is an error in itself, independent from this proposal.

-- MichaelDaum - 19 Feb 2010

Michael, I've been bitten again in another app where we used ADDTOHEAD{text="<meta...><script>"} and ZonePlugin put the whole chunk including meta tags in the body where they can't be processed by IE clients.

I am really beginning to think that we either make ADDTOHEAD truly add to head and let people update their plugins properly to use ADDTOZONE{"body"}, or at least only shove ADDTOHEAD text to the body if it only contains <script> tags.

We will continue to make broken HTML if nothing changes.

Also: could we please not make a convention where we use ADDTOZONE{"body"} directly. I would very much rather ADDTOZONE{"script"}, this would allow us to implement some smarter version of SafeWikiPlugin which I suspect will be very broken for the june release, despite the fact we recommend it in configure as an alternative to {AllowInlineScript}.

-- PaulHarvey - 28 Feb 2010

Which version of IE breaks under which circumstances?

Did you try the {BackwardsCompatibility} switch?

I agree that we most probably will have to make ADDTOHEAD unconditionally add to head without any smartness trying to decide where to put the stuff.

While I was working on optimizing javascript in foswiki, I've found a lot of other places where cleanup is needed, not necessarily related to this proposal, but still desperately needed.

About changing the zone name body to script: I am trying to use labels that relate to locations within the html page rather than what is actually put there to make it more generic.

-- MichaelDaum - 01 Mar 2010

Testing with IE7. I did not use compat mode because I don't need it. I just need ZonePlugin to only convert ADDTOHEAD to ADDTOZONE{"body"} if and only if the text being added contains only <script> tags.

Failure mode on TinyMCEPlugin was that foswiki_tiny iterates over all <meta> tags looking for TINYMCEPLUGIN_INIT, but can't find it because apparently IE's get-elements-by-tag-name feature fails to present <meta> tags unless they are in the <head>.

In my recent adventure, the failure mode was that <meta name="foswiki.foo.bar" /> tags which get landed into the <body> are not added to the JQueryPlugin foswiki object (again on IE7).

-- PaulHarvey - 02 Mar 2010

Will try to repro.

-- MichaelDaum - 02 Mar 2010

I think some of us have doubts, but I know I am convinced: we need JQueryPlugin shipped as a core/default extension, the current madness we have with javascript in Foswiki is unacceptable.

Now, what fewer people are convinced about is addToZone(). Unfortunately JQueryPlugin has become intertwined with this thing: current trunk doesn't work without it (see Tasks.Item8655).

I think it is safe to say that most of the breakage we have seen with ZonePlugin is not actually bugs in the addToZONE() implemenation itself, but the monkey-patching and overriding of the old behaviour of addToHEAD().

These are my hopes, which I believe are reasonable goals for the june milestone:

• No more magic in addToHEAD():
  • addToHEAD() always uses the old behaviour, unless configured otherwise.
    • "configured otherwise": let there be an option to move content which contains only <script> tag(s) to the =<body>=/"script" zone.
  • addToZONE() always behaves like the old addToHEAD(), unless configured otherwise.
    • "configured otherwise": let there be an option to make the special "scripts" zone add <script> tags to the <body> of a document, as this proposal aims to achieve.
      • This configuration may break with old skins/plugins, but it is not the default behaviour until we are confident that most of our extensions/skins have "caught up" and been migrated to the New Way of Doing Things
• Review tests for addToHEAD()
• Write tests for addToZONE()
• Ensure that all usage of addToHEAD() and addToZONE() behave exactly as addToHEAD() did in Foswiki 1.0.x for the out of the box configuration

We do need %ADDTOZONE{"body"}%, make no mistake - I have been testing my wiki extensively with these changes (and suffered the consequences!) but there has been very positive user feedback. But we need a transition period.

Aside: Zones could be come a vital part of modular skin development, unless we get RecursiveTMPLDefinitions out the door.

-- PaulHarvey - 03 Mar 2010

Switching on {ZonePlugin}{BackwardsCompatible} does the trick. It will be enabled by default when it becomes a core feature.

Let me iterate again: this proposal is not about adding ZonePlugin to the standard set of plugins to core!!! I've seen people repeatedly beeing irritated asking why this is not a core feature. This proposal is about making it a core feature.

The ZonePlugin itself is only there to let plugin authors use zones right away, even on older Foswiki engines.

A non-neglectable reason why we have seen breakage is that javascript in Foswiki has been treated rather messy. For instance, we still have sever problems using and adding JavascriptFiles correctly: they are added to the page even though they are not used; they are added multiple times accidentaly.

This was disclosed all too obvious while working on improving page load time and using the zone concept.

Anyway, ZonePlugin comes with a backwards compatibility mode which should be enabled by default and that's it, right?

-- MichaelDaum - 03 Mar 2010

I think the plan you are executing with planning the feature in core, and having a plugin so also 1.0.X users can take advantage of this is great.

And it also allows the many extensions to get to an optimized use of the feature and we get the feature stable before we put it in core in a release.

My concern is only rushing it and I will soon remove my concern.

Great work in an important area Michael

-- KennethLavrsen - 03 Mar 2010

I still feel "zone" is not the right word. The global structure of an HTML document calls it parts (and also sections).

So other naming suggestions:

• ADDTOPAGEPART
• ADDTOHTMLPART
• ADDTOPAGEELEMENT

-- ArthurClemens - 04 Mar 2010

These are good suggestions but more complex and not really more intention reveiling

• page part
• html part
• page element

These consist of two words while "zone" is just one.

What about "area" ...woud read ADDTOAREA ? Sounds a bit more CMS-ish.

-- MichaelDaum - 05 Mar 2010

Yet another idea: PAGEINSERT.
For example:

%PAGEINSERT{
"footer"
tag"PATTERNSKIN"
text="<script src='...js'></script>"
}%

-- ArthurClemens - 07 Mar 2010

Does not reflect the fact that this is an additive operation. Can we agree on either ADDTOZONE or ADDZOAREA, please?

-- MichaelDaum - 07 Mar 2010

I will not take part in the nameing discussion. None of them really say what the thing does but I cannot come up with anything better. And this is not really a macro for the normal end user. Adding JS is for the geeks so I can tolerate a geeky macro name.

I wanted to halt the decision till the feature got some time to settle in.

I think we are there now so I remove my concern.

Michael, even though I have removed by concern I would like some benchmark numbers. How will the end user experience the improvement? What is your experience? I am not looking for stop watch numbers. Just some average figure based on experience.

You have some examples we can test and benchmark ourselves?

-- KennethLavrsen - 09 Mar 2010

As noone else is proposing a better name (or it doesn't matter to others), I will no longer hold up this request. If I have to choose between the 2 it would be ADDTOAREA.

-- ArthurClemens - 09 Mar 2010

Kenneth, sorry I don't have benchmarks. I am not even aware of a benchmark tool that simulates this browser behavior.

However, the effect of moving javascripts to the bottom of the page is well understood:

Here's a quote from yahoo:

The problem caused by scripts is that they block parallel downloads. The HTTP/1.1 specification suggests that browsers download no more than two components in parallel per hostname. If you serve your images from multiple hostnames, you can get more than two downloads to occur in parallel. While a script is downloading, however, the browser won't start any other downloads, even on different hostnames.

The overall downloading process is bottlenecked by every single javascript, blocked for the time it takes to download it as well as execution time of the javascript snippet. Only after the js parser finished this one file will it download anything else. The same process then repeats for every js file being scheduled for download+execution too early. So if the series of files the browser schedules to download is mixing up javascript and non-javascript files (css, images, whatever) in a rather unfortunate way - and this is the case for Foswiki-1.0.9 - then this is more like a stuttering motor.

Compare this to a situation where all non-javascript material is downloaded first as fast as possible, before the browser schedules to download the first javascript file. Obviously, that's much better.

-- MichaelDaum - 09 Mar 2010

Michael, I am aware that no tool exists to give an exact bench mark. I was more looking for what kind of human impression you have now that you have implemented it as plugin and done the development.

What is the impression. Does the page settle in 1 second instead of 2? Can you feel the difference when you load the same pages with some JQUERY stuff or the Wysiwyg editor?

I just curious about how much of an improvement to expect based on the experience you have until now.

-- KennethLavrsen - 09 Mar 2010

Roughly speaking, there is an extra page load of delay for each Javascript file, placed before any content, before the user will see any page rendering. Exactly how long that is depends of course on the server and the network connection from the client to the server. (I have seen this illustrated graphically with image captures of Firebug's Net panel, showing the time taken to load pages with the Javascript at the top, and with the Javascript moved to the bottom, but I cannot remember where to find this example, and my quick search on the web did not turn it up.)

-- IsaacLin - 10 Mar 2010

Yes, I too have read the story. But I want to hear the practical experience. The experience from Foswiki and real browsers. Also to validate that this method - when used in Foswiki context - does not trigger some old browser bugs that make browsers hang for a long time. We have seen this before. I would like some validation of this method and not just blindly trust what we have read in an article on the internet. We may have missed some important detail in the implementation.

-- KennethLavrsen - 10 Mar 2010

The only problem reported so far is about IE6 not being able to handle <meta> tags in the body. It shouldn't be placed there for any browser.

There are a couple of web developer tools that help in evaluating performance problems of a web page:

• Firebug: as Isaac already mentioned, its network analysis is great to watch how an individual page loads, including times when the DOM tree is ready.
• YSlow for Firebug: an extension to Firebug, which provides a kind of expert system to analyse a web page, including recommendations; default Foswiki is ranked rather low with a couple of critiques
• Lori: Life-of-request-info: lean way to profile individual pages; great way to show times for initial and finish of a response as well as its size

YSlow provides a very clear analysis of what's going on. Without ZonePlugin, foswiki ranges around rank D at best, which is bad. With ZonePlugin enabled (and backwards compatibility mode switched off), all javascript and css files gzip compressed, etags removed from static files the rank goes up to B, which is okay. Remaining critique is the lack of a content delivery network, which is off for most intranets.

-- MichaelDaum - 10 Mar 2010

In my testing, it "feels" like pages are ready for me to use a good whole 0.5-1 second quicker when just clicking around our wiki. It just "feels" snappier, although in actual fact the times might be smaller (I'm sure there is some very subjective human perception going on that exaggerates the improvement).

Here is a video:

• addToBodyTest.ogv: addToBodyTest.ogv

It shows first, an ab -n 10 run against a version of our wiki that is identical to the other (on the same server), which has ZonePlugin in "compatibility mode" (all script added to head). The other is an almost identical setup without this option (all scripts in body).

Then the video shows two firefox tabs: the first, showing ZonePlugin compatibility mode site - following a link to a "moderately heavy" js page (well, all of our site is js heavy, this page happens to have a little app in it).

Then the second tab shows the same page click on the other site which uses ZonePlugin with scripts added to body.

The effect is most notable when the browser hits the page with a cold cache, which is not surprising. Although I am sure I can "feel" a difference even with a warm browser cache, it is much harder to show in the video. Which is why I cleared the cache both times.

Hope it helps. If anybody wants to browse the wiki using the special test URLs to compare with vs without scripts in head please ask on IRC.

-- PaulHarvey - 10 Mar 2010

500ms is a lot, it seems. What's the percentage is it compared to the total time til the page finishes?

Cleaned up the specs and added a link to the ZonePlugin for a more detailed documentation of the intended feature.

@Arthur, ADDTOAREA is fine with me. Bit of renaming work, but okay.

-- MichaelDaum - 10 Mar 2010

Firebug's net panel shows the pages in the video show ~6 seconds to load 60 files/135KiB (25 files/53KiB is JS - and constitutes ~3 seconds of page wait time). This is over my home 6MBit DSL, and with cold cache.

-- PaulHarvey - 10 Mar 2010

I don't like the topic and section params described in ZonePlugin (sounds like duplication of other efforts to me) but see enough good stuff that I will remove my concern here.

-- CrawfordCurrie - 10 Mar 2010

Now that the last concern is lifted, we can declare the proposal accepted by consensus.

Michael. You can go ahead and merge your plugin into core on trunk.

-- KennethLavrsen - 10 Mar 2010

Thanks to everybody who participated in this discussion.

Tasks.Item8696

-- MichaelDaum - 12 Mar 2010