Feature Proposal: .spec files are under-used and only by configure. Would be nice to use them more

Motivation

More often than none users are bitten because they don't know some stuff is missing.

They go to configure, which states everything is fine, then they scratch their heads as to why this feature isn't working. And it's just because configure reads the .spec files for their plugin, but the core doesn't. So all they have to do is tell configure to save. Lost 2h yesterday with JQuery, George lost some too, and I'm sure many of us got bitten.

Description and Documentation

All idea is to move all .spec files into one place, so they're easy to scan and parse and use, and have configure AND the core read them for default values. A nice side-effect will be that LocalSite.cfg will only contain the modified values, and not the default ones at the time it's run. Which should make upgrades much easier (think of switchboard entries or search operators).

First step is to move the spec files in a central directory, then we can start using them by:

• Having the core read default values from them. This is easy, all that needs to be done is:
  1. Read all .spec files in lib/specs
  2. Read LocalSite.cfg which might override stuff
  3. Expand variables at the end, so .spec files can use stuff which are defined only in LocalSite.cfg, and vice-versa
• Having extra information in the .spec, such as:
  • VERSION information, which would help configure (no need to parse / load the .pm to figure out the version)
  • MD5SUM of installed file, which would allow clean removal, safe upgrade... (Check EnablingIncrementalCoreUpdates and AddRemoveFileFeatureToManifest which wouldn't be needed)

Also, this would mimick closely the other feature proposal for the i18n stuff: Enhancei18nArchitecture

Examples

Now, we have spec files called Config.spec for everything (including core), and stored in the plugins directories, which is a pain to scan:

• lib/Config.spec
• lib/LocalSite.cfg
• lib/Foswiki/Plugins/JQueryPlugin/Config.spec
• lib/Foswiki/Plugins/CompareRevisionAddOn/Config.spec

This proposal would (as a first step) move these into their own specs directory:

• lib/LocalSite.cfg
• lib/specs/core.spec
• lib/specs/JQueryPlugin.spec
• lib/specs/CompareRevisionAddOn.spec

Impact

WhatDoesItAffect:

Implementation

-- Contributors: OlivierRaginel - 16 Nov 2010 (with some minor inputs from GeorgeClark and CrawfordCurrie)

Discussion

The only concern being that having every foswiki request looking for and reading all these files may slow things down a little (more to do with filesystem cache flushes than direct speed)

so I wonder if this can be done in a similar fashion to the .po -> .mo compilation - heck - is Storable faster to load than a .cfg?

and related - yes please, I've raised a talk or 2 because configure doesn't even tell you about some .spec file defaults being saved, so even when you do the right thing, you're discouraged, because configure tells you nothing.

-- SvenDowideit - 16 Nov 2010

This will have a negative performance hit and seems to me to be a developer issue and not seen by normal installations.

When you install a plugin you run configure. And you need to both install and enable the plugin. Configure should be reading the Config.spec file in this situation and merge it in.

I think the reason you see it is that you run pseudo-installs where a Config.spec silently gets updated and then the configs are not merged in.

Are we fixing he right problem then?

I would want to look more carefully at pseudo-install and configure first before letting the core code read more files.

We split the perl code up in more and more and more files. Look at the number of files Foswiki has become and compare this with how the old project looked like in 2005. We are a factor 10 up now. We cannot continue this trend. The speed is 30% for a normal render compared to a normal rendering and the only reason we have survived this is that machines are also 3 times faster than 2005. But we have a speed problem and proposals that clearly will go in the wrong direction should be avoided.

Also note that a speed test on a fresh Foswiki with only few 100 topics gives a false result. The disk cache will probably contain all the perl files. On a large production site the files cache also contains the traffic from 100000 topics being looked at all the time, and the disk heads which are the bottlenecks of any computer are constantly moving around. The more source files the more they are moving around. You often see 5 second answering times or more when there is traffic.

We need speed. Not more slowdowns.

-- KennethLavrsen - 18 Nov 2010

You can see the configure action as being the compilation process that Sven suggests - it "compiles" the .spec files into the LocalSite.cfg. I seriously considered moving LSC to a binary format at one point, but couldn't get a portable Storable to work reliably (plus I personally hack LSC). The "looking for" that Sven refers to is a lot better if all the spec files are in one place, because you are simply enumerating the contents of one directory.

BTW as an experiment a while ago I collapsed all the Foswiki modules into one ginormous file (those that could be collapsed, anyway) and ran some benchmarks on it.

It made bugger all difference.

Of course I have a fast disk, but all the same, don't look to this as a source of major speedups. (Speedups will come from the simplification and ajaxification of the skins. Template rendering, and the rendering pipeline, is IMHo the major bottleneck)

-- CrawfordCurrie - 18 Nov 2010

Regarding Lavr's concern about this being just an issue with pseudo-install, There are still issues for everyone when new releases of an extension add to the Config.spec. In those cases, the user does not necessarily have to save, and configure gives no indication that the config requires save. Also when the "default" changes in the Config.spec, I don't believe we have a mechanism to update the previous default saved in the LocalSite.cfg. (I suspect that Tasks.Item2103 is related to this.) I've been burned a number of times by omitted Switchboard changes.

Another option might be to have the package installer update the LocalSite.cfg when the package is installed. An added benefit would be that the installer could also have an option to auto-enable the installed extension. However this doesn't address the user who installs by tar/unzip into the Foswiki root. It could also detect that defaults have changed by comparing to the previously installed specifications.

One thing not mentioned above is that the [SomeExtension]_installer file is also becoming somewhat a specification file, containing the extension's Manifest and Dependencies. It also contains a "MANIFEST2" section which is parsed to retrieve a per-file MD5. It's missing the config spec, and version information. With 1.1, the most current version is written to working/configure/pkgdata/ directory, and is removed from that directory when an extension is uninstalled. The "Package.pm" object loads the installer to obtain the specifications of what to install/remove/backup ... However the core extensions are NOT recorded there, and probably should be.

-- GeorgeClark - 18 Nov 2010

Crawford I do not believe any results done on a machine with no traffic and no huge number of topics. As long as the entire Foswiki sources can be in cache in RAM you do not see a difference. The minute the cache is full and the disk system starts spinning harddisk heads the speed goes down dramatically. This test should be repeated with a site with traffic fetching DIFFERENT topics in parallel all the time and with access rights setup and with INCLUDEs so each topic view is realistic. It is the same reason that Apache performance is better when .htaccess is disabled to avoid disk heads flying around.

I do not believe we have ONE bottleneck. We have several. And to improve speed we have to consider everything.

If configure can install a plugin and the installation does not update the LocalSite.cfg or give a warning about it in configure then maybe this is where we should start.

-- KennethLavrsen - 18 Nov 2010

The server indeed serves only foswiki (and MySQL, but the load from that is low). I usually run benchmarks on a Foswiki with ~100 webs, 100K+ topics which includes 50K+ topics in a single web. IMHO a web server that does not have the sources cached is not a web server, it's a toy. Of course we have several bottlenecks. But when you look at the performance profile, it is clear that rendering (or more accurately, macro expansion during rendering) that is the key stage at the moment. It is the difference in rendering performance that is the major change from early (tm)wiki.

Anyway, back on topic, Olivier and George are right, we have a clunky install process because of the mysterious configure step. Plus rationalising the _installer and .spec makes a lot of sense, especially if access to these files is available via a simple API (something Sven has wanted for a long time, for the package installers).

-- CrawfordCurrie - 19 Nov 2010

First, Kenneth, I do not believe any results without proof either. Saying that Apache performance i poor with .htaccess because disk heads fly around... Did you track down the bottleneck? It's slow with .htaccess enabled, it's even slower when used, because it has to read them every time. Any, we all agree it shouldn't impact performance, and would be best if it could improve them instead of hinder them. So, I guess we could reduce the extra to 1, or even 0 files:

• As Sven suggested, Have configure or some script (some people don't use configure), build a cache of the .spec, and have Foswiki read this. Apply then the LocalSite.cfg (which would only have differences, so small) => 1 extra access, which could be stored in Storable, and would be re-created when upgrading perl.
• Cache everything in the cache, so all .spec + LocalSite.cfg. This means no extra access, and is more or less what we have now. Difference would be that LocalSite.cfg would store configuration changes, and the cache file would store everything, so it's easier to upgrade.

George suggestion is exactly the kind of thing I was aiming at: have something like a recipe of how to install a plugin / extension / contrib / ... in a small file, preferably a JSON one. Then have something (configure, extension_installer, ...) download this file and do it. Basically, it would contain:

• List of files contained in the extension (MANIFEST)
• md5sum / SHA1 / GPG signature of the files - to check their integrity before upgrading / removing
• List of requirements needed to make this extension work (DEPENDENCIES) - Ideally, several formats could be given, so it's suitable for Debian repositories, CPAN installation, RPM ones, Gentoo, etc...
• Extra configuration (.spec)
• Version number / release information

So basically, what I propose is to merge stuff, and cache the configuration somewhere. And use some decent language for all of this, so NOT perl (it's highly insecure, even though it can come in handy sometimes, but I don't think anybody ever put some perl code inside the LocalSite.cfg), and not another home-made description language. I propose JSON because it's what most webapps use nowadays, but I would be fine with YAML or even XML if we really have to.

Pushed the date by 1 month because I had too much work to work on it

So yes it's ambitious, but I don't like to do half a patch if it's possible to go all the way.

-- OlivierRaginel - 25 Nov 2010

Adding myself as another developer. I fully support this. Merging together all of the extension metadata will enable a much improved extension install / update / remove process. I also agree with Lavr that whatever we do in extension packaging should not be allowed to negatively impact the Foswiki core performance. But this improvement in packaging, along with moving extensions to some sane revision numbering scheme for dependency purposes are badly needed. The refactored Package.pm in 1.1 got us part way there. The *_installer file is mostly treated as metadata now in 1.1. Now we need to agree on a better format for the package and finish merging the functions.

-- GeorgeClark - 25 Nov 2010

OlivierRaginel, I would like to dispel your allusion that 'but I don't think anybody ever put some perl code inside the LocalSite.cfg'. Quite a number of people have put perl code into LocalSite.cfg. I strongly hope they've moved this to that LocalLib.cfg instead, but its simply not true that it isn't done.

I do agree that it would be better not to allow it however

Additionally, some quick profiling has made it clear to me that we should start using JSON::XS whenever we can, rather than the JSON module. This means we should start having a requirement on JSON::Any - which will use the 'best' one available....

-- SvenDowideit - 25 Nov 2010

From some IRC discussions, could the "package" be defined to "install" the =_Installer= package along with the extension.tgz file into some location within Foswiki, and then run the Foswiki extension installer to actually map/copy the files into the running Foswiki installation. This would accomplish a couple of things:

• Detect and warn if topics have been locally modified.
• Properly "check-in" topics that have local modifications, (following the noci directive as appropriate) preserving RCS revision history
• Merge in package spec's required to properly integrate extensions into Foswiki.
• Support renaming of Webs and Topics, which I suspect may become more important as we add better I18N support.

The configure should be told / detect that Foswiki is installed by a package manager and disable the "install" feature of FindExtensions. However it would still be useful as a report to show the latest versions on Foswiki.org (and points out old repo's)

-- GeorgeClark - 27 Nov 2010

If the feature is implemented so the many .spec files are cached with LocalSite.cfg in order not to read maybe 20 extra files for each page view then my concern is gone.

The commit date is used for timing the 14 days to raise concern. Not for telling the world that you commit to do it. We have 1+ year old proposals that have been accepted and just wait to be implemented.

I flip the date back to not goof the decision system.

Unless someone else raises another concern this proposal (with the change that files are merged to a cache) will be accepted in a few days by concensus

-- KennethLavrsen - 27 Nov 2010

Sven point is right. It's true I worded my sentence as if nobody ever did that. I more meant that they shouldn't, and that the few abuse I could think of I haven't seen (like VirtualHostingContrib could have been just a bunch of lines in LocalSite.cfg :)). It's true however that we should make this clear, and define what's allowed in LocalSite.cfg (recursive expansion is a must, but do we need anything else?).

Kenneth's point that this shouldn't bring any slowdown is right, and Sven's point of using JSON::XS when available should help in getting this maybe even faster that it used to be.

Thanks for explaining me the Date of commitment signification. I got fooled by its name

-- OlivierRaginel - 27 Nov 2010

I'm removing my name as a CommittedDeveloper, and will push this off to Release 2.0, It's rather late to do major changes for 1.2. Plus it probably needs some tweaking if we plan to eliminate the requirement for configure.

-- GeorgeClark - 02 Apr 2014

Developer appears to have left the project, and a lot has changed in configure and .spec file. Flipping this to a parked proposal until it gets adopted.

-- GeorgeClark - 18 Nov 2015

Commenting is disabled while not logged in