Feature Proposal: Moving hash based configuration into external file

Motivation

Now configure stores configuration data as perl-script. This is hard to manage and it is loaded with require. This causing the need reload server under Modperl (and PSGI) environment. The main point is - reloadable configurations without server restart.

Description and Documentation

IMO would be much better (and easier manageable) when the configuration was in the YAML file. YAML is widely accepted and well-known format, easily editable with different sw tools (or with simple vi).

Alternatively, JSON or XML.

As another enhancement, the development (testing) installation should check the load-time and the last modification time of the config.yaml, so the engine can reload the config when it is changed - without sever restart. (this can be done with some OS file-change-notification support too.)

When considering XML, should consider Apple's plist serialization format (XML file), see Wikipedia article and Apple page.

For all 3 format are already here CPAN modules.

Plist advatange is:

• well tested and well defined DTD
• DTD allow store indexed arrays (think as Array of Hashes)
• here is are already module in CPAN
• Mac users has nice gui-tool
• for everyone others, who are not familiar with plist, it's like a simple XML file
• but, bloated

YAML:

• well known
• simple ascii
• space-saver
• but, the format is indentation based, so very sensible to edit errors

JSON:

• allow direct read by javascripts, but this is not really advantage for the FW config
• but, ugly and very error sensible

The configuration should be stored in more files, not only in one - in the ConfigDir. One for CORE, and several others for plugins. This allow easy management for plugin configurations. Loading configs from more files taking more time, but this happens only at server start or config change, so its not meaningful. So,

• ConfigDir/Foswiki.plist - for core
• ConfigDir/ImagePlugin.plist - for ImagePlugin
• and so on...

Constant monitoring for configuration changes for automatic reload (e.g. for development), is possible for example with: Filesys::Notify::Simple or File::ChangeNotify.

Problems

Storing config in a file does not allow referencing an early config value without additional postprocessing, like:

$Foswiki::cfg{Htpasswd}{FileName} = '$Foswiki::cfg{DataDir}/.htpasswd';

YAML has a sort of crossrefs, like:

master: &id01
  path: /some/where
mypath: *id01

but is is ugly.

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: JozefMojzis - 25 May 2012

Discussion

How are the relative performance of these. The hash is ugly but is pretty fast. Would you map the external storage back into the hash for compatibility? (the config hash is referenced everywhere.

The load overhead is minimized only for those running persistent perl. Although it's a best practice, there are a lot of sites that still run plain old CGI. This would probably hit them pretty hard - especially merging multiple config files. It's why the plugins all have defaults in their .../Config.spec files, but they are merged once by configure instead of during runtime.

-- GeorgeClark - 25 May 2012

I don't agree that a perl style cfg is hard to manage. And the other options you give are just as brittle.

So if your reason to change everything, requiring lots of retooling all, is just reloading - why not jut reload the cfg?

and then I go look at the code, and notice that we use .... do.... which means the code is already capable of reloading, you'd just need to trigger it....

-- SvenDowideit - 26 May 2012

While I agree with George's concerns about the performance impact, Sven, try manage config file not from foswiki's configure... but by external app. Foswiki can be (and IMO should be) not standalone, but as a part of bigger ecosystem, and therefore config should be not "perl specific". Think a bit above foswiki horizons. One of general foswiki problems is, than everything inventing yourself and ignoring usual best practices beyond of perl's world - with all impacts what can cause in its integration.

And in perl too - generally - config files be never should do ed, because of security problems, especially, when the config should be manageable outside of perl...

-- JozefMojzis - 26 May 2012

don't forget - there are already lots of tools that are used to manage the existing cfg file.

my point is that each of your reasons (other than the not invented here issue) may well not outweigh the damage it does to the existing toolsets used to manage foswiki.

I already do manage the config file from outside foswiki's configure, and its likely other deployments do too.

but the real point is that none of the other options are very much better.

-- SvenDowideit - 26 May 2012

About what tools you talking (mean now FW tools). Is here some standardized Perl-hash managing tool? I'm happy when I can learn something new.

When the current is ok, why everyone who running FW under modperl (or any persistent perl) must reload the whole FW?

I can manage my configs with standardized tools and write an simple conversion tool to LocalSite.cfg format, and merging different config.spec files and what will allow me testing easily different configs and so on. But what is really bugging is the need reload foswiki when the config is changed.

-- JozefMojzis - 26 May 2012

I am with Sven here.

The only best practice I know of config files is plain ASCII so you can hack then with a text editor. We have some geeky things in our perl config but most is plain simple variable = value lines

None of the fancy XML etc is easy to hack. Yes, hard core programmers think it is. And they know all sorts of fancy tools you have used before that can hack XML and all sorts of fancy so called standard formats.

But most admins and installers are not programmers and they do not find it easy to edit XML and similar.

I personally do not find it easy to edit XML files. Give me plain simple variable=value configs. That I can understand.

I don't know what eco systems need to hack a Foswiki config file. That must be something rather geeky for the few. It would be good to see some examples that are of general interest.

The cost of changing format is very very high. Too high in my view

The config variable are used in almost every bit of source code we have. We will have to rewrite everything we have

All the plugins use these variables. Some of these variable are API! Plugins depend on them. Changing the format means that ALL plugins must be rewritten. Also the private ones that exists in many companies.

We are majorly breaking the commitment to keep a stable API by changing format for the part of the config that is documented to be used by plugins.

There is nothing the end users get out of such a change on the positive side. We will break everything. We will cause a major code rewrite. Our users and admins will hate us. Plugins from the old project will be even more difficult to port to Foswiki. Performance will be the same or worse. It is all negative.

Only positive is that the format will follow one sort of standard and a very few people can make a tool that changes the config for a purpose I do not get. You don't normally need to alter the config once a Foswiki has been in production for a while.

Sorry. The benefits are microscopic. The cost is enormous. If this was a new project or the benefit was doubling the speed of the code it would make sense to consider something different.

I can also tell you that we have heard many admins that have implemented small tailorings by adding a few lines of code in the config file. They will also hate us. Concern raised also from me.

-- KennethLavrsen - 26 May 2012

Jozef - thats what I was trying to get at - file format change is a big distraction from the real need and solution:

But what is really bugging is the need reload foswiki when the config is changed.

this is a feature req that is worth doing. right now, Foswiki::Configure::Loader calls a do on the LocalSite.cfg, but there's no mechanism to kick an existing process into reloading.

clearly, we need that. (I find it a little annoying to need to kick all 12 fastcgi processes too..

(perl hash mgmt tool - yes, I use perl. and sometime just awk and sed)

-- SvenDowideit - 26 May 2012

I was pretty sure I remembered coming across some code in the Engine or somewhere that attempted to detect that LSC had changed and reloaded it. But I cannot find it now. That would be nice to have, provided the check for changes doesn't add too much overhead.

Another point that I'm surprised that Sven didn't mention - The configure command really ought to be usable from the shell, as the external interface to the configuration. It's needed for the Debian package installer to set config variables for example.

configure is a lot more than a simple file editor. It does cross-validation of parameters, Conditional dependency checks, merging of defaults, general sanity checks, and many other little tasks. If a big effort is being undertaken, it would be preferable to make configure or a shell equivalent the external interface into the configuration rather than getting hung up on the internal file syntax. Even if the storage format is changed, we still need a shell configuration tool.

-- GeorgeClark - 27 May 2012

Maybe my english is too bad - and can't express myself enough clearly. I don't want change all source code. The current Foswiki::cfg should remain. SImply, need add one YAML import module, what will convert YAML file into the current %Foswiki::cfg. Replace do "LocalSite.cfg with loadconfig("LoaclSite.yaml"); Not the internal hash Foswiki::cfg is bad, the solution: (do -ing) is wrong. Any perl best practice will tell you using at least "Safe.pm" when do -ing any piece of code and generally - it is simple a bad practice.

If you don't like XML (what is strange, when you positioning Foswiki as enterprise tool), here is the YAML. YAML (i hope you agree) is easily editable.

But this is the place where i need say: This is one of a major problem of FW: the majority of the team assume FW as an hackable tool for programmers and, not as stable, standard-conformant tool for endusers. IMO, Foswiki should not be primary "hackable" tool but easy to use tool.

Reloading server when the config is changing, isn't an typical representative for an easy to use tool. Perl code as config - neither! You can disagree with this, but ask users - who trying implement foswiki and need read his config by external sw.

The solution is: kick out do -ing perl code, and replace it with a short module with YAML::Tiny::LoadFIle($config). If don't want do it (for example for performance reason - what I can understand and accept) - ok - one more thing will remain where FW doing "standard things" by "non-standard way".

As I telling above, the loading config is no problem for me - i can hack FW myself for loading external YAML and converting it into %Foswiki::cfg.

But, would be nice make a solution for the server-reload.

-- JozefMojzis - 27 May 2012

George, I think that code is in FastCGIEngineContrib.

It's not unusual for a config file to be in a programming language. For example, the WordPress config file is in PHP, Buildbot in Python. It is often compiled languages, like Java, which need to use XML, etc for config files.

I think the days of XML being required for enterprise are over.

-- AndrewJones - 28 May 2012

I like both approaches:

• add a comandline mode to configure
• use YAML to load the Foswiki:cfg

Both make total sense. Both fit slightly different purposes.

Editing perl code to configure Foswiki is scaling rather high on the geek scale, doesn't it. For my taste even higher than any XML what so ever, even more for people unfamiliar with perl. But we are talking about YAML here now, and that's a lot better and leaner anyway.

What we gain is:

• more robustness,
• approaching industry standards and
• easier to use tools.

That's worth the efforts.

Most of the work seem to be impacting configure but not the core engine as this will still see the good old Foswiki::cfg hash, in theory. There might be difficulties in practice as this idea is tested out hands on.

With regards to having to reload a persistent foswiki process as a consequence of a changed configuration: as far as I know other servers do that as well. The only point here should be uptime of services to guarantee a safe transition from one state to another without disrupting the overall services. Presumably that's more complicated than most other servers realize too, resulting in temporary downtimes, or slowdowns. But Foswiki isn't in high availability in the general case, so I am quite sure we better leave that aside for now.

-- MichaelDaum - 28 May 2012

I have not yet heard a single argument that can be documented.

You use big words like "Enterprise" and "Robust". And you ignore all the trouble. Like requiring plugins using config having to change. The security part is rubbish. If configure converts YAML to Perl config code I can still break into the server and hack the converted perl file.

I am sorry guys. This is a whole lot of trouble that neither endusers, nor admins, nor plugin maintainers will get any benefit from.

We have a long list unattended features requests pending attention. Features the endusers will see and love. We have plenty of need of help with the new storage that should enable Foswiki to scale. We have unicode project that lacks resources.

I do not understand proposals like these that are more of the nature "i need to hack something" and "this new lib is cool"

-- KennethLavrsen - 28 May 2012

The foswiki configuration is a simple hierarchical hash. You could store such a hash using many methods - the choice of storing it in an executable perl file is simply convenience i.e. perl provides a robust, high performance parser, why not use it.

However storing the configuration is possibly the most trivial aspect of this discussion. A small piece of seagull shit on the tip of the iceberg.

Configuration in Foswiki is much, much more than the storage format. There is a huge amount of work in the configure engine, providing checkers, documentation, dependencies etc. Of course configure is not bound to storing the config in a perl hash; it could equally work on a YAML file. It does not require editing perl code.

The only reason we haven't changed the storage format - and I seriously considered doing so when I first wrote configure is that there is simply insufficient reason for changing. Industry standards on their own are not enough. AFAIK there are no existing "easier to use tools" - configure is way ahead of any perl tools I have seen for interactive management of configurations. There is no justification for claiming that any other format would be more robust - otherwise I would expect, at the very least, a list of severe bugs against configuration in FW. Against this, consider that it would be a work of hours to:

• Work out how to monitor and reload a FW perl configuration in a running system

Then, if you are feeling motivated, you might tackle:

• Use core functionality of configure to write a command-line configuration tool,
• Tie the configuration hash to an abstract configuration store in configure, which would allow you to use any underlying configuration engine below configure.

This seems a lot simpler than completely re-implementing the entire configuration infrastructure for a piece of seagull shit.

-- CrawfordCurrie - 28 May 2012

Guys, the above is not a hot issue. It's a proposal - quo vadis. Of course, here is much-much hotter issues, like utf8, PSGI etc.. (btw, I asked a couple of questions in UseUTF8 and looking for answers).

So for ending this, summarizing:

• Foswiki reading config as next:

    # readConfig is defined in Foswiki::Configure::Load to allow overriding it
    if ( Foswiki::Configure::Load::readConfig() ) {
        $Foswiki::cfg{isVALID} = 1;
    }

• the above fragment is from lib/Foswiki.pm. Clearly state, than it is designed for override!
• The major redesign is in the configure itself, e.g. Foswiki::Configure::Load::readConfig
• noboby want change the good old runtime config-hash ($Foswiki::cfg), only its storage format.
• so, the redesigned Foswiki::Configure::Load::readConfig in short:
  • will read config.yaml
  • convert it into $Foswiki::cfg, and continue as currently, so
  • do usual checking and so on..
• nothing more, so any plugin what uses the API Foswiki::Configure::Load::readConfig not need change even a bit - just nothing. So here isn't any compatibility problem.
• plugins, what uses directly the file lib/LocalSite.cfg - are bad by design. Plugins never should require LocalSite.cfg directly - here is defined API for it.
• here are plugins what uses lib/LocalSite.cfg directly:
  • BuildContrib/lib/Foswiki/Contrib/Build.pm
  • FastCGIEngineContrib/lib/Foswiki/Engine/FastCGI.pm
  • MigrationScriptsContrib/tools/MigrationScriptsContrib/CSVDemo.pm
• and ofcourse
  • Foswiki/Contrib/Build.pm
  • Foswiki/Configure/*

I hope i don't missed any. So concerns about plugins are unnecessary.

• all those changes are not hot and need a bit more thinking about it - in the both side..

I don't want talk more about the standards. If someone don't even want seeing them - I can live with the current. As i already told x-times, i can manage the changes myself - locally. (and again: this is not a hot issue, nether for me).

As Kenneth clearly stated - here is much hotter things, e.g. utf8 - so, please, go discuss to UseUTF8.

-- JozefMojzis - 28 May 2012

Jozef, just for the records. Foswiki and TWiki discussions around proposals have always been like this.

You will most likely get answers from community members with a quite conservative mind set.

That's of course because we have a long history behind the project and a need to protect investments. At the same time we seem to lack resources to experiment more. This in effect might end up not resulting in the best technical decisions to emerge.

This in itself is an issue for the project's health.

In this sense I'd like to remind everybody else in this round to be a tiny bit less conservative for the sake of encouraging new experiments.

Telling people that there are more important things to do sounds a bit odd on an Open Source project. Contributors donate their work on whatever they like to do. Of course this all needs coordination from the elder ones. But mind you not to discourage contributors on these kind of discussions. This is so so important, even more as we have quite few people given the code base and history we are managing within Foswiki.

I personally like the idea to have a look at YAML and think about it in how far it could replace the current configuration system.

Using a more standard configuration system based on YAML is even more important as soon as you start deploying Foswiki and plugins automatically, for instance using the Debian packaging mechanism. That's the reason for this proposal as far as I understand, a need that you can't just ignore.

There are shell-scripts from hell that hack the perl code in LocalSite.cfg, but one can't seriously call this best practice.

YAML is one possible solution with many other advantages and disadvantages at the same time.

A cmdline interface to configure would be cool as well and would find its main use in auto-installers.

-- MichaelDaum - 29 May 2012

Commenting is disabled while not logged in