PendingConfigureWork

Pending work on configure

_Moved from ReleasePlanDiscussion02x00 _

To understand configure requires a brief history lesson.

• In the beginning, there was testenv. This checked the server environment and did nothing else.
• We re-engineered testenv to create configure, maintaining as much code as possible from testenv for checking the CGI environment. The general architecture is described in the POD comment at the top of Foswiki::Configure
• Configure tries to do as much as possible in the Foswiki namespace. There were two reasons for this; first, because it was easier to re-use testenv code that way, and second, because it mirrors the 'real' Foswiki environment, making checks easier and higher value and allowing code sharing between configure and the mainstream Foswiki scripts.
• The main value of configure exists in the checkers. These process the server environment and Foswiki::cfg to check that the settings are compatible. Everything else is fluff.
• The secondary value of configure is in the UI that allows the changing of values in Foswiki::cfg and their checking.
• Our original design intent was that the checkers would be entirely decoupled from UI concerns. A Value object would be provided to the checkers that would encapsulate UI. In recent changes, this intent has been missed / lost, and UI concerns have bled heavily into the checkers.

configure has had a lot of interesting work over the last couple of years as a part of AJAXOnDemandCheckersForConfigure. However:

• now require JS. This is Not Good.
• a lot of new interesting stuff has been done thatt wasn't covered in the feature proposal. The scope of the changes is such that detailed review is required to ensure we're implementing a new architecture we'll be happy with for the next few years.
• CrawfordCurrie is very concerned that the newest code has introduced a raft of problems to the code architecture. The problems are best summarised as:
  • Pollution of concerns. Checkers are much more closely engaged with the UI than before, and know far too much about how configure works,
  • Spaghetti code - check calls provideFeedback which calls check which calls provideFeedback,
  • Extensive use of globals, violating encapsulation,
  • Wizard code (e.g. email setup) is conflated in the Checkers, making it very difficult to decouple cleanly without breaking something,
  • Very little documentation of new features, and much of it is incomplete or at least wildly inaccurate.
• UI management has become very complex due to the use of AJAX, which has been bolted on in a rather unstructured way.

These problems make it very difficult to abstract out calling checkers outside of the configure framework.

It's pretty clear configure has lost sight of its fundamental requirement to be a tool to make Foswiki operational and has become almost a product unto itself. It's just too complex and heavy to be the first thing a new user to Foswiki sees. And there are just too many little things that go wrong: strange validation errors, empty dialogs, and other little quirks that are certain to send a new user into a tailspin.

Work with ConfigurePlugin has shown that, with minimal changes to the core, checkers can be called from a plugin in the running Foswiki environment. This means that 99% of the functionality of configure can be done from the plugin - vastly simplifying auth (admins only), AJAX (JQueryPlugin and REST handlers) and JS functionality (JQueryPlugin).

So what's the plan?

The long term vision is to break down configure into two phases

1. st. Initial basic configuration that gets a Foswiki up-and-running.
   • No configure script required.
   • All files in their default release locations.
   • Foswiki starts up with no LocalSite.cfg (LSC) or LocalLib.cfg (LLC). .spec files provide sensible defaults, and/or code works them out on the fly using lightweight sniffer code.
   • No external services required (use PlainFileStore, no email etc) as far as possible.
   • Nothing under Foswiki::Configure is called (except to try to load LSC, and read Foswiki.spec iff LSC could not be found)
2. nd. ConfigurePlugin takes over management of the LSC
   • Handles complex or slow checks, such as file permissions
   • Invokes Wizards as appropriate to assist configuration of complex items, such as email, database stores etc.

To get there from where we are today, we need to refactor configure so that the component parts are decoupled and reusable. This requires us to identify and isolate components as:

1. "Ainimal" configuration (configure, getting started, absolute minimum of checkers that always run) from "refinement" configuration (ConfigurePlugin, tweaks etc, checkers run on change)
2. Simple checkers, associated with single keys, with no UI or deep check capability
3. Deep checkers, associated with single keys, which perform more complex analysis and may assist in simple, non-interactive guessing of suitable values for configuration items, based on the environment
4. Wizards, which are a step beyond Deep Checkers and help the user select complex configuration settings for multiple keys.

All of these things need to be invokable using AJAX calls. Initially we will keep all this coupled into the configure monolith, but check continuously that we are being honest about APIs by ensuring ConfigurePlugin is also able to invoke them. We must also:

1. Remove all pollutants from the Foswiki namespace
2. Minimise external requirements to achieve a minimal configuration (e.g. email is not essential and should not block)
3. Move checkers to the Extension packages where they belong, creating extension packages as appropriate

Status and resources

List of status points and important resources (such as lost topics)

• 10/6/14 - GeorgeClark, JanKrueger and CrawfordCurrie had a design discussion on IRC to talk about CrawfordCurrie's work on the code so far
• 11/6/14 - CrawfordCurrie refactored this topic

The Gory Detail - Design Discussions

Initial User Experience

Regarding the initialization case where a LocalSite.cfg does not exist, I've been trying to follow the BEGIN blocks and the initialization functions of ConfigurePlugin. What about the following process when LocalSite.cfg does not exist:

• Call a ConfigurePlugin::Bootstrapmethod.
  • This is a pluggable architecture that attempts to create a minimal 'Starter' LocalSite.cfg.
  • Bootstrap::Spec Merge in the Foswiki.spec / Config.spec files
  • Bootstrap::PathsAttempt to derive the minimal URL path settings from the url used to load this page, possibly "sniff" for a pub path, etc.
    • Need to deal with bin/view may have redirected to (empty) in the event apache is already set up for short URLs.
  • Bootstrap::Platform::<platform> to deal with specific platform quirks. Location of rcs, file temp issues, etc.
  • Bootstrap::Password (generate an initial admin password to protect configure and provide an admin id)
  • Bootstrap:* Other possible initialization that may be necessary, pluggable to handle unusual configurations ... I can't think of any right now.
  • Write out the starter LocalSite.cfg
• Perform a sanity / dependency check and build a static html response:
  • Report the important parts of the Bootstrap process: (expected URLs, paths, initial password, etc.)
  • Dependency summary for missing dependencies
    • ex: HTML::Parser missing: Wysiwyg editing will not be operational
  • Recommended next steps
  • Link to reload homepage - Disable if critical missing dependencies?

-- GeorgeClark - 05 Jun 2014

Email wizard

Discussion for moving this into a separate wizard function and out of configure. Note that I'm still confused as to where this exists. When you say it moves out of configure ... exactly where?

• Part of configure (today)
• Still in configure but a separate rest handler, crafted html forms or some other interface. (No idea how to do this.)
• A Foswiki topic through a combination of forms, macros and a configure rest handler. (I have security concerns)

The reason this exists is because time and time again sites end up with non-functional email.

• Configuration is complex. Servers listen on 3 possible ports, supporting multiple protocols. Some require ssl, some require login, And Login uses different authentication methods.
• Dependencies are complex
• Getting it "right" is difficult.

On a unix platform, just defaulting to the MailProgram sendmail just works provided that the underlying platform has a functional email configuration. This is not necessarily always the case. The other option is the Perl modules

When it doesn't work, debugging is difficult. Yet again this week I spent a couple of hours on IRC helping someone get email working. And then they fell into the short URL mis-configuration. So we were doing really well for a newbie.

So for restructuring into an email wizard. (And documenting some of what the autoconfigure button does today)

• General dependency check. No sense presenting options if they can't be used. Only show what is not available.
  • [ Net:SMTP is not available - Foswiki will attempt email using the native unix email clients ]
  • [ Secure protocols are not available - Email will fail if the server requires a secure connection ]
  • [ Authentication modules not available - Email will fail if server requires a login ]
  • [ IPv6 dependencies are missing - Email will fail if server requires an IPv6 connection ]
  • [ Certificate / S/MIME dependencies are missing - Email cannot be signed ]

• Enter your administrator email address. This address will be used for three purposes:
  • Email generated BY foswiki will be send FROM this address. (Registration emails, change notifications, etc.)
  • Admin notification email will be sent TO this address.
  • This address is shown in the contact links for the site webmaster.

• Do you want foswiki to validate the email server's certificate. (Not typically enabled). Enable this option if you want foswiki to validate that the server where you connect to send email is who it claims to be.

• Do you want foswiki to cryptographically sign the generated emails. (Not typically enabled). Enable this option to add a signature to the emails generated by Foswiki.

• If your email server requires that the sender log to identify themselves, enter the userid and password here:
  • Userid
  • Password
  • Dependency check for Authen::SASL and Authen::NTLM

• Enter the hostname of the outgoing email server

• Autoconfiguration wizard: Click this button to perform checks to attempt to configure your email server:

• Probe the mailhost using multiple options.
  • Determine if the hostname is reachable, IPv4 and/or IPv6
  • starttls with cert validation. (587 and 25)
  • starttls without validation. (587 and 25)
  • tls with cert validation (465)
  • tls without cert validation (465)
  • ssl-v3 with cert validation (465)
  • ssl-v3 without cert validation (465)
  • smtp ( 587 and 25)
• For the first server connection that worked:
  • Attempt to login with provided userid & password
  • Attempt to send a test email.
• If no connection was successful, or the test is not possible (no mailhost configured)
  • Probe for to find and identify any mailprogram
  • Attempt to send a test email using the mailprogram

-- GeorgeClark - 10 Jun 2014

---

Unrefactored Discussion

So what do we do for 1.2?

• Can we have a sprint to make Foswiki run without configuration for basic functionality?
• move the top 10 config requirements into webforms
• let configuration remain as is for the advanced stuff

Would something like this be doable over the next 3-6 months? I suspect that due to the complexity we'd be just as bad off trying to fix the current code. Other options- revert to 1.1 but that would be a big effort as well to then re-fit the important 1.2 checkers.

-- GeorgeClark - 03 Jun 2014

3-6 months, that's a long time. I'd rather see 1.2.0 out much earlier.

I am all set for a configure sprint where everybody hammers the tool and tries to get a deeper understanding of the code during the sprint.

All it needs is a date and a commitment by people. From there on I am pretty sure that we will be able to fix the remaining bugs pretty fast, i.e. it doesn't take more time but more eyes and hands.

A couple of above questions target more a long-term perspective on configure. For now I decided to refuse considering alternatives - no matter how much I may currently lean towards one or the other you already mentioned - until 1.2.0 is out using a configure based on the current trunk code, only fixed.

-- MichaelDaum - 03 Jun 2014

I have a many-changes checkout that fixes MANY of the problems with configure (except the godzilla complex) and would at least be a better platofrm for moving foward with ConfigurePlugin. However I would be nervous of checking anything in without the full, committed support of at least 2 other developers to help me fix any problems that might emerge with it (and support my view that it is better to drop esoteric functionality than it is to try to support bad code)

-- CrawfordCurrie - 03 Jun 2014

I'd be willing to help. I'd like to understand though what "esoteric parts" you plan to discard. "Esoterica is in the eye of the beholder"

I ran through an initial install "user experience" to see what happens: (This probably ought to be in another topic somewhere)

I cleaned my trunk install, pseudo-installed CpanContrib, and ran configure from scratch to see what's going on:

• Configure ran without any additional dependencies
• Set of paths went well, no issues there.
• Full configure reported several errors:
  1. Error: You must select a store implementation: This appears to be a bug. The Foswiki::Store::RCSWrap was selected, but the error still is reported. I think it's ordering on the initial guesses.
  2. Error: The system umask (077) is not compatible with the configured directory and file permissions. That's due to my system running under apache suexec. So it's a valid error.
• Error: Missing Dependency: HTML::TreeBuilder.
  • I'll add this to CpanContrib.
• Warning: '' does not appear to be a valid email address. Fair enough
• Error: Mail server specification required: Message might be a bit clearer. "Enter a mail server address in the format ....."
• Error: Either or both {Email}{SSLCaFile} and {Email}{SSLCaPath}. This one is esoteric, {Email}{SSLVerifyServer} should be disabled by default IMO. SSL server certificate validation may be a best practice, but it probably ought to be optional.

Okay, on to Email setup:

• Entered email address. Error auto-cleared. Entered a hostname, error auto-cleared. Entered userid & password and clicked Auto-configure:

Net::SMTP and IO::Socket::SSL are required to autoconfigure mail
Attempting to configure a mailer program
Unable to locate a known external mail program
Identified sSMTP (/usr/sbin/ssmtp) as your mail program
Configuration accepted.

• Good, so this worked, but Net::SMTP and IO::Socket::SSL probably ought to be in CpanContrib. Let's send a test email.
  • Fail: [<-] 535 5.7.8 Error: authentication failed: no mechanism available /ssmtp: Server didn't like our AUTH LOGIN (535 5.7.8 Error: authentication failed: no mechanism available) So this is an issue with my local ssmtp configuration. I normally use Net::SMTP.

-- GeorgeClark - 03 Jun 2014

And now we go downhill. The email auto-configure requires SSL, especially if it's going to work on Google's gmail. We are missing a dependency, Net::SSLeay, but that requires OpenSSL and is unavailable in a pure perl implementation. Unfortunately configure lies, and only reports that Net::SMPT or IO::Socket::SSL are missing. So we cannot add this one to CpanContrib, and it's complex to install, especially on windows. So, we need to:

• Add Net::SSLeay as an optional dependency.
• Update the sniffer to still try non-SSL email if it can't handle use SSL.

Moved comments on other CPAN dependency issues to the MoveCPANLibToSeparateContrib proposal.

-- GeorgeClark - 04 Jun 2014

Allowing any Admin to edit the configuration raises a concern for me. There are "Wiki" administrators and "System" administrators. The "sudo" login is the System administrator, AdminGroup are the Wiki admins. I wouldn't think that everybody in the AdminGroup has the ability to view / edit parts of the configuration. I believe that anybody that modifies LocalSite.cfg should have some higher level of authority than the AdminGroup.

Another concern. We've seen some serious pushback for even having a sudo login. Corp. types view that as a "back door", a shared secret that can be used to gain access to the Wiki without really identifying who you are. Timothe addressed that by detecting if configure was accessed using external auth (http or certificate) and allowing the sudo password to be removed. This is an important consideration ... not sure how to address. it as configure moves into the Wiki / rest realm.

-- GeorgeClark - 05 Jun 2014

In my local checkout I have a mostly working version of configure that runs with the 'current' interface but has the checkers re-encapsulated so that they have no UI concerns. I have re-done the checker parts of the UI so that error message generation is delegated back to the UI object (the Value). This allows drop-in replacement of the 'core' Value object with the 'plugin' Value object. This has involved the loss of some checker functionality - specifically where error messages from other checkers were parsed to determine error statuses of other checkers (more pollution of concerns).

I'd really like to discard most of configure and focus instead on a plugin-based architecture for configuration. ConfigurePlugin works using REST handlers, which generate JSON representations of subsets of the configuration tree and support checking leaf entries in the tree, in the context of a working Foswiki. The main issues with this are:

• how to generate a "minimally working" foswiki that ConfigurePlugin can run with to complete configuration
• the lack of explicit dependencies between certain items, where these dependencies have not been explicitly coded. For example, in the general case, a default setting of Foswiki::cfg{Store}{Xyz} may stand on it's own and have a simple checker. However, an optional plugin or contrib may depend on this value, and have a suite of checkers that need to be invoked if the value of Foswiki::cfg{Store}{Xyz} changes. There is no way to express this sort of dynamic dependency at the moment.

Here are some goals for configure / ConfigurePlugin (refactored above)

-- CrawfordCurrie - 05 Jun 2014

OK, let's postulate the following:

1. We maintain a working configure as far as possible, while modifying/abstracting the checkers so they can be used from 'elsewhere' (wherever that may be)
2. We create a ConfigurePlugin/tools/bootstrap script that does the bootstrap process you describe, re-using or, if necessary, transplanting code from Foswiki::Configure
3. ConfigurePlugin picks up the slack of calling the checkers etc.

-- CrawfordCurrie - 05 Jun 2014

Generally I'm in complete agreement. For now configure remains stable, if we get too aggressive we'll never get 1.2 out. We do need to keep making some progress on the most serious operational issues. The migration can happen over time as configure becomes less important.

I read that a tools/bootstrap implies that this is a shell tool. That won't fly on many hosted sites. I believe the bootstrap has to be web based.

• A bin/boostrap script would resolve some of the shorter URL snooping challenges, but it involves changes to the apache configuration, and a separate script that should probably be deleted after foswiki is running. (Some other web tools I've seen take that approach, removing the bootstrap script after initial configuration). Leaving it around is one more script that could potentially be attacked.
• Testing if LocalSite.cfg exists very early in Foswiki.pm and using the Bootstrap::xxx classes to build a starter config reduces the need to add to the Apache / Nginx / Lighttpd configuration. It might even be advisable to fire off the bootstrap from the bin/configure script, and eliminate the special early pass from configure.

If this works, then regardless of whether you first visit bin/configure, bin/view or even just Main/WebHome, if LS.c doesn't exist, the bootstrap gets control, builds what it can, returns a static page with the sanity check, (is even CGI available?) and links to either visit configure, or go try foswiki.

-- GeorgeClark - 05 Jun 2014