You are here: Foswiki>Tasks Web>Item12952 (05 Jul 2015, GeorgeClark)Edit Attach

Item12952: Move Configure into a Contrib

pencil
Priority: Urgent
Current State: Closed
Released In: 2.0.0
Target Release: major
Applies To: Engine
Component:
Branches: master trunk
Reported By: CrawfordCurrie
Waiting For:
Last Change By: GeorgeClark
There are no functionality changes to this task - it is an internal re-ordering of code.

configure has gone beyond the point of feasible manFagement - the configure code alone is a significant part of the Foswiki core, and has been recorded elsewhere, configure is overkill for the job it has to do.

Preparatory to splitting the configure code up into more helpful, smaller "wizards", we are moving the Configure codebase into a contrib. This will make it easier to decouple (and test the core without the configure code in place). When the wizards are ready we will deprecate configure.

-- CrawfordCurrie - 02 Jul 2014

Looks like bootstrap process for a new install is broken. And there are still some issues showing up on trunk.foswiki.org

MichaelTempest initially reported it on IRC and http://pastebin.com/pVZTT9wb I tried to recreate with different results. I've pasted the two failures here. First Michael's then mine:

I have just done a fresh checkout of the master branch from github and a pseudoinstall. 
I have no LSC.  configure dies with this error, every time:
Use of uninitialized value $id in hash element at /home/mtempest/development/_allFoswiki/core/lib/Foswiki/Configure/TypeUI.pm line 46.

Relevant backtrace:
called from https://github.com/foswiki/ConfigureContrib/blob/master/lib/Foswiki/Configure/UI.pm#L246
called from https://github.com/foswiki/ConfigureContrib/blob/master/lib/Foswiki/Configure/Dispatch.pm#L435
The $stub does not have typename or keys, which are expected by Foswiki::Configure::UI::loadChecker()

Although the backtrace looks strange, he got the github checkout working by rewinding to prior to your big checkin:
(09:32:04 AM) gac410: If you un-install ConfigContrib,    then cd to core and   "git checkout a61b70abd7a7b0c26516340f200e9e67f4c6888d"       That should roll you back to before cdot's big checkin
(09:32:21 AM) mtempest: Thanks, I'll give that a go.
(09:33:14 AM) mtempest: I'll pastebin what I found since I'm not up to delving into the new configure code just yet.
...
(09:40:54 AM) mtempest: I hope cdot can make sense of this: http://pastebin.com/pVZTT9wb
...
(10:04:20 AM) mtempest: Thanks, gac410: that worked

and now for my attempt to recreate:

Software error:

Failed to load the perl module Foswiki::Configure::Dispatch. The module was found at /var/www/foswiki/trunk/core/lib/Foswiki/Configure/Dispatch.pm

Please ensure that:
   1 Foswiki::Configure::Dispatch is installed,
   2 that the module is available on the @INC path,
   3 that the webserver user (gac) has permission to read the Foswiki/Configure/Dispatch.pm file.
The detailed error seen was:
VERY BAD at /var/www/foswiki/trunk/core/lib/Foswiki/Configure/MainScreen.pm line 554.
Compilation failed in require at (eval 11) line 2.
BEGIN failed--compilation aborted at (eval 11) line 2.

And, one more issue from http://trunk.foswiki.org/bin/configure - General Paths Settings tab: below the {ScriptUrlPaths}{view}, the following is displayed instead of the checker results:
Assertion (view) failed!
 at /usr/home/trunk.foswiki.org/core/lib/AssertOn.pm line 28.
   Assert::ASSERT("", "view") called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Checkers/ScriptUrlPaths.pm line 53
   Foswiki::Configure::Checkers::ScriptUrlPaths::check(Foswiki::Configure::Checkers::ScriptUrlPaths::view=HASH(0x8035998a0), Foswiki::Configure::Value=HASH(0x802eb20f0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/UIs/Value.pm line 73
   eval {...} called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/UIs/Value.pm line 73
   Foswiki::Configure::UIs::Value::renderHtml(Foswiki::Configure::UIs::Value=HASH(0x803806348), Foswiki::Configure::Value=HASH(0x802eb20f0), Foswiki::Configure::UIs::Root=HASH(0x80380cac8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/UIs/Section.pm line 164
   Foswiki::Configure::UIs::Section::_renderValues(Foswiki::Configure::UIs::Section=HASH(0x803806480), Foswiki::Configure::Section=HASH(0x803a0b3f0), Foswiki::Configure::UIs::Root=HASH(0x80380cac8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/UIs/Section.pm line 65
   Foswiki::Configure::UIs::Section::renderHtml(Foswiki::Configure::UIs::Section=HASH(0x803806480), Foswiki::Configure::Section=HASH(0x803a0b3f0), Foswiki::Configure::UIs::Root=HASH(0x80380cac8), "") called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/UIs/Root.pm line 111
   Foswiki::Configure::UIs::Root::endVisit(Foswiki::Configure::UIs::Root=HASH(0x80380cac8), Foswiki::Configure::Section=HASH(0x803a0b3f0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Section.pm line 109
   Foswiki::Configure::Section::visit(Foswiki::Configure::Section=HASH(0x803a0b3f0), Foswiki::Configure::UIs::Root=HASH(0x80380cac8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Section.pm line 106
   Foswiki::Configure::Section::visit(Foswiki::Configure::Root=HASH(0x802ee0648), Foswiki::Configure::UIs::Root=HASH(0x80380cac8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/UIs/Root.pm line 73
   Foswiki::Configure::UIs::Root::createUI(Foswiki::Configure::UIs::Root=HASH(0x80380cac8), Foswiki::Configure::Root=HASH(0x802ee0648), Foswiki::Configure::Valuer=HASH(0x802d87ba0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/MainScreen.pm line 584
   Foswiki::configureScreen("") called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/MainScreen.pm line 89
   Foswiki::_actionConfigure("Configure", CGI::Session=HASH(0x801e53e88), CGI::Cookie=HASH(0x801e5b2d0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Dispatch.pm line 975
   Foswiki::dispatch("_action", "Configure", CODE(0x80169fdb0), CGI::Session=HASH(0x801e53e88), CGI::Cookie=HASH(0x801e5b2d0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Dispatch.pm line 206
   require Foswiki/Configure/Dispatch.pm called at (eval 12) line 2
   main::BEGIN() called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Dispatch.pm line 0
   eval {...} called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Dispatch.pm line 0
   eval 'use Foswiki::Configure::Dispatch
;' called at /home/trunk.foswiki.org/core/bin/configure line 92
   main::_loadBasicModule("Foswiki::Configure::Dispatch") called at /home/trunk.foswiki.org/core/bin/configure line 181

-- GeorgeClark - 02 Aug 2014

Parsing config specs is a lot less permissive resulting in an error saying:

Failed to load the perl module Foswiki::Configure::Dispatch. The module was found at /home/www-data/foswiki/trunk/core/lib/Foswiki/Configure/Dispatch.pm

What follows is an explanation / perl backtrace why things got wrong. I've been able to spot some brocken config specs actually. Still some more show up. For instance I can't have comments in PERL structures anymore. They seem to get flattened with lineendings being removed.

Fore instance XSendFileContrib has this

$Foswiki::cfg{XSendFileContrib}{AccessRules} = [
  {
      # Example 1: require change rights to download pdfs from Sandbox.TestUpload
      web => "Sandbox",
      topic => "TestUpload",
      file => ".*\.pdf",

      # test change rights
      requiredAccess => "CHANGE",
  },
  {
      # Example 2: grant access to thumbnails produced by ImagePlugin
      file => "igp_.*\.[png|gif|jpe?g|bmp|tiff]",

      # grant access unconditionally
      requiredAccess => "",
  },
];

Error message:

Cannot eval value '[ { # Example 1: require change rights to download pdfs from Sandbox.TestUpload web => "Sandbox", topic => "TestUpload", file => ".*\.pdf", # test change rights requiredAccess => "CHANGE", }, { # Example 2: grant access to thumbnails produced by ImagePlugin file => "igp_.*\.[png|gif|jpe?g|bmp|tiff]", # grant access unconditionally requiredAccess => "", },]': Missing right curly or square bracket at

Of course once lineendings are removed the perl expression becomes invalid. So let's better not.

-- MichaelDaum - 03 Aug 2014

Looks like another significant issue. Pseudo-install gets quite confused. Since extensions are free to install checkers into Configure, the first one to install creates the link for the Checkers directory. The rest get collisions. No idea how to fix this.
WARNING Confused by
     lib/Foswiki/Configure/Checkers -> '/var/www/foswiki/trunk/CommentPlugin/lib/Foswiki/Configure/Checkers' doesn't point to the expected place
     (should be /var/www/foswiki/trunk/ConfigureContrib/lib/Foswiki/Configure/Checkers)

Maybe core needs to ship Foswiki/Configure/Checkers as a skeleton directory, so that ConfigureContrib and CommentPlugin and others then link into that directory?

-- GeorgeClark - 03 Aug 2014

Never mind. Adding lib/Foswiki/Configure/Checkers/Plugins/.keep to core resolves the pseudo-install. And we don't need to ship the file. Git doesn't track empty directories, so we need a file in it.

-- GeorgeClark - 03 Aug 2014

Here is one. I can't get into my trunk test system locally:
Use of uninitialized value $Foswiki::cfg{"DefaultUrlHost"} in concatenation (.) or string at /var/www/foswiki/trunk/core/lib/Foswiki/Configure/ModalTemplates.pm line 28.  

-- GeorgeClark - 12 Aug 2014

Trying to save configuration at trunk.foswiki.org. After entering password to save:
Undefined subroutine &Foswiki::Configure::VerifyCfg::verify called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Configure/Checkers/ConfigureGUI/Modals/SaveChanges.pm line 145.

-- GeorgeClark - 13 Aug 2014

I just committed the bulk of the changes. Notes from IRC:
(17:42:58) CDot: ok, so, that's new configure. A short intro for the unwary; if you create a new checkout and point a webserver at it, it will just work
(17:43:18) CDot: you will be recommended to visit configure, however, as you will be running *as an admin*
(17:43:51) CDot: so, once you visit configure (and break it, probably) and save a configuration, you *must* be a Foswiki admin to be able to visit it again
(17:44:09) CDot: so it's *critical* that you configure authentication correctly *first time*
(17:44:32) CDot: there isn't a wizard to check the save allows logins yet. That will come.
(17:44:59) CDot: There are a number of wizards e.g. email config that are basically Timothe's work in the new framework
(17:45:11) CDot: they are untested, lumpy, and need work to clean them up
(17:45:34) CDot: the checkers are good, though. they all run, both from command-line and from web
(17:46:03) CDot: so editing an exiting config *should* be safe - though I would take a backup first, if I were you!
(17:46:43) CDot: the UI is pretty horrible looking - I have spent no time on it, and it should be easy for anyone with a bit of style to polish
(17:47:14) CDot: right now, I'm looking for people - especially you, gac410 - to start trying to use it
(17:47:39) CDot: oh, btw, you will need to install ConfigurePlugin and JsonRpcContrib for it to work
(17:49:00) CDot: BTW if you see any bugs, please dive in and try to fix them. I want to know where the architecture is unclear or obscure.

-- CrawfordCurrie - 28 Aug 2014

Crawford, I'm having some issues getting this working. Could you have forgotten to add some files? I get a lot of files missing when the plugin is pseudo-installed.
(01:17:20 PM) gac410: CDOt ... with a "bootstrap" config - no LS.C    I get 100's of 404's in logs.  Took minutes for the unformatted home page to display
(01:17:24 PM) gac410: GET /Main/NOT%20SET/System/DocumentGraphics/notify.png HTTP/1.1" 404 
(01:24:05 PM) ***gac410 is getting absolutely nowhere with configure
(01:24:29 PM) gac410: CDot ... did you forget to "git add" a bunch of files.    I get 20-30 files not found during pseudo-install
(01:24:40 PM) gac410: WARNING: Cannot find source file for /var/www/foswiki/distro/ConfigureContrib/lib/Foswiki/Configure/CGI.pm      That one stops configure dead.
trunk.foswiki.org also isn't working. It generates a js popup with just the word "Error" and then renders a mostly blank screen just with a small amount of explanation verbiage at the top.

-- GeorgeClark - 28 Aug 2014

Added an urgent one above. The Wizards don't seem to be able to change any settings that are hidden. So with your recent change, {EnableEmail} starts out as disabled, so the expert settings, including {Email}{MailMethod} is hidden. The Wizard runs, but MailMethod doesn't get changed. Everything works if I either enable email first, or remove the DISPLAYIF from the {Email}{MailMethod}.

And another. Array corruption Probably related to issues detecting element types.

It's slow going but I'm working on a change to the Save wizard to report all the actual changes to the saved config hash. This should make it easier to detect corruption and will become part of the change logger.

-- GeorgeClark - 31 Aug 2014

Strange about the hidden settings; hidden/DISPLAy_IF is purely a UI thing, and should have no effect on wizards :-/

There should be no changes to the saved config hash other than those passed in the parameter block the the json request. If there are others, then we need to consider how to detect them in the UI before the save is posted.

The principle I want to adopt with configure is that there is no way to save a config that has errors. An error detected in the UI should block the save (it doesn't at the moment for pragmatic reasons, but that's where it needs to head). So it's OK for the Save wizard to check and refuse to save if there are any errors, but not for it to save anyway and then report fails.

-- CrawfordCurrie - 31 Aug 2014

I just ran a firebug trace of running the mail wizard. The wizard showed:
Changed: {SMTP}{SENDERHOST} = cardinal.asdfasdf.com
Changed: {EnableEmail} = 1
Changed: {SMTP}{Username} = gac
Changed: {Email}{SSLVerifyServer} = 0
Changed: {Email}{MailMethod} = Net::SMTP (STARTTLS)
Changed: {SMTP}{MAILHOST} = lists.asdfasdf.org:587
Changed: {SMTP}{Password} = xxxxxxxx
Changed: {SMTP}{Debug} = 0

And the JSON response from the wizard included:

jsonrpc
   "2.0"
   
id
   "icw-EnableEmail-_24"
   
result
   Object { report=[2], changes={...}}
   
report
   [Object { level="notes", message="<div class='configureOk...</code></strong>.</div>"}, Object { level="changes", message="<div class='configureOk...{SMTP}{Debug} = 0</div>"}]
   
changes
   Object { {SMTP}{SENDERHOST}="cardinal.asdfasdf.com", {EnableEmail}=1, {SMTP}{Username}="gac", more...}
   
{SMTP}{SENDERHOST}
   "cardinal.asdfasdf.com"
   
{EnableEmail}
   1
   
{SMTP}{Username}
   "gac"
   
{Email}{SSLVerifyServer}
   "0"
   
{Email}{MailMethod}
   "Net::SMTP (STARTTLS)"
   
{SMTP}{MAILHOST}
   "lists.asdfasdf.org:587"
   
{SMTP}{Password}
   "xxxxxxxx"
   
{SMTP}{Debug}
   "0"

Following that response are 4 posts for changed fields: EnableEmail MAILHOST, Usename and password and the Save confirmation shows:
Save Changes to: {EnableEmail} {WebMasterEmail} {SMTP}{MAILHOST} {SMTP}{Username} {SMTP}{Password} 

So it appears to be something in the javascript that is not picking up the changes to the other panels. Senderhost, Mailmethod SSLVerifyServer and debug are all missing.

And the "diff" of the configuration hash for the save events: (I Currently just a print to STDERR, but I think this is valuable enough to check it in. It shows the growth in the regexes even though they were not posted during the save.)

=======  FROM BOOTSTRAP to First Configuration ===========
  {DataDir}: NOT SET => /var/www/foswiki/distro/core/data 
 {DefaultUrlHost}: NOT SET => http://foswiki.fenachrone.com 
 {LocalesDir}: NOT SET => /var/www/foswiki/distro/core/locale 
 {PubDir}: NOT SET => /var/www/foswiki/distro/core/pub 
 {PubUrlPath}: NOT SET => /bin/../pub 
 {SandboxWebName}: Sandbox => Litterbox 
 {ScriptDir}: NOT SET => /var/www/foswiki/distro/core/bin 
 {ScriptUrlPath}: NOT SET => /bin 
ADDED:: {ScriptUrlPaths}{view} value 
 {TemplateDir}: NOT SET => /var/www/foswiki/distro/core/templates 
 {ToolsDir}: NOT SET => /var/www/foswiki/distro/core/tools 
 {UsersWebName}: Main => Usersweb 
 {WorkingDir}: NOT SET => /var/www/foswiki/distro/core/working 

============== SAVE after running the Wizard ================

{Email}{ValidTLD}: qr/(?^:(?^i:AERO|ARPA|ASIA|BIZ|CAT|COM|COOP|EDU|GOV|INFO|INT|JOBS|MIL|MOBI|MUSEUM|NAME|NET|ORG|PRO|TEL|TRAVEL|XXX))/
 => qr/(?^:(?^:(?^i:AERO|ARPA|ASIA|BIZ|CAT|COM|COOP|EDU|GOV|INFO|INT|JOBS|MIL|MOBI|MUSEUM|NAME|NET|ORG|PRO|TEL|TRAVEL|XXX)))/
 
 {EnableEmail}: 0 => 1 

{LoginNameFilterIn}: qr/(?^:(?^:^[^\s\*?~^\$@%`"'&;|<>\x00-\x1f]+$))/
 => qr/(?^:(?^:(?^:^[^\s\*?~^\$@%`"'&;|<>\x00-\x1f]+$)))/
 
{NameFilter}: qr/(?^:(?^:[\s\*?~^\$@%`"'\x26;|\x3c>\[\]#\x00-\x1f]))/
 => qr/(?^:(?^:(?^:[\s\*?~^\$@%`"'\x26;|\x3c>\[\]#\x00-\x1f])))/
 
{RCS}{asciiFileSuffixes}: qr/(?^:(?^:\.(txt|html|xml|pl)$))/
 => qr/(?^:(?^:(?^:\.(txt|html|xml|pl)$)))/
 
 {SMTP}{MAILHOST}:  => lists.asdfasdf.org:587 
 {SMTP}{Password}:  => xxxxxxxx 
 {SMTP}{Username}:  => gac 

{Session}{AcceptUserPwParam}: qr/(?^:(?^:^view(auth)?$))/
 => qr/(?^:(?^:(?^:^view(auth)?$)))/
 
{UploadFilter}: qr/(?^:(?^:^(\.htaccess|.*\.(?i)(?:php[0-9s]?(\..*)?|[sp]htm[l]?(\..*)?|pl|py|cgi))$))/
 => qr/(?^:(?^:(?^:^(\.htaccess|.*\.(?i)(?:php[0-9s]?(\..*)?|[sp]htm[l]?(\..*)?|pl|py|cgi))$)))/
 
 {WebMasterEmail}:  => gac@asdfasfd.com 

-- GeorgeClark - 31 Aug 2014

As far as the "growing regexes" issues, this is a problem we had once before. I'm drawing a blank as to when and why and how it was fixed. Found it.
commit a0a84c6c6d7714e8b1ce4bcf4cec2ec31d7551b6
Author: GeorgeClark <GeorgeClark@0b4bb1d4-4e5a-0410-9cc4-b2b747904278>
Date:   Wed Nov 30 16:27:48 2011 +0000

    Item11287:  regex cleanup in configure for 5.14

    git-svn-id: http://svn.foswiki.org/trunk@13259 

-- GeorgeClark - 31 Aug 2014

I have a brute force fix. Scrubbing the regexes in Wizards/Save.pm Works, but maybe there is a more elegant way?

 sub _wordy_dump {
     my ( $hash, $keys ) = @_;
 
@@ -236,6 +310,8 @@ sub _wordy_dump {
         my $d = Data::Dumper->Dump( [$hash] );
         my $sk = _perlKeys($keys);
         $d =~ s/^\$VAR1/\$Foswiki::cfg$sk/;
+        while ( $d =~ s#qr/\(\?-xism:(.*)\)/;$#qr/$1/;# ) { }
+        while ( $d =~ s#qr/\(\?\^:(.*)\)/;$#qr/$1/;# ) { }
         push( @dump, $d );
     }
 

-- GeorgeClark - 31 Aug 2014

Checked in the regex cleanup, and the LSC compare routine. It found the hash corruption issue handily by showing the change in the config.

-- GeorgeClark - 31 Aug 2014

Well, after hitting my head for another hour, realized that the short url bootstrap can NEVER work as is. The bootstrap depends upon the view url. "mysite.com/Main/WebHome" to detect short URLs. Doh... bin/configure is the URL for configure and of course the bootstrap is not persistent.

So I'm wondering. If the code that wrote out the bootstrap warning during first visit to view, could generate a urlparam on the configure link to pass through what was detected for the view url.

BEGIN BLOCK for view url - Detects short url for view - Sets configure URL to ...configure?PATHSVIEW="view script"

new() - If bootstrapping - extract PATHSVIEW from the request and set the {ScriptUrlPaths}{view}

The code as checked in will set the {ScriptUrlPaths}{view} to be the same as the default hard-coded in Foswiki.spec. However if I comment out Foswiki.spec, it ends up NOT SET instead of set to the bootstrapped value. Configure shows the value as set by bootstrap, but it's not saved.

-- GeorgeClark - 02 Sep 2014

Fixed: Another major bootstrap / configuration issue. In a "script-suffix" environment, we need to auto-configure the script-suffix. But worse, configure doesn't seem to apply the script suffix to the jsonrpc calls. So even though I have code setting ScriptSuffix during bootstrap, it doesn't make it into the jsonrpc call.

It's hardcoded in configure.js. var json_rpc_url = "jsonrpc", and I don't see any code to append the suffix.

-- GeorgeClark - 02 Sep 2014

I've completely refactored the bootstrap code, moving it from the Foswiki.pm BEGIN block into Foswiki::Configure::Load::bootstrapConfig. It also eliminates the NOT_SET notation, with the assumption that bootstrap will set all of the mandatory settings.

TestBootstrapPlugin has also been refactored to call the bootstrap routine, so that it remains in sync with core.

All of this is in a new test branch https://github.com/foswiki/distro/tree/refactorBootstrap and https://github.com/foswiki/TestBootstrapPlugin/tree/refactorBootstrap

-- GeorgeClark - 11 Sep 2014

Merged the refactorBootstrap branches into master.

-- GeorgeClark - 13 Sep 2014

Crawford, I've reverted your removal of Configure::Util and the ConfigureTests. They are extensively used by the Extension installer, both CLI and web. I've renamed ConfigureTests to ExtensionInstallerTests. And I'll remove anything from Util that you've moved to File::Util. If you want, I can look at renaming Util to Package::Util since that's where it's used mosly.

I've got to figure out what you did to the output from Dependency report. It was used in both CLI and Web extension installation, and that's broken at least in the unit tests.

-- GeorgeClark - 15 Sep 2014

Loading checkers was a big source of jsonrpc slowness. Identified with NYT::Profile. Here is how I ran the profile:

time perl -wT -d:NYTProf -I ../lib ./jsonrpc -POSTDATA '{"jsonrpc":"2.0","method":"check_current_value","params":{"keys":["Extensions"]},"id":"iCheck--Extensions_6"}' configure

-- GeorgeClark - 23 Sep 2014

FredTarbox added to the list.

-- GeorgeClark - 14 Oct 2014

Crawford, ... More info referencing our irc discussion about ref checking.

I applied the below patch, and then did some testing with the ConfigurePlugin PERL type. The log clearly shows that Load::expandingValue is as you suspected returning the string HASH{...} instead of the Hash reference. Something's rotten here but I'm not sure what. Here is the trace from the below patch.

Checkers that access the ::cfg hash directly work fine (ie: my $val = $Foswiki::cfg{AccessibleCFG} ) However checkers that use my $value = $this->getCfgUndefOk(); (ex: PERL.pm) don't.

PERL Checker entered
BEFORE: $Foswiki::cfg{Plugins}{ConfigurePlugin}{Test}{PERL}
expandValue called with $Foswiki::cfg{Plugins}{ConfigurePlugin}{Test}{PERL} 
_expandValue called with $Foswiki::cfg{Plugins}{ConfigurePlugin}{Test}{PERL} 
AFTER: $VAR1 = \'HASH(0x9772738)';

Clearly this isn't correct, but I don't know how to fix it.

--- a/core/lib/Foswiki/Configure/Checker.pm
+++ b/core/lib/Foswiki/Configure/Checker.pm
@@ -192,8 +192,11 @@ sub getCfg {
     my ( $this, $name ) = @_;
     $name ||= $this->{item}->{keys};
 
+    require Data::Dumper;
     my $item = '$Foswiki::cfg' . $name;
+    print STDERR "BEFORE: $item\n";
     Foswiki::Configure::Load::expandValue($item);
+    print STDERR "AFTER: " . Data::Dumper::Dumper( \$item ); 
     return $item;
 }
 
@@ -213,8 +216,11 @@ sub getCfgUndefOk {
     my ( $this, $name, $undef ) = @_;
     $name ||= $this->{item}->{keys};
 
+    require Data::Dumper;
     my $item = '$Foswiki::cfg' . $name;
+    print STDERR "BEFORE: $item\n";
     Foswiki::Configure::Load::expandValue( $item, defined $undef ? $undef : 1 );
+    print STDERR "AFTER: " . Data::Dumper::Dumper( \$item ); 
     return $item;
 }
 
diff --git a/core/lib/Foswiki/Configure/Checkers/PERL.pm b/core/lib/Foswiki/Configure/Checkers/PERL.pm
index 53f9ae9..8ef9549 100644
--- a/core/lib/Foswiki/Configure/Checkers/PERL.pm
+++ b/core/lib/Foswiki/Configure/Checkers/PERL.pm
@@ -12,6 +12,7 @@ sub check_current_value {
 
     #$this->showExpandedValue($reporter);
 
+    print STDERR "PERL Checker entered\n";
     my $value = $this->getCfgUndefOk();
 
     return if ( defined $value );
diff --git a/core/lib/Foswiki/Configure/Load.pm b/core/lib/Foswiki/Configure/Load.pm
index cafd85f..8856df4 100644
--- a/core/lib/Foswiki/Configure/Load.pm
+++ b/core/lib/Foswiki/Configure/Load.pm
@@ -211,6 +211,7 @@ $mode - How to handle undefined values:
 
 sub expandValue {
     my $undef;
+    print STDERR "expandValue called with $_[0] \n";
     _expandValue( $_[0], ( $_[1] || 0 ), $undef );
 
     $_[0] = undef if ($undef);
@@ -220,6 +221,7 @@ sub expandValue {
 # $_[1] - $mode
 # $_[2] - $undef (return)
 sub _expandValue {
+    print STDERR "_expandValue called with $_[0] \n";
     if ( ref( $_[0] ) eq 'HASH' ) {
         expandValue( $_, $_[1] ) foreach ( values %{ $_[0] } );
     }
~

-- GeorgeClark - 15 Oct 2014

 

PLEASE KEEP THIS LIST OF OPEN ITEMS AT THE END OF THE TOPIC

  • DONE Not logging use of configure
  • DONE Directory / file permission not displayed as octal: {Store}{dirPermission} and {Store}{filePermission}
  • DONE "Show Expert" toggle doesn't stick, but gui does. On visiting a new tab, have to click Show Expert twice (turn off / on) to get the settings to display.
  • DONE {ScriptUrlPaths}{view} is undefined error when the field is empty.
  • DONE Need obfuscation of password fields in config, and Mail wizard debug log.
  • DONE Logging and Statistics > Logging > {Log}{Action} has 1 errors reports that it isn't a hashref, but it is. Couldn't figure it out.
    • the checker (which came from old configure) was actually wrong. Something in old configure was masking it.
  • DONE More obvious indication of "unsaved changes" would be helpful
  • DONE Need some way to see all errors without visiting each tab one at a time.
  • DONE After first save, subsequent saves cause Error reading existing LocalSite.cfg: Can't call method "CHANGED" on an undefined value at /var/www/foswiki/distro/core/lib/Foswiki/Configure/Wizards/Save.pm line 80.
  • DONE Email wizard still needs work ... MailProgram config incomplete. Needs more testing
  • DONE Regular expressions are having some expansion issues: UploadFilter for ex is showing as: qr/(?^:(?^:(?^:(?^:(?^:(?^:(?^:(?^:(?^:(?^:(?^:(?^:^(\.htaccess|.*\.(?i)(?:php[0-9s]?(\..*)?|[sp]htm[l]?(\..*)?|pl|py|cgi))$))))))))))))/;
  • DONE Wizards make changes to hidden settings, but they don't get submitted with save. Only visible elements appear to be submitted.
  • DONE Array type configuration elements. Minor change to Store > DataForm settings > {FormTypes} when configuration was saved, became '['. Unable to revert. (Explained below. The new configure dumps every entry in the hash as a separate config entry.)
  • DONE Language selectors should be BOOLEAN checkboxes. They are rendered as text fields, and are deleted from the config if the languages tab is just visited.
  • DONE "Save button" changes counter doesn't set to 0 after save.
  • DONE "Booleans reported as "0" but set as enabled after wizard.
  • DONE Installing and enabling a new plugin failed to auto-set the {Module} entry.
    • Fixed: The Save wizard needed to find and preserve additions made to the configuration hash. Added code to Pluggables::PLUGINS to save the list of keys added to the configuration so that Wizards::Save could recover them.
  • DONE When an extension is installed after the bootstrap is finished, the default settings from Config.spec are never added. First visit to the Extension settings panel populates the spec using the getspec function, but never calls getcfg to actually load the values if they are not already in the configuration. Not sure if this should be done there, or should be done by the extension installer during installation.
  • DONE Fixed: It's impossible to edit the AdminGroup when running in bootstrap mode. Edit redirects to login. So no way for a new site to authorize an admin user before saving the first configuration.
    • Changed from testing isBOOTSTRAPPING in the isAdmin routine, to forcing the session defaulUser to admin when the foswiki object is first created. This feels safer than checking isBOOTSTRAP later in execution.
  • DONE Unable to save fields of type PERL. Reported by FredTarbox trying to install LdapContrib. But same issue happens if you attempt to change the {AccessibleCFG} array. It ends up saved as $Foswiki::cfg{AccessibleCFG} = '['; and can't be reverted.
    • distro:13841b7287ca is a partial fix. It fixes the corrupted Ldap and AccessibleCFG PERL type fields, but a simple quoted string as type PERL gets corrupted. The quotes are removed after first save, and ends up as a bare word.
    • When PERL type is entered as '\'PERL\';' then an extra ; on an empty line appears to be saved into LSC.
  • DUPLICATE The "Changes" report comparison doesn't work when the field type changes during the save. For ex, on PERL types, the input from Json Set is a quoted string, but the config contents is a HASH or an ARRAY. It ends up converted to a matching type during save.
  • When installing an extension the Tab for the extension settings is not created, so there is no way to edit settings immediately after installation and before saving the configuration. But the Extension Installer wizard enables the plugin, so the save to enable (and create the settings tab) could render the system unusable if "undefined" settings results in a failure. Config.spec defaults are automatically added when the extension is installed. The recommended save which enables the extension also saves those defaults. I can't defend against those defaults being SNAFU, however.
  • DONE AuthScripts hidden if something other than TemplateLogin set, but checker flags error for AuthScripts if something other than TemplateLogin set. Detailed errors and warnings should probably be visible at all times. - This is a problem with the checker, and not with configure per se. Item13064 opened
  • DONE Save is not logged. Need what changed, by who, and why. - This can be done later. Item13063 opened.
  • DONE jsonrpc doesn't log events (Should we move event logging up a layer into the engine?) - This can be done later. Item13065 opened

PLEASE MOVE FIXED ISSUES UP ABOVE OPEN PROBLEMS so we have all the problems at the end of the list

Long overdue, but I'm closing this as "waiting for release". Any new (or outstanding) problems need their own task.

-- CrawfordCurrie - 30 Oct 2014

ItemTemplate edit

Summary Move Configure into a Contrib
ReportedBy CrawfordCurrie
Codebase trunk
SVN Range
AppliesTo Engine
Component
Priority Urgent
CurrentState Closed
WaitingFor
Checkins distro:e5ffcbc53da3 Rev 17843 not found distro:98d0ecd64ebb distro:a4685b520fa7 distro:caadc26a5e80 distro:80d94309cbc9 distro:3a0bf348b065 distro:bf407697f686 distro:2fb4a78a54a6 LogDispatchContrib:db6526a425fa distro:0bfd35799414 distro:18e67e306280 distro:9bb93864ecb6 distro:aaf37bd7574a distro:3e5693aac2d1 distro:661a7f535540 distro:8d503d157f08 distro:64d0d7f43057 distro:a2523520dae8 distro:c3cd3374f520 distro:9829d74284d4 distro:48680a89423f distro:387c0588568f distro:1100e27aec91 distro:4d68f278ed7f distro:d119d8ce2a3f distro:12ffa0e3d7b5 distro:f6192c69d2f6 distro:74597af447ca distro:d6d633908dd8 distro:d66593a7a4c1 distro:13841b7287ca distro:b3932c80528d distro:a5ddb65a9ac4 distro:e80d705e240a distro:26e192721620 distro:d48efb56af2b distro:3764a9dd7433 distro:ec7eec55f7c5 distro:652aec227f63 distro:2da012d2cf3b distro:198dc3d38359 distro:825c26c87fb3 distro:7e3eaf253169 distro:0505af8efb1e JsonRpcContrib:6c34e58bb933 JsonRpcContrib:c2516fdd860d
TargetRelease major
ReleasedIn 2.0.0
CheckinsOnBranches master trunk
trunkCheckins distro:e5ffcbc53da3 Rev 17843 not found distro:98d0ecd64ebb distro:a4685b520fa7 distro:caadc26a5e80 distro:80d94309cbc9 distro:3a0bf348b065 distro:bf407697f686 distro:2fb4a78a54a6 LogDispatchContrib:db6526a425fa distro:0bfd35799414 distro:18e67e306280 distro:9bb93864ecb6
masterCheckins distro:7c920d8b1a5b distro:ef9a42cc01e3 JsonRpcContrib:be29b3213198 JsonRpcContrib:add0400c24b5 JsonRpcContrib:9a0e2133f88f distro:4fd70c6350c7 distro:fe0ffeb1491e distro:81475cef48d7 distro:2d50dfca7f16 distro:4265fd5569dc distro:f0f419e8bee3 distro:70cbafde6bd0 distro:8a30f7e13dd7 distro:829f732faa98 distro:fc572d285f1e distro:2595a905d707 distro:993a2abe16b9 distro:362331af047e distro:6d55bef4e9bc distro:fb988bdeb921 distro:1951c598c0e7 distro:393315c7a1c2 distro:c7ff26818709 distro:35af3f4d28a7 distro:fd4154539926 distro:14ba355a2d3e distro:e8efc7a4753a distro:f583815d6fe4 distro:498d88664eb0 distro:71307b995aa5 distro:22805f254716 distro:d0aec8e355d3 distro:18529c971f91 distro:5626f5b17f15 distro:712d00efecc7 distro:2b7b8814e509 distro:75298b4bad00 distro:6904c1518451 distro:fac88d82274c ConfigurePlugin:f943991f3787 ConfigurePlugin:1a0af8466f51 ConfigurePlugin:731ac3db3aff ConfigurePlugin:eac565477c55 distro:f20032729dd9 ConfigurePlugin:2db8b3a0c88e distro:6afdb9a189fd ConfigurePlugin:e4b2f60ae3d4 distro:6ace65014fb2 ConfigurePlugin:ef5209d18c3a distro:3333194655b9 ConfigurePlugin:23d8a5c0dbd0 distro:83999fc0c85a distro:9ea45c67e89e distro:9bd301ffcb01 ConfigurePlugin:8fb0afb277ef distro:d7c52187cc5d distro:42a708eb5455 distro:6de48964fcad distro:793f8faf1a7d distro:c85ac01651be distro:cacfed9eed29 ConfigurePlugin:8e62225f1f2c distro:2c88047c9592 distro:50de5bdb7e7e distro:d84dff2dd3e3 ConfigurePlugin:0cffb2dc4845 ConfigurePlugin:4bfb9ea99a67 distro:d17a67dbb3f3 distro:71e9da562be8 distro:7950cc0d993f distro:401f6dd9ddb2 distro:5ede7e1a2548 distro:51e78f7d373c ConfigurePlugin:ee0ff41a49d3 distro:32a1d6dd1b0d distro:5c2781e9f001 distro:22d051f9f52f ConfigurePlugin:dd928ea60f08 ConfigurePlugin:b49999343a5b ConfigurePlugin:a41af4d737d2 ConfigurePlugin:3352732277bd ConfigurePlugin:1ed6b876cb30 distro:24b6011bc044 distro:e25ec0d00fa4 distro:c7faa04d34a9 ConfigurePlugin:05b0759ebb2b ConfigurePlugin:b65a89dbe4a8 distro:cc42a898e279 distro:aa4df4cbd79a ConfigurePlugin:aba19910722b ConfigurePlugin:111a4f162ff5 distro:ece272152579 distro:0079f6e0d5eb distro:1585193ed18e ConfigurePlugin:5eaad0a75778 distro:bd55e143b5f7 distro:700e532b00da ConfigurePlugin:2a8fd172a8a2 distro:ef2db125b51b ConfigurePlugin:a2f554842045 ConfigurePlugin:050d73ce99d6 distro:cdbfbadf69bc distro:567560b7f848 distro:6cc4bb6250b1 ConfigurePlugin:910366e2d7f0 distro:fda7b8e1cb1d ConfigurePlugin:e42573db6640 distro:a68c7cdd2461 ConfigurePlugin:d14048d4a7a7 distro:56b48a305e01 distro:0a63701aefc1 distro:d50c7e1131d9 distro:e704458c90d2 ConfigurePlugin:160636ec5ebe distro:2e77b83688c5 distro:7798c611c289 distro:3d009b4dc5df distro:1edb29f0b09b distro:336f97927b27 distro:6f2d2b09c985 distro:d63011d63a9a distro:c5c063f0803b distro:c1a9d1a27583 ConfigurePlugin:18790e1bc030 distro:8deecf5f88f8 distro:fc44d647c090 distro:0e2a8ec38736 distro:3630503d610a distro:36b8ec97c1f0 distro:670ac4e42f1a distro:219582218541 distro:3f2295d306ae distro:9e83462a6c7c distro:e6115f0f8931 distro:aa5c17d0033a distro:fcee60439cdd distro:bd9026ffa7ee distro:dc917ee3a71d distro:530ecffd28f9 distro:28c769ce4018 distro:29904de380f9 distro:20b615d71fc4 distro:86a93399386f distro:3d1ef4374a1f distro:5c7bb86e5465 distro:097bd7852c20 distro:2ded95d6e91c distro:041e29bb1559 distro:5afa316df5eb distro:1a18dafacbc3 distro:92abf3aecfca distro:577e0e96b33a distro:1966482de81a distro:18518b41fcee distro:222b8e1a9e69 distro:2c6e865b0956 distro:5980290f2f2f distro:638609782fa8 distro:c3ba0f072aed distro:6149e44d7f62 distro:0d1b8f7eb8b3 distro:baeeb3ac387a distro:09d76219e471 distro:1eecf96ae6c3 distro:a07393b6daf0 distro:dffc76604d07 distro:544f3e2efe68 distro:17d1292ef67f distro:8b46a3b1bff9 distro:21869ac8ae6d distro:0ace70b503b2 distro:b7e84f93a56b distro:c73dfa4e409c distro:afddc8182660 distro:256fa4effa70 distro:dbf3a42e9270 distro:309f9a3dbb73 distro:73213b7188f1 distro:370a685bbc50 distro:a6a3569e9cb6 distro:87cc84982a7b distro:4c9b47c0f779 distro:ad20c6873905 distro:7e8f06382dd7 distro:386a65adbce3 distro:dd0b86dc03f3 distro:77fe999116ea distro:cc84807bddb4 distro:9c4b8a4f737b distro:b2b393d130b4 distro:bebdc1b26983 distro:fae4d7244efc distro:7fa459921e03 distro:8375be2db459 distro:dc1865fec91e distro:e52951e53e63 distro:3db941b09009 distro:7a41df00e6e7 distro:975f24eb099c distro:2b3e0405683c distro:553f2146ea80 distro:afda3a91c00b distro:af619060e73b distro:554b138bad65 distro:67764c38d6c3 distro:f29fc6fd2e68 distro:f6eb44ecbe47 distro:7e3370aeb414 distro:aaf37bd7574a distro:3e5693aac2d1 distro:661a7f535540 distro:8d503d157f08 distro:64d0d7f43057 distro:a2523520dae8 distro:c3cd3374f520 distro:9829d74284d4 distro:48680a89423f distro:387c0588568f distro:1100e27aec91 distro:4d68f278ed7f distro:d119d8ce2a3f distro:12ffa0e3d7b5 distro:f6192c69d2f6 distro:74597af447ca distro:d6d633908dd8 distro:d66593a7a4c1 distro:13841b7287ca distro:b3932c80528d distro:a5ddb65a9ac4 distro:e80d705e240a distro:26e192721620 distro:d48efb56af2b distro:3764a9dd7433 distro:ec7eec55f7c5 distro:652aec227f63 distro:2da012d2cf3b distro:198dc3d38359 distro:825c26c87fb3 distro:7e3eaf253169 distro:0505af8efb1e JsonRpcContrib:6c34e58bb933 JsonRpcContrib:c2516fdd860d
ItemBranchCheckins
Release01x01Checkins
Topic revision: r251 - 05 Jul 2015, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy