Feature Proposal: Move foswiki core lib/CPAN into its own CpanContrib

Motivation

first up, I don't want it, I don't need it, and i always worry about what is and isn't up to date.

Given that the released default distro is made up of lots of contribs, I'd like to do the same for this.

Description and Documentation

Make a CpanContrib, make a build.pl script that gets / updates the CPAN libs we want to ship in foswiki, and then include it in the foswiki core MANIFEST.

the net result is no apparent change for users of the zips, tgz, or normal pseudo-install.pl use, while minimal installers can thus get much closer to minimal.

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: SvenDowideit - 06 Oct 2012

Discussion

Good idea. Also merge in the DateTimeContrib that ArthurClemens created. No sense having multiple CPAN library contribs.

-- GeorgeClark - 09 Oct 2012

And we should review and add any other CPAN dependencies that are not shipped with our minimum perl - 5.8.8, and add them to the CPAN contrib. I don't see how it makes sense to ship only a subset of CPAN that leaves a default install not fully functional.

Rather than shipping the compiled packages, which will cause issues with XS modules, could we work out something to ship the distribution files and a build process:

• Ship foswiki/lib/CPAN/sources containing the dependency tarballs
• Include an operational cpanm for simple installation
• Run cpanm to install each package using:
  • cpanm -L lib/CPAN/lib lib/CPAN/sources/HTML-Parser-3.69.tar.gz

If a network connection is available, we could offer to download the latest versions.

The tools/dependencies_installer.pl tries to do this dynamically, and also tries to use the OS package manager or cpanplus if it's unavailable. I think we'd be better off to just use cpanminus or cpanplus, and use this for non-package-manager based installations. The package manager really ought to be resolving dependencies without need for help from dependencies_installer.

-- GeorgeClark - 23 Oct 2012

Florian's unit tests issue shows that this really needs to be dealt with. Conflicting versions in Foswiki/lib/CPAN/lib and System libraries of Class::Load and dependencies results in failure due to missing XS modules.

Possible solutions:

1. Ship pre-installed CPAN deps by default. (current solution)
2. Change default lib order so local Foswiki/lib/CPAN/libis first.
   • Makes it difficult to install newer libraries locally
3. Ship pre-installed CPAN deps in a separate (optional) contrib.
4. Ship a deplist and tool to build missing deps with cpanm in a private lib.
   • Do we need to ship the dist. files to accommodate sites installing without an Internet connection
5. Ship a dependency installer like tools.dependency_installer.pl it tries to install using multiple tools - .deb, cpanplus (deprecated), rpm ...
6. Simplify / remove CPAN deps so that core doesn't need them.
   • Class::Load required by LogDispatchContrib and Log::Dispatch. Remove as a core extension?
   • Class::Load added to Foswiki.pm as part of DBIStoreContrib:0d42f0bdd3a6 distro:35fc44a8a483 MongoDBPlugin:93c3178c140e - rewrite to eliminate Class::Load?
   • Examine other dependencies to determine if we really need to ship them.

• ... What else?

-- GeorgeClark - 19 Apr 2013

The rationale for shipping CPAN libs was "to avoid DLL hell" (not my words). If we don't ship CPAN libs, then we are throwing users to the lions - perhaps not a bad thing, but it will reflect badly on us even though it's really CPAN's fault.

• I am against building tools to resolve CPAN dependencies.
• I am in favour of simplifying things so that barely-required CPAN modules are not-required.
• I am in favour of not shipping any CPAN libs, but instead check, and report, if the libs found are inadequate.
• The doc on where it looks for libs really needs to be beefed up.

-- CrawfordCurrie - 19 Apr 2013

+1 for not shipping any CPAN libs and instead informing the admin what's missing or too old. And it's only fair to provide basic pointers how to get those modules from one's distro or directly from CPAN, but no custom CPAN installer or contrib - IMHO plain 'cpan' works well enough and is included with Perl.

And yes, the library path thing is too complicated, and I fail to see the benefit of all the different variants (append, prepend, defined but empty...). I'd say if someone installs something locally, that's because they want that to be used in preference to what's in the system path. So all that needs to be done is prepend Foswiki/lib and $PERL5LIB to @INC, and since $PERL5LIB might not be set in the web server environment, one might want to set it in LocalLib.cfg... or perhaps not, and it has to be set in the (web server or cli) environment, thus removing the need for LocalLib.cfg altogether.

-- FlorianSchlichting - 20 Apr 2013

The library path defaults to "Append". AFAIK the reason is because the CPAN modules we ship are intended as a "Last Resort" so that Foswiki will work "out-of-the-box" with the system perl. If a user installs newer versions using CPAN or system package managers, we want to use the system version and allow ours to be ignored. This seems to be good 99% of the time.

The "prepend" is used for when the system supplied version is too old, and cannot be easily updated. (some hosting sites, IT groups, etc.). I don't recall needing to tell users to do this. I guess it's pretty rare.

Developers should occasionally test with Prepend - so that bugs due to our bundled CPAN can be detected. The unit tests might do that automatically - I'm not sure.

The defined but empty and other variants ... I have no idea.

Regarding removing LocalLib.cfg, it's mostly never needed anyway. It's very rare that I've ever run into needing to set the lib. Foswiki generally figures it out. The only time I've ever touched LocalLib.cfg was to enable ASSERTs, and to change to prepend to be sure I was testing with the code we ship.

As far as not shipping any CPAN modules, CPAN can be very painful. The only way I've found that is reliable is CPAN:App::cpanminus. The new dependency on "version" has caused some issues. Class::Load is looking to be a disaster I'll bet. You cannot get it on Debian unless you pull it from Sid.

-- GeorgeClark - 20 Apr 2013

Ditto with respect to only using LocalLib.cfg for developer things, like ASSERT. I would be surprised if the nightly tests look at the lib setup at all - I don't recall ever having seen anything like that.

-- CrawfordCurrie - 21 Apr 2013

I've been experimenting a bit with the cpanm tool, and found some suggestions through various google searches. Here is a revised proposal to address the following considerations:

Revised Proposal

• For sites that have "disconnected servers"
  • Ship a mirror directory with the distribution files for all Foswiki dependencies
  • Populate foswiki/lib/CPAN/lib with =cpanm --mirror <mirror-directory> -L foswiki/lib/CPAN/lib
• For hosted sites that can't locally build perl modules.
  • Ship a populated lib/CPAN/lib directory that can be moved into the distribution
• Dealing with modules that use XS and PurePerl implementation.
  • Build any local modules with --pureperl flag

Foswiki changes:

• Ship with an empty lib/CPAN/lib directory
• Change the default @INC so lib/CPAN/lib is at the head of the list, overriding any local modules
• Either include this contrib "on the side" as part of our normal tarball / zipfile, or as a completely separate download

Implementation

A new installation would have 4 options, in order of preference:

• Use system provided packages. They don't need this extension, so just remove it if it we include it in the tarball.
• OR Use cpanm to build the dependencies. cpanm will fetch the latest versions from CPAN.
• OR Use cpanm with --mirrorto build from the mirror we ship with the contrib
  • Either of these options get the XS optimized versions when available
• OR Copy the lib/CPAN/lib directory from the contrib into foswiki.
  • Only pureperl is available, but no building is necessary

Proposed contrib

• CpanContrib
  • cpanmirror: Directory containing mirrored distribution files
  • /lib/CPAN/lib: Directory with pre-build Pure Perl modules
  • tools/cpanm: Installation of cpanminus
  • shell script / bat file to execute each strategy

There is a bit of a chicken & egg problem in that foswiki configure requires some of the dependencies. I think this should be a separate part of the distribution, either a separate directory, or a separate download. The user would expand the foswiki-1.2.0.tar.gz file, and then follow instructions in the install guide to install the dependencies.

Other notes:

Populating the mirror

cpanm -L /dev/null --save-dists cpanmirror --scandeps Log::Log4perl::DateFormat This command will find all the dependencies of the named package and download all of the distribution files into the cpanmirror directory.

Building a self contained library with pure perl versions

cpanm -L lib/CPAN/lib --pureperl --mirror ~/cpanmirror Class::Load This builds Class::Load and all dependencies as a self-contained lib/CPAN/lib directory.

Building only modules not locally up-to-date

cpanm -l lib/CPAN/lib --no-man-pages --mirror ~/cpanmirror Class::Load This builds Class::Load and missing dependencies into lib/CPAN/lib directory.

Noticed two separate CPAN contribs out in svn. CpanInstallerContrib and CpanContrib. Neither of which are released.

-- GeorgeClark - 13 Mar 2014

Sounds good. Only problem I see is changing the order in @INC pushing lib/CPAN/lib to the top. For now, if you'd like to have your own CPAN libs being preferred over any system-installed ones you'd put them into lib directly.

-- MichaelDaum - 14 Mar 2014

• Updated topic title to CpanContrib and cleared the namespace by moving the work of WillNorris to scratch.
• Moved CPAN deps into CpanContrib using Item12475.

-- OliverKrueger - 14 Mar 2014

The @INC issue is what we discovered with Class::Load. I know it's dropped, but it exposed an issue. It had complex dependencies including some XS code. The local site provided out-of-date deps, and us including a subset of them only made it worse, since we couldn't override without modifying the complicated LocalLib.cfg

I think that if a site chooses to use the CPAN libs we provide, the ones the user builds into foswiki/lib/CPAN/lib need to be ahead of the System perl libs. The old way, we were last as a fallback. New way, we are first, but (nearly) empty and we will override out-of-date local packages if the admin chooses to activate them. Good example beyond the XS packages is version, which ships with some perl but too old for our use.

-- GeorgeClark - 14 Mar 2014

IRC discussion, MichaelDaum comments that we need both a last-resort and an override. We have that today by putting the CPAN package directly into lib. for ex. foswiki/lib/CGI/Session would override the system package, but putting it in foswiki/lib/CPAN/lib/CGI/... is a last resort.

This is a good call-out as it eliminates the need to edit LocalLib.cfg to override the system libs. We run without that file 99% of the time. Looking at what cpanm does building a "self contained" library, it creates lib/perl5/..., so it's worth testing if we could we use an optional foswiki/lib/perl5 structure as the override libs, and a default but nearly empty foswiki/lib/CPAN/lib as the "last resort. This needs to be tested.

The hosted user who cannot build dependencies and is unable to get the hosting site to install dependencies would move the CpanContrib/lib/perl5 directory to foswiki/lib/perl5 and hopefully be done. And a user who needs to override a single package could run cpanm -L foswiki One possible issue, cpanm appears to populate bin as well, creating two files: config_data and package-stash-conflicts. Not sure what to do with them, but I doubt we want them executable from the browser.

Ug... I can't find any way of blocking the update of bin. Based on this, I think it's best to recommend any use of cpanm be run into a temp location and then copy the directory into foswiki

-- GeorgeClark - 14 Mar 2014

As of 3 June 2014. Here are the install experiences on a new perlbrew 5.20.0 installation:

• Error: Missing Dependency: HTML::TreeBuilder.
• Net::SMTP and IO::Socket::SSL are required to autoconfigure mail

The email auto-configure requires SSL, especially if it's going to work on Google's gmail.

• IO::Socket::SSL requires Net::SSLeay, but that requires OpenSSL and is unavailable in a pure perl implementation.
• Add Net::SSLeay as an optional dependency.

Other missing dependencies

• HTML::Entities (WysiwygPlugin)
• HTML::Parser (WysiwygPlugin)
• File::Copy::Recursive (PlainFileStoreContrib)
• JSON: (EditRowPlugin, UpdatesPlugin, SubscribePlugin)
• Time::ParseDate (HomePagePlugin, TWikiCompatibilityPlugin, TopicUserMappingContrib)
• URI (MailerContrib)
• URI::Escape (MailerContrib)

And Foswiki won't run with Asserts enabled.

CGI will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at /var/www/foswiki/trunk/core/lib/Foswiki.pm, line 56.

And configure is very noisy, logging the following repeatedly:

CGI::Carp will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at (eval 1), line 1.
CGI will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at (eval 8), line 1.
CGI::Util will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at /home/gac/perl5/perlbrew/perls/perl-5.20.0/lib/5.20.0/CGI.pm, line 29.
CGI::Cookie will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at (eval 46), line 2.

These can probably be avoided if our CpanContrib ships all these, and puts them at the front of the @INC lib path.

-- GeorgeClark - 03 Jun 2014

HTML::Parser doesn't seem to want to build in a pureperl version. It needs a compiler so we can't distribute it.

-- GeorgeClark - 04 Jun 2014

With the latest version of Configure - it's now just another UI method. The rationale to continue to ship CGI::Session doesn't seem to make sense. We should just eliminate the rest of lib/CPAN/lib. I've reopened Tasks.Item12475

-- GeorgeClark - 06 Nov 2014

Commenting is disabled while not logged in