Feature Proposal: It would be cool to ease Perl module installation or at least to simplify figuring out what is missing

Motivation

A first time installer may for each extension carefully check related dependencies of Perl modules. But if one already knows what to install and quickly does it for quite some plugins we are loosing track of what is in the dependencies or have to re-visit all the pages and then check the Perl installation manually if all modules are available.

Description and Documentation

Configure already provides an overview of required Perl modules but AFAIK this is only for the well maintained parts (default installation and extensions). Also from there installation cannot be done automatically (at least currently no button to say please download from CPAN and install for me).

I wrote a script which could be placed in the tools directory and tries to:

• identify Foswiki related Perl modules and lists them
• offers to install anything missing via CPANPLUS

The script is already available and seems to do its job for me. So I am here merely asking if it may be added to the tools directory, so others could use it too.

There are probably good arguments to better do parts of this functionality via configure but that's at least currently something I cannot commit myself to. This script therefore may also be considered as a prototype of features potentially a nice to have in configure.

Examples

• foswiki_perl_modules.log: Example output when running the script

Impact

WhatDoesItAffect:

Providing the script officially in the tools directory.

edit

Implementation

-- Contributors: IngoKappler - 17 Nov 2010

Code is already available and could be put into trunk quickly.

Discussion

Great proposal.

Interesting to look more carefully at.

And what I say now is not in any way to be taken as a negative input.

When you run with a certain Linux distribution many of the CPAN libraries are available was packages. As debs for Debian/Ubuntu family, or RPMs for the Redhat/Mandriva/Suse family. And when you install your packages you automatically have your system in an automatic security update cycle. And you also have some automation around making sure that the versions are compatible with other packages.

If you install CPAN libraries via CPAN you have to keep yourself up to date manually.

So many of us try to install CPAN libraries using standard packages for our distribution as the first choice. And when it is not available, we go for CPAN.

Whatever we introduce has to take this into consideration. At least for the major distrinbutions.

I have installed 95% of my CPAN libraries via RPM repositities and not via CPAN.

-- KennethLavrsen - 11 Nov 2010

Hi Kenneth,

I fully agree and handled it on my production system in he same way. But sometimes I found it pretty tiring to figure out what distribution package actually contained the module. It is much easier and quicker to find it via CPAN but has the downside of being solely responsible.

CPAN so far was just easier to implement and I like to keep it as an option. But I've been wondering if there could be a way to find out automatically e.g. what Debian package contains a certain module by maybe using the definitions Sven or the Debian maintainers of the Foswiki package define in their text files. I didn't have a chance to dive into that yet, so guiding feedback is welcome here. Maybe it can be parsed out from these files...

-- IngoKappler - 12 Nov 2010

I am currently working on handling Debian packages as well. I am not sure I will remain the right one to handle e.g. RPM too.

-- IngoKappler - 02 Dec 2010

I'm all in favor of having a script to check the dependencies, and maybe making this script RPM / apt-get / ... aware. RPM should be easy, as it provides perl(My::Module), apt-get might be a bit more cumbersome, but it should be doable, worst case, with a hashing table.

I'm, on the other side, totally against putting any of this into configure. We (Ok, I did it) removed this feature from configure because we agreed that the webserver user shouldn't be playing around with CPAN modules, and on many cases this won't work. So, configure could say something like:

• You are missing these perl modules: Algorithm::Diff, Encode, LWP. Please logon to the machine with a user allowed to sudo, or root, and run: $Foswiki::cfg{ToolsDir}/check_perl_dependencies.pl -install.
• This will download the missing dependencies from CPAN, and install them (using sudo if possible).

Last point, there are many CPAN-like thingies: CPAN, CPANPLUS, CPANMINUS, each having its advantages and drawbacks. Not sure it's worth making the code CPAN-backend-pluggable, but it could be.

Or better, maybe we could integrate CPAN:App::cpanminus to our release, or we could even use cpanminus to install itself, as explained in the documentation:

curl -L http://cpanmin.us | perl - --sudo App::cpanminus

-- OlivierRaginel - 02 Dec 2010

I decided against CPAN and use already CPANPLUS while not knowing CPANMINUS. CPANPLUS is part of recent Perl installations (CPANPLUS: 5.009005) and in my environment I didn't tweak or configure it at all to see how it goes with it.

Installing deb packages already works using aptitude while only testing it on Debian so far. I am not clear about Ubuntu. On Debian aptitude does everything required. I am using a hash table and don't see another way so far. There is no place in Foswiki (yet) where we could retrieve the mapping from. Well, having it in some topic and maintaining it there could be an option but for now I am happy with the hash.

Actually, nice to hear that there is already some experience regarding configure. So looks that's not an option and I don't have an issue with it.

-- IngoKappler - 02 Dec 2010

Would the name modules.pl be good enough?

Current version works for me but there might be lots of other features possible including e.g. sudo, certain command line switches and RPM support. What I should do is some more documentation.

BTW: aptitude is also available on Ubuntu, so the script should run there too (aptitude on Ubuntu).

-- IngoKappler - 03 Dec 2010

Ubuntu removed aptitude from the standard install in 10.10. Thankfully, one can still install it via apt-get. So please do list aptitude explicitly as a dependency.

-- MichaelTempest - 03 Dec 2010

It seems to me that this type of script is only useful for people having shell access and root privilege. That is a nice tool anyway, and if it is added to Foswiki, the installation documentation for this type of installation (having shell access and root privilege) should be updated to mention it.

Now, I believe that such a script could be highly useful for people who do not have shell access and root privilege, or who want an extremely simplifed installation process (they don't know Perl, don't use Perl, and don't care to have a system-wide installation of the modules even though they may have root access). For those people, it would be nice if the script could also be run by the Webserver, and could suggest installing the missing dependencies in a local repository used by Foswiki only.

The script could be run when configure is first started and warn the users that, if they have shell and root access, they have the option to run the script from shell if they want a centralised installation of the missing modules usable by the whole system, and then restart configure when they are done.

That would be a huge step towards an installation of Foswiki as simple as Dotclear (a Blog engine http://dotclear.org/ - where you just download an installer and copy it in a Web server, which asks you a few simple questions, and then what is needed is downloaded and installed for you).

For the name, maybe dependencies_installer is a better choice, since tools/ already has an extension_installer.

-- RaulFRodriguez - 03 Dec 2010

Thanks for the feedback.

For some of the stuff e.g. to maybe call it from configure probably someone else may need to do it. If the script is run by a non root user it can install modules locally but I didn't fully test that yet.

However, I will try to take some care of the Ubuntu situation (installing aptitude via apt-get).

dependencies_installer sounds good to me, so this is what it will become unless others raise objections.

Updating related docu will also be done by me, once I made it available and probably got some feedback which hopefully won't kill me.

BTW, it does currently:

• Retrieve modules from:

  # Locations of the DEPENDENCIES files
  my $dep_core_path = "../lib/DEPENDENCIES";
  my $dep_ext_dir   = "../working/configure/pkgdata";
  my $dep_tools_dir = ".";
  

• Then prints out everything with current version numbers, so even if no installation is wished, it will provide an overview.
• Extracts missing modules and offers to install either by using CPANPLUS or for deb packages by using aptitude.
• If a package manager is used, then it checks for a remaining diff of modules not available by the package manager and offers at the end to start over for selecting CPAN based installation to fully close the remaining gap.

• I also believe it should work on Windows by using CPANPLUS but didn't test that yet, so don't know what's lurking around this corner.

-- IngoKappler - 03 Dec 2010

Note that the current Extension installer already deals with CPAN dependencies when run from the Shell environment. It does not do it from the web invocation.

It would be nice if whatever you do could be plugged into the shell version of the extension installer, so at least from the command line, the installation of an extension can be a complete package. Enhancing this to support deb, rpm, and other packages would be nice. The CPAN code is currently in tools/extender.pl, in subroutine sub satisfy.

Also note that there is another proposal, UseEnhanceAndMoveSpecFiles If the extension specifications were better organized, it would be easier for configure to report on all required dependencies.

Oh.. and after a quick look at your attached run log, it would be nice if you identified which plugin was triggering the dependency in the list of extension dependencies. This way it might be possible to decide not to install a dependency for example, or attribute some failure to a missing dependency.

I keep thinking of more.. You should probably leverage Configure::Package::checkDependencies http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Configure::Package#ObjectMethod_checkDependencies_40_41 instead of parsing the packages and DEPENDENCIES files yourself. There is quite a bit of logic in comparing revisions and dealing with conditional dependencies (triggers and ONLYIF) embedded in there, and the module Configure::Dependency http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Configure::Dependency If you can stick with these routines, the restructuring of the Spec files in Babar's proposal will not break your work.

-- GeorgeClark - 03 Dec 2010

Gosh! When I started I thought I stay relatively small. Now I have several requests coming down, so I need a table and set expectations. Generally all this sounds nice and thanks again for the input. I already have some working code and I am interested in getting it out. I am not stopping anyone from taking it further as wished or to abondon it in favor of something else.

Status and Feature Overview (being updated)

No, doesn't mean it couldn't be done later. Following is what I see on my personal horizon considering the time I can spend for it.

, %ICON{choice-no}% | radio, 1, %ICON{choice-yes}%, %ICON{choice-no}%, %ICON{wip}% | textarea |" }%

#	Feature	Current Commitment	Done	Comment
1	Retrieve dependencies on its own			Works currently but carries the risk to break in the future due to other developments, especially due to UseEnhanceAndMoveSpecFiles (1.2).	edit
2	Leverage existing dependencies features.			Better foundation to retrieve dependencies. See this and also see. Looks like ObjectMethod checkDependencies () provides what is required to build on. Interesting to implement but can't commit myself currently. Would be good to have it for 1.3 since 1.2 is working fine with the current method.	edit
3	Provide status overview				edit
4	Offer installation from CPAN using CPANPLUS as root				edit
5	Offer installation via aptitude on Debian and Ubuntu (deb packages)				edit
6	Mapping between CPAN modules and avaialable deb packages			Appears to be work in progress but at least I looked at all installable plugins and hope I covered everything or just missed very few. However, not all modules are covered by a deb package, so some will probably always remain to be installed from CPAN. I may put further details of the current mapping in Tasks.Item10170.	edit
7	Consider that aptitude might not be available and offer its installation via apt-get (especially Ubuntu)				edit
8	Package management for RPM based distributions.			Should be relatively easy because RPM provides perl(My::Module). I am interested in it but have to see when I might find time (next year).	edit
9	If a package manager was used offer to re-run analysis for adding still missing modules from CPAN				edit
10	Considering local CPAN based installation into some directory			It's kind of considered but actually the script itself uses what CPANPLUS is configured for. It doesn't change any cpanp configuration while running but if it detects that a user wants to install locally, it provides some hints about related cpanp configurations. I tested the script within a local perlbrew installation and it works as long as it is started e.g. by using the command perl dependencies_installer.pl. If perl is omitted it uses the shebang line of the script which by default points to /usr/bin/perl but is better changed by first using the script rewriteshebang.pl to address local requirements permanently.	edit
11	Consider usage of sudo				edit
12	Support verbose to not bore everybody with many details				edit
13	In verbose mode relate modules to Foswiki plugins for better overview			Not yet.	edit
14	Provide help and manual page				edit

-- IngoKappler - 07 Dec 2010

Ingo, thanks for clarifying this. About 10., my understanding is that the local directory should be [foswiki base dir]/lib/CPAN/lib as indicated in http://foswiki.org/Support/HowToInstallCpanModules#Using_your_private_CPAN_reposito otherwise you have to modify LocalLib.cfg setting the value of @localPerlLibPath or $CPANBASE (see: http://foswiki.org/Support/HowFoswikiFindsPerlLibraries#local_PerlLibPath_and_CPANBASE) if it is set elsewhere.

I have limited knowledge in this respect, so maybe you would like to cross-check all this with CrawfordCurrie or OlivierRaginel

-- RaulFRodriguez - 08 Dec 2010

No problem, I am aware of these details and plan to cover them by related messages. Maybe it is good to provide a default suggestion pointing at /lib/CPAN/lib/ which can be overwritten by user input and then will remind to also address it in LocalLib.cfg.

-- IngoKappler - 08 Dec 2010

I spent some time looking into local installation via cpanp and finally concluded the script itself shouldn't start changing it on its own. What I did, is to warn a user and provide hints about related cpanp configurations, if the script detects that a user wants to install locally (not being root or using sudo). I expect that a user capable of doing local installations, should better know what he is doing, hence tweaking cpanp configuration manually is good to understand how it works and what was changed.

-- IngoKappler - 09 Dec 2010

Changed to AcceptedProposal as the 14 day rule applies by now.

-- IngoKappler - 16 Dec 2010