Feature Proposal: Ship mod_perl_startup.pl suggested by ApacheConfigGenerator to improve efficiency.

Motivation

The ApacheConfigGenerator suggests that one would exist in the tools category: but it (4.2.0) does not.

Description and Documentation

Distribute a mod_perl_startup.pl script to improve effiency under mod_perl environments by precompiling all scripts and modules needed and used by TWiki during Apache startup instead of during each CGI request. The ApacheConfigGenerator already suggests ModPerl::Registry, and it should be taken full advantage of.

This script works well but is no longer by http://develop.twiki.org/~twiki4/cgi-bin/view/Tasks/Item5656. (so long as you use 4.2.1 or trunk -- SD)

Examples

PerlSwitches -T
PerlRequire /var/www/html/docs/tools/mod_perl_startup.pl
SetHandler perl-script
PerlResponseHandler ModPerl::RegistryPrefork
PerlSendHeader On
PerlOptions +ParseHeaders

Impact

WhatDoesItAffect:

Performance %WHATDOESITAFFECT%

edit

Implementation

Example mod_perl_startup.pl:

#!/usr/bin/perl -w
use strict;
use warnings;
use ModPerl::RegistryLoader;
use ModPerl::RegistryCooker;
use ModPerl::RegistryPrefork;
use ModPerl::Registry;
use File::Spec;

my $binurlbase = '/twiki/bin'; # must be set by the user
my $binbase = '/var/www/html/twiki/bin'; # must be set by the user
my $libbase = '/var/www/html/twiki/lib'; # must be set by the user
my $registrymodule = 'ModPerl::RegistryPrefork'; # must be set by the user
use lib '/var/www/html/twiki/lib/CPAN/lib'; # must be set by the user
use lib '/var/www/html/twiki/lib'; # must be set by the user

use CGI;
CGI->compile(':all');
use Algorithm::Diff;
use CGI::Carp;
use Config;
use Cwd;
use Data::Dumper;
use Error;
use File::Copy;
use File::Find;
use FileHandle;
use IO::File;
use Text::Diff;
use Time::Local;
use Archive::Tar;
use CGI::Cookie;
use CGI::Session;
use Digest::base;
use Digest::SHA;
use Jcode;
use Locale::Maketext::Lexicon;
use Net::SMTP;
use Unicode::Map;
use Unicode::Map8;
use Unicode::MapUTF8;
use Unicode::String;
use URI;
use Scalar::Util;
use TWiki;

find({ wanted => \&found, untaint => 1}, $libbase);
sub found {
   if (m/.pm$/) {
      $File::Find::name =~ m|^$libbase/(CPAN/lib/)?(.+\.pm)|;
      my $module = $2;
      my $modbase = $module;
      $module =~ s|/|::|g;
      $module =~ s|\.pm$||;
      print "\nno $module\n" unless $INC{$modbase};
   }
}

my $rl = ModPerl::RegistryLoader->new(
   package => $registrymodule, # must be set by the user
);

chdir $binbase;

foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   $binscript =~ m|^(.+)$|;
   my $bin = $1;
   open BINSCRIPT,'<',$binscript;
   if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
      $rl->handler("$binurlbase/$script", $bin);
   }
   close BINSCRIPT;
}

1;

-- Contributors: JoshuaCharlesCampbell - 26 May 2008

Discussion

Benchmarks on my system (Centos 5.1 (Apache 2.0 perl 5.8.8)):

ab -n 100 -c 2 twiki/bin/view

plain CGI: 2.37/sec

ModPerl::RegistryPrefork: 3.63/sec

ModPerl::RegistryPrefork w/ startup.pl: 3.49/sec

ModPerl::Registry (worker): 2.66/sec

ModPerl::Registry (worker) w/ startup.pl: crashes (double free())

Though the mod_perl documentation recommends doing this it seems to make it slower. I also thought worker was supposed to be faster. Maybe it's opposites day?

-- JoshuaCharlesCampbell - 27 May 2008

#!/usr/bin/perl -w
use strict;
use warnings;
use ModPerl::RegistryLoader;
use File::Spec;

my $binurlbase = '/twiki/bin'; # must be set by the user
my $binbase = '/var/www/html/twiki/bin'; # must be set by the user
my $libbase = '/var/www/html/twiki/lib'; # must be set by the user
my $registrymodule = 'ModPerl::RegistryPrefork'; # must be set by the user
use lib '/var/www/html/twiki/lib/CPAN/lib'; # must be set by the user
use lib '/var/www/html/twiki/lib'; # must be set by the user


my $rl = ModPerl::RegistryLoader->new(
   package => $registrymodule, # must be set by the user
);

chdir $binbase;

foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   $binscript =~ m|^(.+)$|;
   my $bin = $1;
   open BINSCRIPT,'<',$binscript;
   if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
      $rl->handler("$binurlbase/$script", $bin);
   }
   close BINSCRIPT;
}

1;

This improves performance to 4.10 on this system.

-- JoshuaCharlesCampbell - 27 May 2008

#!/usr/bin/perl -w
use strict;
use warnings;
use ModPerl::RegistryLoader;
use File::Spec;

my $binurlbase = '/twiki/bin'; # must be set by the user
my $binbase = '/var/www/html/twiki/bin'; # must be set by the user
my $registrymodule = 'ModPerl::RegistryPrefork'; # must be set by the user

my $rl = ModPerl::RegistryLoader->new(
   package => $registrymodule, # must be set by the user
);

chdir $binbase;

foreach my $binscript (<$binbase/*>) {
   my (undef,undef,$scriptname) = File::Spec->splitpath($binscript);
   $scriptname =~ m|^([^/]+)$|;
   my $script = $1;
   if ($script !~ m/configure|register|resetpasswd|statistics/) { # don't precompile uncommon commands especially configure which has a ton of unnecessary s***
            $binscript =~ m|^(.+)$|;
            my $bin = $1;
            open BINSCRIPT,'<',$binscript;
            if (<BINSCRIPT> =~ m|^\#\!/usr/bin/perl|) {
                    $rl->handler("$binurlbase/$script", $bin);
            }
            close BINSCRIPT;
   }
}

1;

Best version so far.

-- JoshuaCharlesCampbell - 27 May 2008

I think we should put this into 4.2.1 - as its not going to do anything bu make lives easier.

-- SvenDowideit - 27 May 2008

I have no problem with adding this to 4.2.1.

Joshua - the ApacheConfigGenerator contains a 4 liner startup script today. You are very welcome to replace this by your own better script and maintain improvements there as well. That should also benefit users of older versions if they desire to update just the apache config.

-- KennethLavrsen - 27 May 2008

Kenneth - sure thing.

-- JoshuaCharlesCampbell - 27 May 2008

Note also that as of version 4.2.0, many of TWiki's "occasionally used" modules like e.g. TWiki::Attach are compiled lazily using require during runtime and so will not be caught automatically by use TWiki; in a mod_perl startup script. That said, I'd prefer to distribute only a very defensive subset of required CPAN modules in the mod_perl startup routine. If one of the modules is missing in an installation ( CGI::Session would be the most prominent example), Apache won't even start. In that case, there's no way to proceed to the sanity checks in configure, regardless of whether you run configure itself under mod_perl.

-- HaraldJoerg - 27 May 2008

Harald - Please note that the latest version of the script does not include any CPAN modules directly. It does include them by compiling commonly used scripts (add attach to the regular expression to exclude it if you consider it to be not worth precompiling). The goals of being able to sanity check the existence of modules and preload them obviously conflict. Obviously CGI::Session could very well be used on most if not every view -- not precompiling it defeats the purpose of precompiling at all. The solution to this is to not use the mod_perl_startup.pl script until you've got TWiki working without it.

Yes, this script assumes that your TWiki is configured and installed correctly.

-- JoshuaCharlesCampbell - 27 May 2008

Why not make $binbase look like this: use File::Basename;

my $binbase = dirname( $0 ); $binbase =~ s/\/tools$/\/bin/;

The less an administrator has to edit, the better!

-- TimotheLitt - 14 Oct 2008

I'm not 100% sure how this has changed since we switched to the FSA codebase - would anyone that uses mod_perl like to pick this up?

-- SvenDowideit - 07 Mar 2010

this needs to be compared and integrated as appropriate with the tools/mod_perl_startup.pl distributed with ModPerlEngineContrib

-- WillNorris - 07 Mar 2010

See ModPerlStartup

-- CrawfordCurrie - 06 Jan 2014

Commenting is disabled while not logged in