LogDispatchContrib

Advanced logging using Log::Dispatch

Introduction

LogDispatchContrib is a pluggable logger implementation that can replace the default loggers shipped with Foswiki. It provides logging methods similar to the Compatibility logger shipped with Foswiki, and adds several alternative methods to support alternate destinations.

The following log methods are shipped by default:

• LogDispatch::File: A simple file based logger. It does not do any file rotation or date stamping of the files.
• LogDispatch::FileRotate: A file based logger that rotates files on recurring dates while keeping only a certain number of files.
• LogDispatch::Screen: Logs to STDERR, which is typically written to the Apache (Web Server) error log..
• LogDispatch::Syslog: Logs to the local Syslog daemon. By default logs to the user facility.

The architecure of LogDispatch is pluggable. It is expected that additional loggers will be added, such as Log::Dispatch::DBI, Log::Dispatch::Jabber, etc.

Note that all loggers can be enabled to log all levels to all methods. When a log message is requested by a Foswiki function, Log::Dispatch examines the log level of the request, and passes the message to each enabled logger in sequence that has requested that log level.

Foswiki Logging

Log::Dispatch supports the 8 logging levels typical in most logging systems. debug-0 info-1, notice-2, warning-3, error-4, critical-5, alert-6 and emergency-7, however Foswiki uses only a subset of these levels:

• debug level 0 is used for some error reporting, and when debugging is enabled in the core and for some extensions.
• info level 1 is used for transation logging. The following fields are written by default:
• warning Errors and other failures are written to the warning log level

Fields available for logging

Field	Description
timestamp	A Foswiki timestamp in the format of 2013-12-04T03:12:39Z
level	Log level: debug, info, warning
user	Typically the logged in user / WikiName
action	The web transaction: save, view, viewfile, etc.
webTopic	The fully qualified web.topicname
extra	Catch-all field contains messages and other information
agent	The browser agent when available
remoteAddr	The remote IP address
caller	For errors. the perl module that triggered the error

extra is a catch-all that is used in a number of different ways:

• Topic actions, it includes things like new topic name for move/copy, special commands like delRev, etc.
• Login/Logout actions including authentication failures
• From / To revision levels for RDiff
• Email address changes
• Whether topic was cached in view operation
• Registration email, login name, etc.
• Filename for attach operations
• The internal stack call trace for failures
• ... many others

It's our intention to eventually move much of this information to fixed fields to facilitate more powerful log processing.

Logger features and configuration

Common configuration for all loggers

The followng common features apply to all Loggers:

• $Foswiki::cfg{Log}{LogDispatch}{MaskIP}: Hide IP Addresses logged in the event log. Default: IP addresses are logged. Two possible options can be configured: x.x.x.x will cause the remoteAddr field to be replaced with x.x.x.x. hash will cause the IP address to be scrambled, so that the original address is not easily identifiable, but multiple transactions from the same IP address will be logged using the same address. Note: the hash method does not work for IPv6 addresses.
• $Foswiki::cfg{Log}{LogDispatch}{EventIterator}: Specified which logger should be used to retrieve messages when the eachEventSince function is used. This is an expert level setting. See the help in bin/configure for more information.

The log layout feature applies to all of the flat record loggers.

• $Foswiki::cfg{Log}{LogDispatch}{ <LoggerName> }{Layout}; Specifies an array of fields that defines the flat log record layout. This is an expert level parameter

The log file layout default settings are set to match the Plain and Compatibility file loggers as closely as possible. The setting is an array of fields, the first entry being the delimiter used when joinin the fields. The left side of the list info => is the filename. The right side is the list of fields. So for a simple example,

   fileprefix => ['delim', 'field1',   'field2'    ]
   iphits =>     [  '|',   'timestamp'.'remoteAddr']

would write out a simple file iphits.log, consisting of the timestamp and remote IP address.

|2013-12-04T03:12:39Z|127.0.0.1|
|2013-12-04T03:12:43Z|10.20.1.1|

In order to generate logs fully compatible with the old loggers, the following additional values are allowed:

• Delimiter can include leading/trailing space, " | ", Space at beginning/end of record are trimmed.
• The array can be nested one level deep. For example, in the below example, the first field combines the timestamp and log level joined with a space
• The special field '*' is a wildcard, and logs any extra fields that were omitted from the layout.

   $Foswiki::cfg{Log}{LogDispatch}{File}{Layout}=  {
        info => [' | ',                   # Delimiter
                   [' ',                    # Field 1 - inner delimiter
                    'timestamp',            # Field 1 first part
                    'level'                 # Field 1 2nd part
                   ],                       # End of field 1
                 'user',                  # Field 2
                 'action',                # Field 3
                 'webTopic',              # Field 4
                  [' ',                     # Field 5 - inner delimiter
                   'extra',                 # Field 5 first subfield
                   'agent',                 # Field 5 2nd subfield
                   '*',                     # Field 5 - anything left over after record is built.
                  ],                        # End of Field 5
                 'remoteAddr'             # Field 6
               ],
        DEFAULT => [' | ',              # This layout covers any other log levels not explicitly defined
                    [' ',                   # Field 1 - 2 subfields joined with space
                     'timestamp',
                     'level'],
                    [' ',                   # Field 2 - 2 subfields joined with space
                     'caller',
                      'extra']
                   ],
        };

LogDispatch::File - simple file logging

When enabled, the default configuration:

• debug messages (Level 0) are writen to debug.log
• info messages (Level 1) are written to events.log
• notice-emergency (Levels 2-7) are written to error.log

This can be overridden with an "Expert" level configuration variable, which assigns a range of log levels to a filename prefix. The left side of the => is the filename prefix, and the right side is the lower:higher log level to be written to that file.

There is an extra filter option that can be used to direct a subset of log messages to a file. The log request is assembled into the complete message, and then the optional 3rd parameter is applied as a regular expression against that message. If the regex matches, the record will be logged.

This example implements the default log levels as shown above, and then applies two filtered loggers:

• Authentication messages (matching string AUTHENTICATION) are sent to auth.log
• registration start, requests and rejections are logged to reg.log

$Foswiki::cfg{Log}{LogDispatch}{File}{FileLevels} = {
    debug  => 'debug:debug',
    events => 'info:info',
    error  => 'notice:emergency',
    auth   => 'info:info:AUTHENTICATION',
    reg    => 'info:warning:\\| [Rr]eg(start|ister|istration)',
    };

LogDispatch::FileRotate - simple rotating files

When enabled, the default configuration:

• debug messages (Level 0) are writen to debug.log, debug.log.1, debug.log.2, ... , debug.log.<max>
• info messages (Level 1) are written to events.log, event.log.1, event.log.2, ..., event.log.<max>
• notice-emergency (Levels 2-7) are written to error.log, error.log.1, error.log.2, ... , error.log.<max>

The FileRotate logger is very similar to the simple File logger while rotating files on recurring dates and only keeping a certain number of files. For instance to keep monthly logs of the recent year use below configuration:

$Foswiki::cfg{Log}{LogDispatch}{FileRotate}{Recurrence} = 'monthly';
$Foswiki::cfg{Log}{LogDispatch}{FileRotate}{MaxFiles} = 12;
$Foswiki::cfg{Log}{LogDispatch}{FileRotate}{FileLevels} = {
    debug  => 'debug:debug',
    events => 'info:info',
    error  => 'notice:emergency',
    };

Like for LogDispatch::File messages may be filtered using a third parameter in the {FileLevels} configuration. See above for an example.

Log::Dispatch::Screen - log to STDERR

This method directs messages to the STDERR output. STDERR is normally recorded in the Apache error log. When enabled, the default configuration sends error, critical, emergency and alert messages to STDERR. There are two configuration options, specifying the lowest and highest log level to be sent to STDERR.

Log::Dispatch::Syslog - system logging

The Syslog logger uses the system Syslog facility to send messages to the system log. It is disabled by default. It has 5 configuration options:

• Facility - specifies the Syslog Facility set in the log message. Default is user.
• Identifier - specifies a string that is prepended to each log message. Default is Foswiki.
• MinLevel and
• MaxLevel specifies the range of log levels sent to the Syslog logger.
• Logopt Logger options understood by Syslog. This is an expert level configuration. See http://perldoc.perl.org/Sys/Syslog.html and bin/configure help for details.

Possible future enhancements

• Ensure consistent format for all logs - include the User, Web.Topic and IP address
• Add DBI logger
• IPv6 support
• Add Jabber, DBI, Windows even type loggers

Installation Instructions

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. "Extensions Operation and Maintenance" Tab -> "Install, Update or Remove extensions" Tab. Click the "Search for Extensions" button. Enter part of the extension name or description and press search. Select the desired extension(s) and click install. If an extension is already installed, it will not show up in the search results.

You can also install from the shell by running the extension installer as the web server user: (Be sure to run as the webserver user, not as root!)

cd /path/to/foswiki
perl tools/extension_installer <NameOfExtension> install

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See https://foswiki.org/Support/ManuallyInstallingExtensions for more help.

Dependencies

Name	Version	Description
Time::ParseDate	>=2015	Required
Log::Dispatch	>=0	Required
Log::Dispatch::FileRotate	>=0	Optional

Change History

27 Oct 2020:	2.10 - fixed FileRotate based loggers
01 Nov 2018:	2.00 - rewrite and code reorg; replaced FileRolling with FileRotate
15 Oct 2012:	1.01 - improved filter implementation, improved file layout implementaiton