Mail In to Foswiki

Supports submissions to Foswiki via e-mail This Contrib supports the submission of content to Foswiki topics via e-mail.

What it does

Mails can be pulled out of mail folders using POP3, IMAP, or anything else supported by CPAN:Email::Folder.

The implementation is very simple; a script called mailincron is run every so often (usually by cron or an equivalent offline job scheduler). The script trawls the mail folders you specify and grabs messages that it recognises as being for the wiki.

For example, your wayward child might send a mail like this from an internet cafe in Thailand:

To: foswiki@mum_and_dad.org.uk
From: gapper@isp.co.uk
Subject: Travels.DailyDiary: cashless

*Phuket*: I've run out of money!

The message lands in your email folder at your ISP. Once an hour, a cron job runs the mailincron script, which scans the folder. If it finds any messages that have nothing but a correctly formatted Web.TopicName in the subject line, that topic is appended to (created) with the plain text of the email. The Web must exist, though the topic can be created if necessary.

In our example, the web exists, and so does the topic, so the following text gets appended to Travels.DailyDiary: Attachments to the mail get treated as attachments by Foswiki, and attached to the target topic.

Note that mailincron will only process messages that have arrived since the last time it ran. So if there was an error adding a mail, it won't attempt to add it again even if it is still in the inbox.

Where the mail gets added

Which topic

By default the target topic for mail is specified in the Subject: of the mail. You just put Web.Topic as the first thing on the subject line, optionally followed by a colon and other subject text. If there is no web name specified, then you can configure a default web to accept those mails.

• A mail with a Subject: of Schedule.MyClass will be saved to Schedule.MyClass
• Subject: Schedule.MyClass: timetable will be saved to Schedule.MyClass
• Subject: MyClass will be saved to <default web>.MyClass.
• Subject: MyClass: timetable will be saved to <default web>.MyClass, if the default web is set as Schedule.
• If a topic specified on the Subject: line doesn't exist, it will be created.

You can also configure the module to examine the To: and CC: fields in the mail, and get the name of an existing topic from there instead.

• If a non-existant topic is specified in To: or CC: then it is ignored.
• The valid, existing topic found in the To: or CC: fields is taken, and overrides the Subject:.
• A mail addressed to <[email protected]> will be saved in <default web>.MyTopic
• A mail addressed to <[email protected]> will be saved in Schedule.MyTopic
• A mail addressed to "Mylene Classe" <[email protected]> will be saved in Schedule.MyTopic
• A mail addressed to "Mylene Classe" <[email protected]> will be saved in <default web>.MyTopic

You can also define a 'spambox' for each mail folder. A spambox is a topic that will take all messages that do not have a valid topic.

Where in the topic

The module can use special HTML comments in the topic to decide where to insert new emails within the text.

If it sees the comment <!--MAIL--> it will insert the incoming mail immediately below the comment. MAIL is rather like a Foswiki macro; you can give it parameters. For example, <!--MAIL{where="below"}-->. The following options are available:

• where - can be above, below (relative to the comment) top, bottom (of the topic)
• template set to the name of one of the templates (see below) to change the formatting of comments. The default format is "below" format.

If there is no such comment in the topic, then it just appends it to the end. If there is more than one, only the first is recognised.

Any attachments in the mail get added to the target topic as attachments. Attachments are also listed immediately below the mail body in the topic.

The templates are loaded from System/MailInContribTemplate, and the recommended way to change the look and feel of the output pages is to copy MailInContribTemplate as MailInContribUserTemplate and edit to taste.

How the contributor is identified

The user identity is used for access control checks on the target topic, so you can use Foswiki access controls to protect target topics.

You can configure the module to look at the From: entry in the email, and if a registered user has set that email (see ChangeEmailAddress), then they are identified as the contributor. Note that there is a security risk here, as the From: address in e-mail can easily be spoofed.

You can also optionally set a default user for an inbox, so if the user can't be identified from the mail, it will fall back to the default.

Which part of the mail is added

The module defaults to extracts the plain-text portion of the mail; the HTML portion (if present) is discarded. When the module extracts the plain-text portion, it also discards inline images associated with the HTML (because, whilst they are part of the mail, they are not attachments).

Experimental: You can configure the module to extract the HTML portion instead, in which case the plain-text portion is discarded. The module then also extracts and attaches inline images referenced in the mail (MailInContrib discards inline images if they are not referenced).

Content processing

You can also configure the module to process the mail content before adding to the target topic.

This is particularly important for html email, which could potentially contain web bugs and/or dangerous javascript. The default processing for html email tries to remove scripts, references to resources that are not part of the mail and references to resources that are not on the same server as the wiki. See the WARNING, below.

Available content processors

• Foswiki::Contrib::MailInContrib::NoScript
  Removes script tags and attributes like onclick, onload, onmouseover etc

• Foswiki::Contrib::MailInContrib::NoInlineContent
  Removes tags that reference content (e.g. images) contained in the message itself. Note: This does not remove attachments.

• Foswiki::Contrib::MailInContrib::FilterExternalResources
  Removes tags (e.g. img, script and style) tags that reference content on external servers. By default, only inline content (i.e. content attached to the message) and content that might be attached to the wiki is allowed. This processor is configurable via two options (pick one):
  • safedomains - a comma-separated list of domains which it is safe to reference, e.g. 'example.com, intranet.example.com'. Prefix a domain with *. to allow all sub-domains e.g. '*.example.com, services.local'. Do not include the http:// portion of the URL, that will be added automatically.
  • safeurlpattern - a regular expression that matches URLs that are safe to reference. That must include the http:// portion, if any.

Example:

 {
   folder => 'pop://example_user:password@example.com/Inbox',
   onError => 'log',
   onNoTopic => 'error',
   onSuccess => 'delete',
   topicPath => 'to subject',
   content => {
     type => 'html',
     processors => [
       { pkg => 'Foswiki::Contrib::MailInContrib::NoScript },
       { pkg => 'Foswiki::Contrib::MailInContrib::FilterExternalResources', 
         safedomains => '*.example.com' },
     ],
   },
 },

Creating custom content processors

Content processors must be implemented as a perl Package that sub-classes Foswiki::Contrib::MailInContrib::Filter. Content processors must provide the process method:

sub process {
    my $this = shift;
    # my ($content) = @_;
    #
    # You can work on $content in place by using the special perl
    # variable $_[0]. These allow you to operate on $content
    # as if it was passed by reference; for example:
    # $_[0] =~ s/RandomDigit/int(rand(10))/ge;
}

Foswiki::Contrib::MailInContrib::Filter provides the following methods:

• $this->options - returns a reference to the options hash for this processor, from $Foswiki::cfg{MailInContrib}->[...]->{content}->{processors}->[...]

• $this->mime - returns the MIME object for the content being processed

• $this->topLevelMime - returns the MIME object for the mail that contains this content - useful for accessing the mail headers

• $this->box - returns a reference to the hash for this mailbox, from $Foswiki::cfg{MailInContrib}->[]

• $this->processTag($content, $filter, $handler) - Processes $content as HTML, and calls a handler on each tag that matches the filter criteria. The tag is replaced with the handler's return value. The $content is modified in-place.
  • $filter is a hash reference. Keys:
    • tag - value is a reference to an array of tag names to match on. If the array is empty, then the handler is called for all tags.
  • $handler is a reference to the callback method. It is called as $this->$handler($tagAndContent, $tagName);
  • Example: To remove all table tags: $this->processTag( $_[0], { tag => ['table'] }, sub { return ''; } );

• $this->processAttribute($content, $filter, $handler) - Processes $content as HTML, and calls a handler on each attribute that matches the filter criteria. The attribute is replaced with the handler's return value. The $content is modified in-place.
  • $filter is a hash reference. Keys:
    • tag - value is a reference to an array of tag names to match on. If the array is empty, then the handler is called for all tags.
    • attr - value is a reference to an array of attribute names to match on. If the array is empty, then the handler is called for all attributes.
  • $handler is a reference to the callback method. It is called as $this->$handler($attributeAndQuotedValue, $tagName, $attrName, $attrValue, $quote);
  • Example: To remove all class attributes from div and span tags: $this->processAttribute( $_[0], { tag => ['div', 'span'], attr => ['class'] }, sub { return ''; } );

WARNING

This software is provided in the hope that it may be useful. The authors make no warranty, implied or otherwise, about the suitability of this software for safety or security purposes.

The authors shall not in any case be liable for special, incidental, consequential, indirect or other similar damages arising from the use of this software.

If in any doubt, do not use it.

Installation

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. Use "Find More Extensions" to get a list of available extensions. Select "Install".

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See http://foswiki.org/Support/ManuallyInstallingExtensions for more help.

Note: The CPAN Email modules uses many pluggable modules to support different folder types. It's impossible to cover all the options here; all we can suggest is that you try running the script from the command-line and resolve missing modules as you find them.

• Set up cron (or equivalent) jobs to run mailincron. You must cd to the bin directory, and invoke the sript from there (this is so mailincron can read setlib.cfg)

Because of the security issues involved (passwords for the mailboxes etc.) configuration uses variables set in your LocalSite.cfg. The easiest way to set them is using configure.

The mailincron script takes one optional parameter, debug, which takes a boolean value e.g. debug=1. If you pass anything other than 0 or the empty string in debug, the script will scan the mail folders, describe what it would have done, and exit, without modifying any folders, the wiki, or sending any mails.

To run the script you need to set up a cron job. For example, to transfer mail into the wiki once every hour you might write:

0 * * * * cd /home/foswiki/bin && ../tools/mailincron 2>&1 >> /home/foswiki/logs/mailincron.log

You must run the script from the bin directory. Make sure that the cron is run by a user with the permissions needed to read and write the data directory.

Contrib Info

Another great Foswiki extension from the WikiRing - working together to improve your wiki experience!

Many thanks to the following sponsors for supporting this work:

• Evolved Media Network

Author:	Foswiki:Main.CrawfordCurrie (http://c-dot.co.uk)
Copyright ©:	2008-2014 Foswiki Contributors 2005, TWiki Contributors
License:	GPL v2 (GNU General Public License)
Dependencies:	Name Version Description Time::ParseDate >=2003.0211 Required. Available from the CPAN:Time::ParseDate archive. Email::Folder >=0.84 Required. Available from the CPAN:Email::Folder archive. Email::FolderType::Net >=1.02 Required. Available from the CPAN:Email::FolderType::Net archive. Email::MIME >=1.82 Required. Available from the CPAN:Email::MIME archive. Email::Delete >=1.02 Optional, recommended. Available from the CPAN:Email::Delete archive.
Version:	3.0
Release:	7 May 2014
Change History:
3.0 (7 May 2014)	Foswikitask:Item8384: Support HTML email. Foswikitask:Item12890: MAILHEADER macro supports inclusion of arbitrary mail headers in the output. Foswikitask:Item8523: Add configure setting to ignore mail timestamp Foswikitask:Item8595: Remove the topic name from the subject. Foswikitask:Item11379: Stamp the run with the time it was started, not the time it ended, to avoid race condition on incoming mails. Foswikitask:Item8293: start id numbering at 1 for POP3. Foswikitask:Item9887: don't skip on* actions at start and end of string.
2.0 (3 Feb 2009)	Foswikitask:Item456: ported Foswikitask:Item4955: config options moved to 'Mail and Proxies' Foswikitask:Item5829: force new revision on each submission Foswikitask:Item5843: extract target topic from CC: field - general improvement to rules for extracting the target topic
1.0 (10 Mar 2005)	Initial version
Home:	Foswiki:Extensions/MailInContrib
Support:	Foswiki:Support/MailInContrib

Related Topics: DefaultPreferences, SitePreferences