Technical Overview

Describing what Foswiki is, and how it works, in 3,000 metre terms, for developers.

Apologies in advance that the some of the language in this overview assumes you have a basic grounding in the following:

• Object oriented design
• Design patterns
• Web applications
• Backus-Naur form (BNF)
• Regular expressions

Don't worry too much if some of these terms are unfamiliar to you, you will still understand most of it.

What is it?

Foswiki is an application that supports editing of web pages, in a standard, unmodified web browser, by end users. Further, it supports the embedding of active and passive macros in these pages that enhance the content. Pages are stored in a file system.

Foswiki started out as a fork of the commercial open-source TWiki^® project, which was itself based on JosWiki. For the most part Foswiki and TWiki are still compatible (for example, you can run TWiki plugins in Foswiki) though Foswiki has grown a number of new features since the split, so readers interested in TWiki beware!

Main features

If you want some (non-compulsory) background reading, Foswiki makes heavy use of the following technologies:

• Common Gateway Interface (CGI) scripts (also refer to FoswikiStandAlone)
• Templates
• Text substitution macros
• Plugins
• Perl
• JavaScript
• Cascading style sheets (CSS)
• Representational state transfer (REST)
• Databases
• Application program interfaces (APIs)

The basic idea is that a request from a browser (such as a command to view a page) is implemented by a user interface method (called by a CGI script or reached by whatever execution technology is in use). This method selects a template and expands the macros in it, calling optional plugins in the process. It then interprets some special syntax constructs in the pages (called topic markup language, or TML) and expands them to HTML. Finally optional Javascript may be used in the page to enhance the user experience, for example to support WYSIWYG editing. This process, from selecting a template through to delivering HTML back to the browser, is called the rendering pipeline. See LifeOfATopic for an in-depth look at this process.

(If you are already familiar with Model-View-Controller (MVC) pattern, then this should all sound very familiar. See ModelViewController for information on how Foswiki modules map to this pattern).

Other operations, such as page saves, are also performed by user interface methods, though more recently increasing use has been made of REST techniques.

Macros vary in power from simple text substitutions to full-text searches.

User authentication and authorization are supported, allowing page-grained access control over pages.

Files can be uploaded (attached) to pages, and optionally embedded in content (for example, GIF images).

Basic tabular data (called "forms") can be instantiated in topics, allowing users to define simple database schemas. Special support is provided for searching and extracting content from these forms. Note that these are not 'normal' HTML forms to submit data.

The default database is a custom text database. this has the advantages of openness and flexibility, and is valued highly by a lot of users. However open APIs are used in several places in the architecture, supporting alternative implementations¹, user authentication, search engine etc.

The basic core is relatively lightweight, but in a typical installation to be really usable it will be extended using other contributed code. There are hundreds of extensions available, but only a small subset is installed by default (for example, email notification, table editing, WYSIWYG editing).

Foswiki has its origins in TWiki, which has its origins in JosWiki. At each stage in its evolution it has picked up nuances in its implementation that appear to the casual observer to be architectural inconsistencies. They usually are. The developers have eliminated a lot of them, but backward compatibility is a continuing concern, so don't be surprised when something appears to be implemented "rather strangely" - there's generally a good reason, if you dig deep enough.

¹ The default text database can store pages using RCS, a custom lightweight RCS-alike, or Subversion. There are currently no publicly available "true" database backends (e.g. MySQL) though an unreleased DBI interface has been developed.

Actions

An overview of the main actions is helpful to understand how it all hangs together. In a default setup, each action corresponds to a CGI script. For full details of the parameters accepted by these actions, visit CommandAndCGIScripts

• view - renders pages for view
• viewfile - (optionally) used for access-control of data file attachments
• edit - renders an "edit" page supporting changes to the page content
• save - saves edited content and returns the browser to a view of what was saved
• rest - invokes a single function. typically invoked using XML RPC
• rdiff - presents a page showing differences between versions of the same topic
• upload - uploads a new data file attachment
• manage - supports management functions, such as renaming
• configure - administrator interface, used to configure the tool

Macros

Macros (formerly known as "TWiki Variables") are similar to macros you may have encountered in programming language preprocessors such as cpp, or other template engines.

To give an example of a typical macro, FORMFIELD renders the value of a field in a topic form:

%FORMFIELD{"fieldname" topic="MyTopic" format="$title=$value"}%

Macros generally conform to the macro syntax specification but be warned! Plugins authors are in a position to parse and interpret their own macro syntax, so this only holds for macros defined by the core, and those defined using the RegisterTagHandler API.

Macros are parsed out of the template and topic text and mapped to a function. This function returns the text that substitutes for the macro in the output. Macros are parsed in left-right-inside-out order i.e. a macro may be embedded in the parameters to other macros, but will be expanded to its full final value before the enclosing macro is in turn expanded. If delayed expansion of argument values is required, most macros support a set of standard "format tokens" that act like escapes on certain characters; for example, $percnt will expand to a % sign. Note that if expansion of a macro results in another macro, then that will also be expanded before any enclosing macro is processed.

The parser that handles extracting the tags can be found in the top level Foswiki package, function _processTags. Argument strings are parsed during the construction of a Foswiki::Attrs object.

Templates and Skins

The framework of an page rendered to the browser is based on a template. These templates are text files that contain HTML pages, generally constructed using macros. In addition to the user-level macros described above, templates have a number of additional template-only macros that are only expanded when a template is loaded. The macros are all named TMPL: -something (e.g. TMPL:DEF). See SkinTemplates for more information.

Skins are simply collections of templates that can be selected between to provide different look-and feel.

A simple skin may only implement the view page, while more elaborate skins implement all available pages. Skins are often rendered additive, with one skin building upon elements defined in a base skin.

Extensions

In addition to the Skins, other extensions to the Foswiki base code include:

• Plugins are perl code modules that hook into specific points in the rendering pipeline. Typically plugins only implement new macros or tags, but some plugins are a lot more sophisticated and have much greater effect on the rendering pipeline (for example, TablePlugin). The best advice for plugins authors is to go and read someone else's code; many plugins authors have written good comments that can help you engineer your own solutions. Have a look at lib/Foswiki/Plugins/EmptyPlugin.pm that provides boilerplate code to get you started. EmptyPlugin contains a full list of the rendering pipeline points that can be hooked into, and is the definitive documentation of these hooks. It also clearly shows how you can easily extend the Foswiki platform with custom REST services.
• Contribs may contain code modules and topics similar to plugins, but generally provide lower-level APIs and are therefore designed to be reused by other extensions. An example is the MailerContrib, which implements mail notification of topics changes.
• Wiki Applications are any combination of Foswiki topics, macros, plugins or contribs that together implement new "end user" applications. These applications can be as simple as a single page with some macros on it ¹, or sophisticated, fully-featured applications that integrate data² as well as entire applications³ from multiple external sources.

¹ For example, a page that lets you easily add comments - an ur-blog
² For example, you might integrate data from third-party bug tracking database with data from a project planning tool with forms data from the wiki to create a "Management Dashboard".
³ Other projects call this sort of thing a "mashup."

Getting down and dirty

Structure of an Installation

Let's take a look at the layout of a typical installation. This should help you get a handle on where things are.

• bin - where all the CGI scripts live
• tools - where Perl scripts that are not CGI scripts live
• locale - where translation files live (I18N)
• templates - where (most of) the templates live
• data - where pages are, if you are using the default (text) database
• pub - where data file attachments are, if you are using the default (text) database
• lib - where all the perl code libraries are
• working - where temporary and working files are kept (also used by extensions to store persistent data)

Now let's kick down a gear and take a look in more detail at a typical installation.

• bin/ - where all the CGI scripts live.
  • The CGI scripts are deliberately as thin and as similar as possible. the real meat of these scripts is in the code library, in the lib/Foswiki/UI directory. In more recent releases this division is even more pronounced, as we are moving to eliminate the duplication inherent in the CGI scripts. Non-CGI scripts are held in tools because in general they must not be accessible through a web browser.
• tools/ - where Perl scripts that are not CGI scripts live. These include maintenance scripts and also example scripts that a site may derive custom scripts from.
• locale/ - where translation files live (I18N).
  • Usually you won't want to touch this (unless you are a translator, in which case see the internationalisation guidelines for more information)
• templates/ - where (most of) the presentation templates live
• data/ - where pages are, if you are using the default (text) database
• pub/ - where data file attachments are, if you are using the default (text) database
  • The directory structure in the data and pub directories mirror each other. The reason for the separation is that many installations permit direct browser access to the files in pub, and it has to be configured differently to data in the web server.
• lib/ - where all the perl code libraries are.
  • CPAN/ contains the minimum set of repackaged CPAN modules that are required to run a basic system
  • Foswiki/ is the root of the actual code (TWiki/ may also be present for backwards compatibility)
• working/ - where temporary and working files are kept (also used by extensions to store persistent data)
  • This is a scratch area that, among other things, stores login sessions, temporary data files for extensions, and pending registration approvals.

Developers working in a Subversion checkout will also see another key directory:

• test/ is where unit tests are kept

A Crude Picture (of the Flow of Control)

Assume we are running one of the CGI scripts (e.g. bin/view)

1. initialize static Foswiki using BEGIN blocks in the Foswiki module
   • this step initializes the appropriate Foswiki::Engine for the calling environment (CGI, Command-line or whatever)
2. call the 'run' function on the engine (see details on FoswikiStandAlone#Runtime_Engines)
   1. create an instance of Foswiki
      1. create per-instance singletons (see 'Main code modules...Runtime' below)
      2. evaluate non-user specific preferences
      3. initialise user system, and identify this session's user
      4. evaluate user specific preferences
   2. look up the $Foswiki::cfg{SwitchBoard} for the script and dispatch to the relevant user interface method.
      1. ...different flow for each script... most scripts compile and print an HTML page. A general overview is:
         1. check access permissions (read the code for details)
         2. check data existence (see the code)
         3. for each block of HTML in the output (e.g. header, body, footer):
            1. expand Foswiki macros (as described in the code) (this may be done recursively)
            2. render TML (headings, tables, lists, WikiWords etc etc)
         4. catch exceptions and report them using oops
         5. print the header and HTML

Main code modules

Singletons

• Foswiki::Sandbox - used to sandbox any calls to system functions that could otherwise be exploited by a hacker.

Runtime

Singleton objects of the following types are created for each runtime

• Foswiki - the session object; includes the macros parser
• Foswiki::Engine - abstraction of the calling environment
  • Foswiki::Engine::CLI - command line calling environment
  • Foswiki::Engine::CGI - Common Gateway Interface calling environment
  • ...
• Foswiki::Request - abstraction of the request object (e.g. CGI)
• Foswiki::Response - abstraction of the response object, used to generate output to the browser
• Foswiki::Plugins - dispatcher object that invoked plugin handlers
• Foswiki::Plugin - object wrapper around the "imperative style" plugins
• Foswiki::Store - high level, application specific interface to the database
  • There are a number of packages within Foswiki::Store:: that are related to the low-level implementation of different back-ends. For example, Foswiki::Store::RcsWrap implements a back-end for using RCS on a filesystem.
• Foswiki::Users - generic interface to user authentication
• Foswiki::Prefs - storage and management of macros

Singleton objects of the following types are created on demand, as required by the runtime.

• Foswiki::Render - the actual rendering engine
• Foswiki::Attach - support for attaching data files to topics
• Foswiki::Templates - template discovery and instantiation
• Foswiki::I18N - Internationalisation (language translations)
• Foswiki::Search - search engine
• Foswiki::Access - access control (authorisation)
• Foswiki::Net - network access (e.g. for sending email)
• Foswiki::LoginManager - subclasses of this abstract base class implement the login processes
  • Foswiki::LoginManager::ApacheLogin
  • Foswiki::LoginManager::TemplateLogin
  • other classes may be added here to implement different login manager
• Foswiki::UserMapping - subclasses of this abstract base class implement the user mapping manager that maps login names to wiki names and email addresses, and handles groups (among other things)

Topic Object Model

We are slowly but surely moving towards a formalization of a topic as an object. The following classes are key to this:

• Foswiki::Form - object representing a data form definition
  • Foswiki::Form::FieldDefinition is the abstract base-class for form-field definitions. A number of other Foswiki::Form::* classes derive from this to provide form definition support.
• Foswiki::Meta - abstract interface to meta-data about a topic, and form data stored in the topic. This is likely to become the "topic object" in future.

Plugins interfaces

The following APIs are exposed to plugins authors.

• Foswiki::Func - this is a thin wrapper over a number of different objects and functions in the core that provides an abstraction suitable for use in plugins (and other extensions). It is never called from within the core. Together with the hooks into the rendering pipeline described above, it isolates plugin developers from core architectural changes.
• Foswiki::Meta

Exceptions

The CPAN:Error module is used for the management of exceptions. The following exception classes extend Error::Simple

• Foswiki::AccessControlException
• Foswiki::EngineException
• Foswiki::OopsException

Utilities

The following low-level utility classes are also defined

• Foswiki::AggregateIterator
• Foswiki::LineIterator
• Foswiki::ListIterator
  • Provide "Java-like" iterators over collections.
• Foswiki::Compatibility - Provides compatibility support for old data formats. Loaded on demand.
• Foswiki::Merge - Provides a number of methods supporting 2 and 3 way merges of topics.
• Foswiki::Plurals - Simplistic English singular/plural mappings
• Foswiki::Time - Parsing and generation of time strings.
• Foswiki::Infix - Classes implementing a simple abstract stack-based parser for infix expressions.

The Developer Environment

Getting started is where new developers start. Once you are familiar with how things work, reference documentation can be found at the developer's bible.

---

TWiki^® is a registered trademark of Peter Thoeny