Test Cases Tutorial

Introduction

Foswiki is written in Perl. Like all large and complex software projects, it requires developers to be disciplined and focused on improving the maintainability of the code whenever they make modifications. Like all good Perl projects, this one has a number of test cases designed to aid maintenance and prevent bugs from coming back.

Test cases is one discipline we use to make it easier for other people to maintain our code. As well as helping prevent bugs in the code, well written tests can help reveal the intention behind the code. As long as the tests pass, they represent an inarguable specification of what the code is supposed to do, much more reliably than any documentation ever can.

What's that you said? You were out in the wilderness on a camping trip, and a voice spoke to you out of a burning bush? Now you have seen the light, and you want to write some tests for Foswiki? Good for you! But where do you start? This article is intended to give you an introduction to Foswiki testing methods, and discuss some of the finer points of writing unit tests and topic test-cases.

What's available in Foswiki

The Foswiki codebase has support for internal integrity checking (asserts), two types of automated test, and a methodology for manual tests.

An assert is a section of code that is disabled in production releases, but developers can enable these lines to check for error conditions in the code.

A unit test is a perl program that usually focuses on testing a single API or object in the core.

• Unit tests may test individual modules or APIs in the code, or may spoof an entire user transaction, checking response at each step.
• Unit tests run without a browser.

An automated test case is a topic in the TestCases web. These topics are simple "stimulus-response" tests, designed for testing the Topic Markup Language ( TML) and other more browser-oriented features of Foswiki.

• A testcase is a set of "actual" blocks which contain source TML, each of which has a corresponding "expected" block that contains the HTML expected after post-processing,
• The TestFixturePlugin (in git only) compares the actual result of formatting against the expected result to give a pass/fail,
• Run from a browser,
• You will usually switch between manual inspection and automatic runs of the testcase, so the testcases web has a bunch of infrastructure to support this.

A manual test case is a topic in the TestCases web that documents a series of steps to be followed by a tester, and describes the expected outcome.

In general, automated test cases are preferred to manual test cases, and unit tests are preferred to both. Asserts are used as part of normal programming practice, irrespective of the other test methods being used.

You really ought to read this whole topic; but if you are in a hurry and just want to get a test environment set up (e.g. you are extending a plugin and want to run the unit tests) then you can jump straight to Setting up a Test Environment.

Note: the examples and setup descriptions below are written for Linux, but the test environment also runs under other shells and perl versions, such as Active State perl on Windows.

Asserts

Asserts are enabled by setting the environment variable FOSWIKI_ASSERTS to a non-zero value. This is done automatically during unit tests, but for all other types of test you should edit LocalLib.cfg and add the line $ENV{FOSWIKI_ASSERTS}=1; to the top of the file. This will slow Foswiki down very slightly as it has to execute the tests, so don't benchmark with asserts enabled.

Asserts are implemented using the Assert module. This module defines the function ASSERT which is used thus:

use Assert;
sub do_something {
   ASSERT($i>0) if DEBUG;
...

This will cause an exception to be thrown if $i ≤ 0 when do_something is called. The if DEBUG is required (it is used for conditional compilation).

Asserts should be used whenever a boundary condition needs to be verified before allowing Foswiki to continue. For example, they can be used to check the parameters to functions, or check that the results of a computation are in-range. Asserts are not a substitute for good testcase practice - they are merely a handy sanity check for programmers.

Asserts are normally disabled on production servers because any exceptions are reported to the users' browsers. However, asserts can be a useful diagnostic aid for some bugs that only seem to manifest on production servers. Foswiki's asserts have a "soft" mode where the exception messages are redirected to the error log (typically the apache error.log file) and Foswiki continues running. "Soft" asserts do not interfere with Foswiki's normal operation, but they do slow Foswiki down slightly like regular asserts. To enable "soft" asserts, add the line $ENV{FOSWIKI_ASSERTS}='soft'; to the top of LocalLib.cfg and make sure that the configure Miscellaneous expert option {WarningsAreErrors} is not selected. Be sure to disable asserts completely when you have finished debugging.

Perl Modules from CPAN

Foswiki ships with a number of critical CPAN modules that are not included in core perl distributions. In a normal Foswiki installation, these libraries are appended to the end of the Perl @INC library path and are used only if they are missing from the system libraries. However when running unit tests, the libraries are prepended to @INC so that the tests run with the distributed CPAN modules. If you see different results from the unit tests and the live web environment this can be one of the differences to consider.

The live Foswiki environment can be modified to prefer the distributed CPAN modules over the system installed ones. This is documented in more detail in bin/LocalLib.cfg.txt. Copy the file to bin/LocalLib.cfgand un-comment the line

#$CPANBASE = '';                     # Uncommented: Default path prepended

, and Foswiki will use the shipped CPAN libraries.

Unit Tests

Unit tests for the Foswiki core are kept in git in the UnitTestContrib extension, in the test/unit directory. Extensions such as plugins keep their unit tests in subdirectories below test/unit e.g. test/unit/FuncUsersContrib (this is to avoid accidentally overwriting, or otherwise confusing, the system tests).

Unit tests use a simple custom test framework, in core/lib/Unit. This framework is inspired by JUnit, the famous Java unit testing framework, and provides extensive support for writing object-oriented test programs.

Test packages are divided into Test Suites and Test Cases. Test Suites are just collections of Test Cases and other Test Suites. Test Cases are collections of Test Functions that usually share some common Test Fixtures. A test fixture is the generic term for the environment required to run a test - for example, a test fixture may comprise a set of configuration settings, Foswiki instance, and a set of webs and topics.

Most core test cases test a single class, or group of classes that are closely inter-related (a subsystem). Others may test externally visible components, such as the CGI scripts. So for example, there is a PrefsTest for testing everything to do with preferences, StoreTests for testing the store, and SaveScriptTests for testing the save script. The unit tests are collected into a single Test Suite, called FoswikiSuite.

Extensions, such as Plugins, typically have a single test case that performs all the tests for that extension. Some larger extensions have suites of test cases, organised similarly to the core tests (e.g. ActionTrackerPlugin)

Running unit tests

First set up a test environment.

There are two ways to run tests; you can either run individual tests / test suites (recommended for the core), or you can use the test target in BuildContrib (recommended for extensions) For example, to run all the core unit tests for Foswiki, you just

$ cd test/unit
$ perl ../bin/TestRunner.pl FoswikiSuite.pm

You can also run individual test cases to focus in on a failure.

$ cd test/unit
$ perl ../bin/TestRunner.pl RegisterTests.pm

To run the unit tests for an extension built using BuildContrib - in this case the CommentPlugin

$ cd CommentPlugin/lib/Foswiki/Plugins/CommentPlugin
$ perl build.pl test

If the test fails, a useful tip is to copy-paste the command line for the test run and run it outside the build script.

Sometimes when a test fails (or is interrupted) it may leave parts of the test fixture lying around. It does this so you can debug what went wrong. However the next test run may refuse to run if it detects parts of the fixture still in place. In this case you can pass the -clean option to TestRunner.pl to force it to clean up the previous test run before it starts.

To run tests for an extension, you are best to use the BuildContrib (which must be installed in the Foswiki installation pointed at by the environment variable FOSWIKI_HOME). For example,

$ cd FuncUsersContrib/lib/Foswiki/Contrib/FuncUsersContrib
$ perl build.pl test

If you want to run the tests without using the build script, you need to set up a rather convoluted @INC path. The easiest thing to do is to perl build.pl once, and then copy-paste the command line it prints out. This allows you to focus in on different test files, and can help home in on a failing test.

See the BuildContrib and BuildContribCookbook pages for more information.

Writing unit tests

In brief, tests are organised into test cases, which are collections of test functions which share common setup requirements, or test similar things. Test cases are in turn collected into test suites - for example, FoswikiSuite.pm.

Structure of a test case

A test case is a collection of test functions that usually share some common test fixtures. A test fixture is a bunch of code that creates a private, controlled, environment that a test can run within.

As a simple example here we will use VariableTests, the test case that checks the function of many common Foswiki Variables. The following code comes from an old version of UnitTestContrib/test/unit/VariableTests.pm:

A test case is a Perl class, and for Foswiki it is a subclass of FoswikiTestCase, so a testcase always starts with:

package VariableTests;
use base qw( FoswikiTestCase ); # This base class sets up the basic test fixture

use strict;

sub new {
    my $self = shift()->SUPER::new(@_);
    return $self;
}

This is followed by the two functions set_up and tear_down. To ensure that the test fixture is "clean" before each individual test function runs, the entire fixture is built up before running each test function, and then torn down afterwards. It is critically important that you understand this; test functions are run in random order and one test function can never depend on the results of another.

Building test fixtures

set_up is used to build the fixtures, and tear_down is used to remove them again once the test is complete. Foswiki unit tests are usually based on one of two standard base classes, the basic FoswikiTestCase, or (most commonly) FoswikiFnTestCase, which is derived from it. These are described in more detail below, but first, let's continue to set up a test fixture based on FoswikiTestCase so you can see how it's done:

# Constants used in this test case
my $testWeb = 'TemporaryTestWeb'; # name of the test web
my $testTopic = 'TestTopic';      # name of a topic
my $testUsersWeb = 'TemporaryTestUsersUsersWeb'; # Name of a %MAINWEB% for our test users
my $session; # Foswiki instance

sub set_up {
    my $this = shift; # the Test::Unit::TestCase object

Now invoke the superclass setup. It is very important that this is called first, as it saves the $Foswiki::cfg configuration (which comes from LocalSite.cfg) before we start tailoring it for this test case. It also redirects the log files to files in the test directory.

    $this->SUPER::set_up();

Configure $Foswiki::cfg with appropriate setup for this test. Do not use local paths, and make sure you configure everythingthat might affect the test results. In this case, some of our test functions are going to use user data, so we will have to create some known fake users. That means we have to configure the password manager and protect all the places that password manager uses to store user info.

    $Foswiki::cfg{UsersWebName} = $testUsersWeb;
    $Foswiki::cfg{MapUserToWikiName} = 1;
    $Foswiki::cfg{Htpasswd}{FileName} = '/tmp/junkpasswd';
    $Foswiki::cfg{PasswordManager} = 'Foswiki::Users::HtPasswdUser';

Now fake a simple query and create a Foswiki instance and some test webs.

    # Make up a simple query
    my $query = new Unit::Request("");
    $query->path_info("/$testWeb/$testTopic");
    my $response = new Unit::Response();
    $response->charset("utf8");

    # Create a Foswiki instance
    $session = new Foswiki(undef, $query);

    # and use it to create some test webs
    $session->{store}->createWeb( $session->{user}, $testWeb );
    $session->{store}->createWeb( $session->{user}, $testUsersWeb );
}

If you want to pass some parameters with the query, you can do the following.

    my $query = new Unit::Request({
        webName   => [ $web ],
        topicName => [ $topic ],
        search    => [ $searchString ],
    });

Now, tear_down is responsible for cleaning up after the test has run, so has to restore the state to how it was before set_up was first called:

sub tear_down {
    my $this = shift; # the Test::Unit::TestCase object

    # This will erase the test webs
    $this->removeWebFixture( $session, $testWeb );
    $this->removeWebFixture( $session, $testUsersWeb );

    # This will destroy the Foswiki instance.  We use eval to suppress errors
    eval { $session->finish() };

    # This will automatically restore the state of $Foswiki::cfg
    $this->SUPER::tear_down();
}

Writing test functions

Now we are ready to write some test functions. Test functions are simply functions in the package that start with 'test'. As described above, each test function is run after set_up, and before tear_down, so we know that the individual functions can change anything in the Foswiki environment as long as it was protected by set_up, and will be restored by =tear_down=== . So we can do what we like in the =$testUsersWeb web, because we created our own version of it. But we must not under any circumstances write to the Foswiki web, or any other web that we didn't create in =set_up.

This test case we are writing is testing some Foswiki variables, one of which is %SCRIPTURL%. We want to test this variable with a range of parameters. So, let's write a test function for it.

sub test_SCRIPTURL {
    my $this = shift; # this is an instance of Test::Unit::TestCase; see the online docs for more help

    # We can munge $Foswiki::cfg safely, because it will be restored in tear_down
    $Foswiki::cfg{ScriptUrlPaths}{snarf} = "sausages";
    undef $Foswiki::cfg{ScriptUrlPaths}{view};
    $Foswiki::cfg{ScriptSuffix} = ".dot";

    my $result = $session->handleCommonTags("%SCRIPTURL%", $testWeb, $testTopic);
    $this->assert_str_equals(
        "$Foswiki::cfg{DefaultUrlHost}$Foswiki::cfg{ScriptUrlPath}", $result);

    $result = $session->handleCommonTags(
        "%SCRIPTURLPATH{view}%", $testWeb, $testTopic);
    $this->assert_str_equals("$Foswiki::cfg{ScriptUrlPath}/view.dot", $result);

    $result = $session->handleCommonTags(
        "%SCRIPTURLPATH{snarf}%", $testWeb, $testTopic);
    $this->assert_str_equals("sausages", $result);
}

Rinse, and repeat for everything else you want to test!

There are a wide variety of test cases, both system tests and extension tests. Some are cleaner than others. The quickest way to get a new testcase up and running is usually to cut-and-paste an existing testcase that does something similar.

Some simple rules for writing test functions:

1. never produce output on the terminal (except when debugging)
2. never make a test function wait on user input
3. always build fixtures / set up the configuration for every aspect of the thing you are testing. If your code only works because you happened to have the right setting in LocalSite.cfg, you will regret it later.
4. never rely on the person running a test to "check by eye". They won't.
5. avoid using Foswiki APIs to build test fixtures that are "higher level" than the ones you are testing. There are no hard and fast rules for what "higher level" means, but in general, avoid using an API if there is a chance that it in turn relies on the functionality you are trying to test.
6. you cannot rely on test functions being run in any specific order
7. watch out for caches many Foswiki classes cache data, and it can banjax your tests if you are not careful.

Repeating the same test for different environments ( verify_ functions)

A very common requirements is a test that has to be repeated for a number of different environments. For example, say we want to test Foswiki::Func::getViewUrl for a number of combinations of different values of $Foswiki::cfg{ScriptUrlPath} and $Foswiki::cfg{ScriptSuffix}. We could manually write tests as follows:

sub run_test {
    my $this = shift;
   # Do some tests
}

sub test_1 {
   my $this = shift;
   $Foswiki::cfg{ScriptUrlPath} = 'http://test.one';
   $Foswiki::cfg{ScriptSuffix} = '';
   run_test();
}

sub test_2 {
   my $this = shift;
   $Foswiki::cfg{ScriptUrlPath} = 'http://test.one';
   $Foswiki::cfg{ScriptSuffix} = '.pl';
   run_test();
}

sub test_3 {
   my $this = shift;
   $Foswiki::cfg{ScriptUrlPath} = 'http://test.two';
   $Foswiki::cfg{ScriptSuffix} = '';
   run_test();
}

sub test_4 {
   my $this = shift;
   $Foswiki::cfg{ScriptUrlPath} = 'http://test.two';
   $Foswiki::cfg{ScriptSuffix} = '.pl';
   run_test();
}

This works fine but rapidly becomes tiresome when you have a large number of different combinations. So Unit::TestCase provides support for fixture groups. This function lets you define combinations of environment setup functions that are then used by the framework to generate the actual test functions from template test functions.

This is most easily seen by recoding the above example to use fixture groups:

sub sup_1 { $Foswiki::cfg{ScriptUrlPath} = 'http://test.one'; }
sub sup_2 { $Foswiki::cfg{ScriptUrlPath} = 'http://test.two'; }
sub ss_1 {  $Foswiki::cfg{ScriptSuffix} = ''; }
sub ss_2 { $Foswiki::cfg{ScriptSuffix} = '.pl'; }

# Implement this to return an array of arrays, each of which is a list
# of the names of fixture setup functions.
sub fixture_groups {
    return ( [ sup_1, sup_2 ], [ ss_1, ss_2 ] );
}

sub verify_it_works {
    my $this = shift; # This is the testcase object, just like in a test_ function
    # This is the analog of run_tests in the previous example
    # Do some tests
}

This will result in the following test functions being generated automatically:

verify_it_works_sup_1_ss_1
verify_it_works_sup_1_ss_2
verify_it_works_sup_2_ss_1
verify_it_works_sup_2_ss_2

So verify_it_works is called for all combinations of settings of ScriptUrlPath and ScriptSuffix. You can have as many verify_ functions as you want in a single testcase module, but you can only have one fixture_groups function.

Dealing with expected failures

Every so often we have a testcase that has know failure conditions; for example, it might not work on some platforms. Rather than disabling the failing testcase for all platforms, you can register an "expected failure" for a test under certain conditions. For example, a test that checks filename case may fail on Windows because Windows isn't sensitive to filename case. To register an expected failure, use the expected_failuremethod on the testcase object. For example,

    eval "use Foswiki::Users::ApacheHtpasswdUser";
    if( $@ ) {
        $this->expect_failure();
        $this->annotate("CANNOT RUN APACHE HTPASSWD TESTS: $@");
    }

If an expected failure passes, then it's treated as a test failure.

Unit::TestCase

This is the base class of all unit test classes. It is abstract, almost entirely Foswiki-independent, and provides some basic functionality used in testing.

assert($condition, $message)

Tests if $condition is boolean true (using perl's definition of 'true')

assert_equals($expected, $actual, $message)

Tests equivalence using eq. Use assert_str_equals and assert_num_equals in preference to this method (it is just provided for compatibility with CPAN:Test::Unit)

assert_not_null($ref, $message)

Tests if $ref is defined or not.

assert_null($ref, $message)

Tests if $ref is defined or not.

assert_str_equals($expected, $actual, $message)

Tests two strings for equality using eq.

assert_str_not_equals($expected, $actual, $message)

Tests two strings for inequality using ne.

assert_num_equals($expected, $actual, $message)

Tests two numbers for equality using ==.

assert_matches($expected, $actual, $message)

Tests if $actual =~ /$expected/

assert_does_not_match($expected, $actual, $message)

Tests if $actual !~ /$expected/

assert_deep_equals($expected, $actual, $message)

Tests equality of two hierarchical data structures.

annotate($mess)

Generates $mess in the test log.

assert_html_equals($expected, $actual, $message)

Does a 1:1 HTML comparison. Correctly compares attributes in tags. Uses HTML::Parser which is tolerant of unbalanced tags, so the actual may have unbalanced tags which will not be detected. Use in test functions.

assert_html_matches($expected, $actual, $message)

Tries to match a block of HTML in a larger page of HTML. $expected must be a well-formed block of HTML.

capture(\&function, ...) -> ($text, $result)

Invokes a function while grabbing stdout, so the "http response" doesn't flood the console that you're running the unit test from, and you can analyse the result in your test function. ... params get passed on to &function. Use in test functions.

FoswikiTestCase

As described above, FoswikiTestCase is a simple base class you can use for generating new tests. All it does it to define a set_up function that manipulates $Foswiki::cfg to set up the test environment. Beyond that the tester is responsible for creating webs, topics etc. Test packages that inherit from FoswikiTestCase can safely modify $Foswiki::cfg in individual tests, and the original will be restored when the test completes (or fails). FoswikiTestCase also provides:

removeWebFixture($session, $web)

Cleanly removes a web. Short for: $session->{store}->removeWeb($session->{user}, $web), but also traps and ignores errors. Use from tear_down.

FoswikiFnTestCase

This is a class derived from FoswikiTestCase, which therefore picks up all the functionality of that class and of Unit::TestCase. It adds some more useful Foswikiness.

1. It predefines $this->{test_web} (the name of a temporary test web), $this->{test_topic} (a test topic), $this->{users_web} (a user web), $this->{session} (a Foswiki instance), $this->{request} (the request object used to create $this->{session}) and $this->{response} (the response object to be used by $this->{session})
2. It sets up Foswiki to use the default password and user mapping managers, and registers a test user (username 'scum', wikiname 'ScumBag')
3. It creates the test web and topic, and the users web.

It also adds the following function:

registerUser($loginname, $forename, $surname, $email)

This function uses the standard Foswiki registration code to register a new user.

Changing $Foswiki::cfg

Tests derived from FoswikiFnTestCase should modify $Foswiki::cfg by overriding loadExtraConfig, like this:

sub loadExtraConfig {
    my $this = shift; # the Test::Unit::TestCase object
    $this->SUPER::loadExtraConfig();

    # Configure the environment for the test
    $Foswiki::cfg{Plugins}{NeededPlugin}{Enabled} = 1;
}

... and these settings will be taken into account when FoswikiFnTestCase creates $this->{session} (the Foswiki instance).

Logging from unit tests

There are two different kinds of logging available from unit tests; logging the progress of the actual test run, and Foswiki log files.

Logging the test run

You can log the test run to a file using the -log option on the TestRunner module. If this option is given a log of the test run will be written to the current directory in a file named for the time and date of the test run, with the extension .log.

Foswiki log files

By default unit tests do not leave any traces in Foswiki's logging files warnYYYYMM.txt nor logYYYYMM.txt. If you always need logging for your particular test cases, you can set $Foswiki::cfg{LogFileName} or $Foswiki::cfg{WarningFileName} in the set_up function of your test package to the desired values. If you want to keep the logs for just one single run, set the environment variable FOSWIKI_DEBUG_KEEP to a true value. In this case the warning and log files will be kept in a temporary directory. Example:

$ export FOSWIKI_DEBUG_KEEP=1
$ cd test/unit
$ perl ../bin/TestRunner.pl RegisterTests.pm
.........
$ find /tmp/ -name "FoswikiTestCase.*" -print
/tmp/7vq2n9yDWT/FoswikiTestCase.warn
/tmp/7vq2n9yDWT/FoswikiTestCase.log
# ... examine the log and warn file as you need, then delete the files
$ rm -rf /tmp/7vq2n9yDWT

General advice for new unit testers

Managing test webs and test topics

A common requirement when testing Foswiki is to create test topics. There are several ways to do this, and you will see the core unit tests using some very low-level functions to create test data. This is because the core test suite is built up so that each level of testing tests as few core packages as possible, to avoid excessive dependencies between packages. However if you are testing a module that depends on a working core - for example an extension - the best advice is to use the functions from the Foswiki::Func package. As long as you inherit from FoswikiFnTestCase and create test topics in $this->{test_web} then the fixtures will be cleaned up for you when the test completes.

Writing unit tests for extensions

To write a unit test suite for an extension, proceed as follows:

1. Create a test/unit/ExtensionName directory under the root of your extension checkout area
2. Copy EmptyPlugin/test/unit/EmptyPlugin/EmptyPluginSuite.pm to test/unit/ExtensionName/ExtensionNameSuite.pm
3. Copy EmptyPlugin/test/unit/EmptyPlugin/EmptyPluginTests.pm to test/unit/ExtensionName/ExtensionNameTests.pm
4. Edit test/unit/ExtensionName/ExtensionNameSuite.pm and s/EmptyPlugin/ExtensionName/g
5. Edit test/unit/ExtensionName/ExtensionNameTests.pm= and
   1. s/EmptyPlugin/ExtensionName/g
   2. Change the base class to FoswikiFnTestCase
   3. Add your tests ( test_self is an example test function)

Some extensions that have good examples of unit tests are: CommentPlugin, ActionTrackerPlugin, WysiwygPlugin. Use these for reference.

Automated Test Cases

An automated testcase is a Foswiki topic that contains a set of "actual" blocks which contain source TML, each of which has a corresponding "expected" block that contains the HTML expected after post-processing (also known as the "golden" HTML). The TestFixturePlugin compares the actual result of formatting against the expected result to give a pass/fail.

An automated testcase is any topic in the TestCases web named "TestCaseAuto...". In your testcase topic, enter the golden HTML surrounded by structured HTML comments:

<!-- expected -->
...your golden HTML...
<!-- /expected -->

The golden HTML should be what you expect to be rendered in the final output.

expected has a number of options that are specified by words after expected in the tag - for example, <!-- expected again expand rex -->

expand	Enables expansion of %variables% ( Foswiki::Func::expandCommonVariables ). Normally you should not use the expand option. It is intended primarily for expanding Foswiki variables in URL components, and is used when testing generated HTML which is specific to the installation. It should be used with extreme caution as it assumes that Foswiki doesn't do anything naughty during this expansion.
rex	If there is text which you know can never be literally matched - for example, a generated time - you can enter a regular expression to match it instead, if the rex option is enabled. For example an RE for a time is entered this way: @REX(\d\d:\d\d). Be very careful about using greedy matches. A number of preprogrammed REs, viz. @DATE, @TIME and @WIKINAME, are also provided to simplify expected code.
again	If you have two tests with the same expected text one after the other, you can re-use the expected text from the previous test using this option. The expected text will then be set to the text expected for the previous test. Remember you may need to repeat the expand and rex options again as well.

Anything else you put into an expected tag will be output if there are any test failures, so you can add random text to help identify which expected block failed - for example <!-- expected TESTEYESIGHT -->

You specify your actual test markup in the same way:

<!-- actual -->
<!-- /actual -->

Some notes about the comparison process:

1. The comparison is performed by CPAN:HTML::Diff, which compares the HTML structures found in the text. See the documentation on CPAN:HTML::Diff for help.
   • whitespace is ignored where it has no impact on the way the HTML is rendered.
   • The comparison is insensitive to the order of parameters to the tags, but all parameters must be present.
   • All HTML entities are normalised to style decimal entities before comparison, so < will match <
2. The actual text is read from the raw source of the topic. No processing is done on it (except as described under expand and rex, above)
3. The comparison is done on the <body> of the topic only. At present there is no way to compare the <head>.
4. expected and actual blocks are matched up in the order they occur;
5. If an actual marker is left open in the text ( has no matching /actual ), all text up to the end of the topic will be taken as part of the test. This allows for testing markup at the end of topics.
6. If a /actual tag occurs before a actual tag, all text from the start of the topic up to that tag is taken as the actual text. This allows for testing markup at the start of topics.
7. actual and expected blocks can occur in any order, but there must be one actual for each expected.
8. If there are differences, the report will indicate which actual / expected pair the difference was found in. The pairs are numbered from the start of the topic (number 1).

If possible, always write unit tests in preference to automated testcases. Unit tests are much faster, and usually require a lot less human interaction to run (so will be run more often).

Manual Test Cases

Manual test cases are simply scripts of steps to be followed to test a feature.

They are not recommended and should be used only as a last resort.

Setting up a Test Environment

This section is a step-by-step guide to setting up a test environment suitable for running unit tests and automated test cases.

1. Prerequisites
   • Unit tests must always be run using an unprivileged user. Do notrun them as superuser, as some of the tests rely on being able to protect files against attempts by the same process to write files.
     • Normally web servers are run using a user like apache:apache or www-data:www-data and it's usually not possible to run tests as this user, as they have access controls that would prevent the tests from running.
     • You are recommended instead to run the tests as a user who is in the same group as the webserver user, and adjust the permissions in the Foswiki tree so that members of the same group are granted the same access as the main user.
     • For example, to add user fred to group apache on Linux, use usermod -a -G apache fred
     • You are also recommended to change your umask if necessary so that the webserver user can access the files you create while testing, though this is optional.
     • On Mac OS X you may run unit tests by writing sudo -u www perl ../bin/TestRunner.pl FoswikiSuite.pm, so creating a new user is not necessary.
   • You need a number of CPAN modules to run the tests. Because the requirements change as people add tests, we can't give a comprehensive list here. The best advice is to run the tests, and watch what it can't find. Then install it from CPAN using (for example) perl -MCPAN -e 'install HTML::Parser'
   • To run the HTML validation tests you have to have the HTML::Tidy module installed. The installation of this module can be tricky, as the documentation is rather confused and installing from CPAN doesn't work. Depending on your development platform, you should be able to follow links from http://tidy.sourceforge.net/tofind the necessary pieces.
     • Debian users are recommended to use the apt package i.e. apt-get install libhtml-tidy-perl
2. Checkout the git repository (e.g. distro) you are working on (see GitBasedInstall) and configure it so it is a running Foswiki. This usually involves following the installation steps up to running configure, then running configure once to set the paths. You shouldn't need to do any more than that.
   • ./pseudo-install.pl -A generates a default config suitable for running tests
   • If you created your web server configuration from ApacheConfigGenerator, make sure to set Options FollowSymLinks for the pub directory, in order to get the static files of symlinked plugins served.
   • Keep as many of the default settings as you can.
   • Don't forget to adjust the access controls on files so that both the webserver user and also the user running the testscan write all files in the checkout.
     • If you followed the advice in the prerequisties, above, that means that g permissions have to be set the same as u permissions on all files. There is a script in SettingFileAccessRightsLinuxUnix that can help you do this; simply modify the permissions in the chmod commands e.g. change 0644 to 0664.
3. From the root of your checkout, cd core and use perl -T pseudo-install.pl developer to install the default plugins. (There is a -u flag to uninstall.) The developer target is the same as the default target, plus 3 modules:
   1. Use perl -T pseudo-install.pl BuildContrib to install the build environment
   2. Use perl -T pseudo-install.pl UnitTestContrib to install the unit tests
   3. Use perl -T pseudo-install.pl TestFixturePlugin to install the test fixture plugin (optional to make the unit test run)
4. Set these environment variables
   • export FOSWIKI_HOME=/path/to/foswiki/core
   • export FOSWIKI_LIBS=$FOSWIKI_HOME/lib:$FOSWIKI_HOME/lib/CPAN/lib
5. If you are developing a plugin or contrib (e.g. PutaPlugin) using BuildContrib, then:
   • perl -T pseudo-install.pl -link PutaPlugin to install the plugin in your test environment
     • Enable the plugin using configure. You should now be able to use the plugin in your test Foswiki.
   • cd to the plugin specific directory lib/Foswiki/Plugins/PutaPlugin
   • perl build.pl test
     • This will run the unit test suite in PutaPlugin/test/unit/PutaPlugin/PutaPluginSuite.pm
6. If you are developing a core feature, then
   • cd test/unit
   • perl ../bin/TestRunner.pl FoswikiSuite.pm to run all the tests
     • if you are on Mac OS X: sudo -u www perl ../bin/TestRunner.pl FoswikiSuite.pm (this runs the tests as user www)
   • perl ../bin/TestRunner.pl TestcaseName.pm to run a single testcase

Using Perlbrew in the test environment

Perlbrew is a good tool to test Foswiki using multiple versions of perl. Foswiki supports perl versions from 5.8.4 (5.8.8 in 1.2), so it's important to test both the older versions, as well as the latest versions to detect recent deprecations, etc.

Much of this information is based on http://blog.fox.geek.nz/2010/09/installing-multiple-perls-with.html

• Install perlbrew using your distribution's package manager. Or follow the instructions at the above location for doing a manual install from the raw package.
  • If your package manager does not do so, run perlbrew init and follow the instructions to append perlbrew to your init scripts.
• List the "available" perl versions. (The "i" indicates that it has been installed. Initially none will be marked).

 $ perlbrew available
  perl-5.17.4
i perl-5.16.1
i perl-5.14.2
  perl-5.12.4
i perl-5.10.1
  perl-5.8.9
  perl-5.6.2

• Install your desired versions. (Note that just because a release is not listed, it might still install.) This step takes a long time as it has to download and build the perl release. The -v option just provides more details on the build and is not required.

$ perlbrew -v install perl-5.8.9
$ perlbrew -v install perl-5.16.1

• For each version of perl you have installed, install cpanm (cpanminus)

$ perlbrew switch perl-5.8.9
$ curl -L http://cpanmin.us | perl - App::cpanminus
$ perlbrew switch perl-5.16.1
$ curl -L http://cpanmin.us | perl - App::cpanminus

• If CPAN is not set up on your system, you might need to do an initial CPAN configuration to set the mirrors, etc.

$ perlbrew switch perl-5.8.9
$ cpan
# Answer all setup instructions
» o conf commit
» q

• Verify that perl is working and update any out of date packages.
  • Note that installing cpanoutdated pulls in a number of dependencies also used by Foswiki. If you don't do this, then you will need to install additional dependencies.
  • I've added a step to install outdated modules by piping the output to cpanm.
  • I'm not sure why CPAN::Fresh is needed, but the source site recommended it. It installs a "cpanf" command to retrieve the very latest "fresh" versions of cpan modules)
  • Repeat this step for each version of perl that you have installed.

$ perlbrew switch perl-5.8.9
$ cpanm --interactive -v App::cpanoutdated
$ cpan-outdated
$ cpan-outdated | cpanm
$ cpanm --interactive -v App::CPAN::Fresh

• Install modules needed for Foswiki Unit Tests, and other optional modules required depending upon the configuration

 
$ cpanm Alien::Tidyp HTML::Tidy Devel::Symdump JSON File::Remove
$ cpanm Authen::SASL Encode::compat Lingua::EN::Sentence =Crypt::Eksblowfish::Bcrypt= =Net::SMTP::SSL Crypt::SMIME=

• To run unit tests against a perlbrew release

$ perlbrew switch 5.8.9
$ cd foswiki/tools
$ perl -I ../lib rewriteshebang.pl -d ../bin -d ../tools -d ../test/bin
[press enter to accept the perl path.  It should be the one just chosen with perlbrew]
$ cd ../test/unit
$ ../bin/TestRunner.pl -clean FoswikiSuite

Testing Javascript

For testing Javascript you are highly recommended to investigate the JUnitContrib

Testing extensions

Foswiki extensions (plugins, skins, contribs etc) are tested in the same way as the core (using unit, automated and manual test cases) as described above. The BuildContrib provides a lot of support for running extension testcases, and you are recommended to use it.

To avoid mixing up core and extension tests, we have adopted the convention that the unit tests for an extension are held in a subdirectory of test/unit. For example, the ActionTrackerPlugin stores its tests in test/unit/ActionTrackerPlugin. BuildContrib automatically looks for a test suite called ActionTrackerPluginSuite.pm in this directory when you run perl build.pl test. Unit tests are not normally included in the released package (are not listed in MANIFEST).