Foswiki2MigrationIssues

Previous to Foswiki 2.1.2, there were issues in the bulk_copy utility. In addition, bulk_copy can be very slow on large wikis with large histories. This document presents an alternate approach.

This document applies only for users

• migrating their Foswiki system from a 1.0.x or 1.1.x release
• not already running with {Site}{CharSet} set to utf-8

The migration to Foswiki 2.0 has several steps

1. Migration to utf-8: This is recommended. Foswiki 2.0 will work best with a utf-8 store. Even if you don't do this, internally Foswiki will convert each topic read from the store to utf-8, and then back to the old {Site}{CharSet} on save, so it is much preferable to do it once.
2. Conversion to PlainFileStore: This is optional, and not recommended until the bulk_copy.pl script is fixed, or an alternative is provided.
3. Conversion of DENY ACLs: This is strongly recommended, but can be skipped if the {AccessControlACL}{EnableDeprecatedEmptyDeny} option is enabled.

The Foswiki release UpgradeGuide recommends accomplishing steps 1 & 2 above using tools/bulk_copy.pl. This is broken

Prepare for all upgrade steps

(This omits steps covered in the UpgradeGuide)

Choose the character encoding to be used in your site. Previous versions of Foswiki defaulted to iso-8859-1 encoding (The "Latin Alphabet 1, intended for US and Western European languages). Foswiki 1.2 defaults to UTF-8 encoding, which provides better support for international character sets. You must match the prior encoding if you intend to use topic created on an older version of Foswiki. If you change encoding of existing topics, you must use the non-default Foswiki:Extensions.CharsetConverterContrib to modify the character encoding. Changes in character encoding must be done using the RCS based store.

Choose your desired Store. When migrating from Foswiki 1.x, you should continue to use the RcsStore. Do not select the PlainFileStore!

Changing Character sets

1. Download and install the Extensions.CharsetConverterContrib into your existing Foswiki 1.x system.
   • This tool operates in-place and cannot be re-run. Be sure to have a good backup or run it on a copy of the production system
2. Take a backup of your 1.x system, or work from a copy!
3. Make sure you have full update rights for the data files If the conversion fails due to a write protected topic file, you will need to restore, correct the issue, and run again.
4. Execute the tools/convert_charset.pl script with the "-i" option to inspect your system

      perl -I lib tools/convert_charset.pl -i
      

5. If all looks okay, run it in-place, and then Update the configuration: Change {Site}{CharSet} to utf-8 in the lib/LocalSite.cfg file.

Note that this utility does not update the .htpasswd password file, or the working directory contents. If any topic or web names have changed, (for ex., they included an umlaut in the name) then the entries in .htpasswd and/or working will need to be repaired.

Proceed with Foswiki 2.0 installation

• Be sure to select RcsLite or RcsWrap stores
• The store encoding will default to utf-8

Copy parts of the working directory

The working directory contains some critical information for some extensions, found below the foswiki/working/work_areas directory. Extensions use it to store persistent information critical to operation. For example, the MailerContrib directory contains the timestamps of the last notification email run per web. If not copied, the next mailnotify run will notify all recorded changes. This is the most common data that should be copied. Review other non-default extensions to determine if anything else should be copied.

Copy content from non-default webs in old installation to the new installation

Be sure to select an "RCS Store" RcsWrap or RcsLite on the new installation. The PlainFile store is not compatible with topic history written on previous versions of Foswiki. If you have created or updated topics using PlainFileStore, you should either start over, or plan to to remove all ,pfv directories from the system so that there is no history in the PlainFileStore format.

If you intend to change character encoding to UTF-8, you should install and run the Foswiki:Extensions.CharsetConverterContrib on your existing data before migrating it to Foswiki 1.2. The CharsetConverterContrib only supports the RCS style Store. Once you have converted the characterset to UTF-8 then you can convert to PlainFileStore

Copy your local webs over to the data and pub directories of the new installation. Do not copy the default webs: <old_system_web> System, Main, Trash, Sandbox, _default, and _empty.

• Make sure the data and pub directories, as well as the files within them, are readable and writeable by the web server user.
• Note: Foswiki's WebChanges topics depend on the file timestamp. If you touch the .txt files make sure to preserve the timestamp, or change them in the same chronological order as the old file timestamps.

Verify that existing topics are operational and (if you converted to UTF-8) that any international characters have been properly converted and are displayed correctly.

Copy users, user topics, and site customizations to Main web

Copy all topics and attachments from <old_users_web>: copy all files from <oldwiki>/data/<old_users_web>/ to <newwiki>/data/Main/, and copy all files from <oldwiki>/pub/<old_users_web>/ to <newwiki>/pub/Main/ . Do not overwrite any topics already present in the <newwiki>/data/Main/ directory.

• In addition to all the user topics, if you have created <old_users_web>.NewUserTemplate in the old installation, this step will copy over your template for user topics to the new installation.
• Ensure that the topic defining the admin group in your old installation is copied over. The admin group is defined in the Security setup pane of the configure page, in the {SuperAdminGroup} setting (visible when Expert mode is enabled). You can do either of the following:
  • Set the {SuperAdminGroup} setting in your new installation to the old admin group.
  • Move the contents of the old admin group to the new admin group. To avoid having to change all references to the old admin group, you must still keep the old admin group defined: set it so its only member is the new admin group, and the new admin group is the only user who can change or rename the old admin group topic.
• If your old installation did not customize {LocalSitePreferences} on the configure page, or if you did customize {LocalSitePreferences} but kept your site preferences within the <old_users_web> web, then this step will also copy over your site preferences to the new installation.

For upgrades from an older Foswiki installation:

• Manually merge all users from the <old_users_web>.WikiUsers topic in the old installation to the Main.WikiUsers topic in the new installation. If the new installation does not yet have an initial Main.WikiUsers topic, then copy <oldwiki>/data/<old_users_web>/WikiUsers.txt to <newwiki>/data/Main/WikiUsers.txt.
• Verify that the following default users are present in the Main.WikiUsers topic:
  • ProjectContributor - the Foswiki documentation is attributed to this user
  • RegistrationAgent - special user used during the new user registration process
  • UnknownUser - used where the author of a previously stored piece of data can't be determined
  • WikiGuest - guest user; used as a fallback if the user can't be identified
• If any of the default users are missing, then add them in manually to Main.WikiUsers, using the corresponding entries in Foswiki:System.UsersTemplate as an example.
• If you use data/.htpasswd for authentication, copy this file from the old installation to the new one.
• If you have customized <old_system_web>.UserRegistration, then either copy over <oldwiki>/data/<old_system_web>/UserRegistration.txt and <oldwiki>/data/<old_system_web>/UserRegistration.txt,v to the <newwiki>/data/System/ directory, or modify System.UserRegistration in the new installation to contain your customizations.

Copy over any topics and attachments you want to preserve from the Sandbox web in the old installation: copy the desired files from <oldwiki>/data/Sandbox/ to <newwiki>/data/Sandbox and from <oldwiki>/pub/Sandbox/ to <newwiki>/pub/Sandbox . Some pages you may wish to preserve are the WebHome topic and the WebLeftBar topic (if you had created it in the old wiki installation). The Sandbox web often contains work-in-progress topics that users will want to keep.

Make sure the data and pub directories, as well as the files within them, are readable and writeable by the web server user.

Execute your test plans for authentication and authorization. Test that users that you have transferred from the old installation can login with any problems, and that access controls work appropriately: check that users are able to view and edit pages for which they have access, and are denied permission to view or edit pages for which they do not have access. Also check that pages restricted to the admin group are not accessible by non-admin users, and that administrators continue to have access.

Convert empty DENY ACLs to ALLOW * wildcards

By default, empty DENYTOPIC rules will be ignored by Foswiki 1.2. You must change them to the equivalent ALLOWTOPIC * rules. The tools/convertTopicSettings.pl utility will scan the Webs & Topics, and will perform several optional conversions on the topics.

Get help text for convertTopicSettings

perl tools/convertTopicSettings.pl -help

Scan all webs / topics, report any topics with empty DENY rules

perl tools/convertTopicSettings.pl

Replace all empty DENY rules with ALLOW * wildcards

perl tools/convertTopicSettings.pl -fixdeny -update

Same, but convert all ACLs into META settings from inline topic settings, for just the Sandbox web

perl tools/convertTopicSettings.pl -fixdeny -convert -update Sandbox

Convert ALL settings into META settings, not just ACLs, for the Sandbox and Customer webs

perl tools/convertTopicSettings.pl -fixdeny -convert -all -update Sandbox Customer

When convertTopicSettings saves the modified topics, they will be saved by user UnknownUser.

-- GeorgeClark - 15 Jul 2015