Feature Proposal: New update method for system topics

Motivation

With every update I have to merge the updates in the distributed topics with my own modifications. As this is a lot of work, I tend to skip several updates before going through the pain of manual merging again. This keeps my users struggling with bugs that are already solved.

Description and Documentation

For custom set-ups it is often necessary to modify topics that are distributed with Foswiki. In the current situation there are two choices:

• overwrite all of your own modifications with the updated version from the distribution, or
• do not update the distributed topics at all, resulting in outdated documentation.

The "update" downloads also contain many topic files. For version 1.0.9 the directory /data/ contains 456 files in the full download and 407 files in the "update" download. Even the subdirectories /_default/ and /_empty/ have all files updated!

Examples

For example, I wanted to remove all links to the topic Main.WikiUsers. There are a lot of links to this page in the System Web (mostly via the macro % WIKIUSERSTOPIC %). After I was finished there was an update available. I had to untar/unzip all files in the data directory to temporary directory and:

• find all files that are part of the distribution that I modified,
• Check all my modified topics with the updated files
• Re-apply my modification in the updated file OR update my version according to the distributed file, whichever seems to be the least work
• Administrate my progress as it takes multiple days with fragmented time periods to do this tedious work.

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

None yet.

-- Contributors: ArnoldAtMallos - 11 Apr 2010

Discussion

The first comment I would make is don't modify System topics. The second comment I'll make is contribute back fixes that affect everyone. Having said that, I accept that some customisation may (on very rare occasions) be required to ease new user training, or interleave your own site-specifics. There are some standard approaches for handling this, the easiest of which is:

• Back up the released System web into a local directory before making your changes. You can then diff your changed System web against this "golden" copy at any time to see the changes in the form of a patch, which can then be applied to the upgraded System web.

A more sophisticated approach involves making a three-way merge (diff3 --merge) to merge the backed-up original, the new release, and your modified version of the original. Other approaches include using a local source-code control repository to track and merge changes.

You do get some conflicts on occasion, but I'm afraid you have to accept this if you're going to modify released topics.

I don't think this should be a feature request; it's more like an appropriate subject for an FAQ in the Support web.

-- CrawfordCurrie - 11 Apr 2010

Arnold, what you describe is a quite pertinent problem that has been discussed in various contexts recently. My advice for you would be to install AutoTemplatePlugin and overlay (system) topics with custom content. Some extensions use this plugin on a regular base to replace parts of standard foswiki without touching it, thus making updates less of a hassle.

There are other mechanisms to address your problem. The upcoming Foswiki 1.1 will come with improved mechanisms to let you hook into standard wiki apps like UserRegistration and the like. But as you outlined above, there are a lot of bits here and there that are altered regularly to model a site to your needs. The amount of modifications are not always as foreseeable as UserRegistration or the WikiUsers topic.

-- MichaelDaum - 11 Apr 2010

Thanks for the constructive comments, I will look further into them. Perhaps 'System' topics was wrong wording. 'Foswiki' topics is perhaps better to describe all topics that come distributed with Foswiki, not only those in the web named 'system'. However, this does not affect the problem or it's solution. Besides, I also changed some items in the system web as well (knowing at the time it would give trouble later on).

There is already this FAQ item to assist people with a bash script to do the three way merge. However, a three way merge on the files will indicate conflicts on almost all modified ones due to changed META data. A filter to ignore this type of conflicts might be a quick solution (I realise now).

Crawford also mentions a source-code control repository to track and merge changes. With Foswiki based on RCS, it is already there. At first sight the problem is the same as the problem of two users simultaneously editing the same topic. I thought it might be possible to solve the problem using this existing Foswiki functionality.

If you think something along these lines is feasible and useful to more people, I am willing to write something in perl. I have little time, so it may take quite a bit of time, but I guess it has no top priority anyway. If you think something like this is not feasible, I just leave it as it is and spend some time now and then. After all, the topics included with Foswiki don't change very often.

Meanwhile it helps to leave out files in the 'upgrade' package that were not changed from version 1.0."previous public release" to 1.0."latest release". But, I'm sure this last suggestion has also already been discussed.

-- ArnoldAtMallos - 11 Apr 2010

The FAQ item you mention is a bit more than a 3-way merge. You install a fresh copy of your current version of Foswiki into a temporary location. The script compares your current installation to the temporary copy. If there are no differences then it knows it can simply copy in the new file. The only time the 3-way merge gets invoked is when your local file is changed from the original distributed file. So changes to the metadata in the new file don't trigger a merge if the original and local installation files are identical.

It works quite well actually, and because the update packages distribute all files rather than just the delta from the prior update, you can upgrade from any prior version of 1.0 release.

-- GeorgeClark - 12 Apr 2010

If you really go to town with modifications, don't forget that you can change the name of the system web and the Main web in configure (%!SYSTEMWEB% and %!USERSWEB% respectivly) and then be free to update from the distributed version, or wholesale INCLUDE topics in.

One sad thing, is that in TWiki 4.0, we did ship a script (Which I spent quite alot of time on) that did the 3 way merge, and other little tests to make this simple. Unhappily it was not popular / used enough so few users reported issues, and eventually I abandoned it as unloved.

-- SvenDowideit - 12 Apr 2010

A late reply to the last comment: unfortunately, the values in %!SYSTEMWEB% and %!USERSWEB% are sometimes hard coded, so I'd better leave them alone... (just look at the use of Main in this topic )

The note at the DownloadFoswiki page

Migration from Foswiki 1.0 to any release of 1.1 should be done using a reinstall.

and my old install (still at 1.0.7) made me think about this whole issue again. This migration really is just an upgrade from one version to the next minor version... Yet, it basically tells me to start again, find out what customisation I've done over time and redo them all. This is IMHO not compliant with Foswiki mission (OurMission) that says:

• Easy to install, use, and administer

I've done some analysis and thinking. This is the result of it. Is this a feasible solution?

Currently Foswiki ships with the following data (between brackets the number of files):

• _empty [1]
• _default [13: 12 basic topics + WebCreateNewTopic]
• Trash [13: 12 basic topics + TrashAttachments]
• Sandbox [19: 13 same as _default + Plugin and Add-on examples]
• TWiki [13: same as _default]
• Main [31: 13 same as _default + 4 users, 3 groups + 4 UserList* + SitePreferences (!!!) + other]
• System [489: 13 same topics as found in _default]

This data comes in the distribution and is used in operation. However,

• Some files in the distribution have to be modified for local installations
• Local modifications have to be preserved during upgrades
  • _empty, _default, Trash and Sandbox have a special function, but are normal webs otherwise. Like normal webs, topics in these webs should never be overwritten during an upgrade, so they should not be in the distribution (please read on ).

Currently, Foswiki philosophy is to copy topics from the System web to the Main web for local, upgrade safe customisation. But, this philosophy is not maintained consequently and it is not documented (like: System. Default Preferences should be copied to Main. Site Preferences). If it is documented, then it is documented for individual topics, not as a general philosophy. This leads to a rather complicated situation that site-wide settings are taken from a place in Main, unless they are not defined in that place in which case they are taken from a place in System... Complicated, both for Foswiki programmers and for administrators!

The current content of System and Main lacks a clear separation of different kinds of content. Basically everything is dumped in the System web (and sometimes copied to Main). Analysing the content, the following data exists:

• Users and groups,
• User documentation (User tools (except change password / e-mail) are in individual webs)
  • basic - editing a topic with WYSIWYG
  • TML - basic editing
  • simple macros
  • advanced macros, though this might also be in the admin doc.
• Admin documentation (operation) and admin tools
• Site-wide preferences and customisations
• skins
• TWiki(tm) compatibility
• distribution data like Release notes, install guide, upgrade guide

These types of data can have different access rights:

• Read only for all: distribution data
• Read only + add (and edit own additions) for admins: skins, Admin doc, documentation on advanced macros
• Admin read/write: preferences and customisations, TWiki(tm)
• Admin and/or User read/write: User documentation (to make it specific for specific users/environments; localisation)
• User read/write: Users and Groups

Of course, read only for admins should be clearly documented, but I guess there's no need to enforce it by the system, so skins, admin doc, doc on adv. macros can be read/write by admin as well.

Combining the above I suggest the following webs.

• _distribution - containing release notes, etc. + the sub webs containing default content for each of the following webs + a sub web containing skin template topics. This web + sub webs may be removed after installation
• _empty
• _default
• Trash
• Sandbox
• TWiki
• Main - containing the default users and groups only
• System - including site wide skin template files,
• Userdoc - documentation on basic use, TML, simple macros
• Admindoc - admin tools and docs + advanced macro docs (search, etc.)

At first/fresh install the admin has to do the following:

1. Read the basic install guide (README or INSTALL file)
2. Unpack the new package at the desired location
3. Adjust OS user and access rights of the files
4. Read the fresh install documentation topics in _distribution
5. Copy subdirectories (sub webs) of data/_distribution/ to data/ except _distribution/skin that also has to be copied to data/System
6. Run configure script
7. Create admin user and normal user for him/her self
8. Adjust preferences in System
9. Adjust files in other webs, like WebPreferences
10. Create web for users
11. Invite users

At upgrade time an admin has to do the following:

1. Unpack the new package over the existing installation
2. Read the release notes and upgrade documentation in _distribution
3. Typically:
   1. copy _distribution/Admindoc over Admindoc
   2. check for user modifications and if none found copy _distribution/Userdoc over Userdoc (otherwise, do so respecting user modifications)
4. Check release notes/upgrade doc for
   1. modified behaviour of macros or depreciated macros
      • find their use on the site and take action accordingly
   2. modified behaviour of preferences or depreciated preferences
      • find their use on the site and take action accordingly
   3. modifications in skins and templates
      • find their use on the site and take action accordingly
      • (typically copy _distribution/skin over System)

Of course, steps 3.2 and 4 may be some work, but it is a straightforward procedure (better than 'figure out your local customisation and re-apply them').

As far as I can see, this approach can be used immediately. It affects only distribution. The new Main and System web are (from program point of view) compatible with current code. For this approach the following is needed:

• Disciplined tracking of changes in (behaviour of) macros, preferences, skins and templates by the Foswiki developers to get the detailed Release notes.
• New approach for distribution (the _distribution web and its sub webs)
• Splitting of the files in the current System web over the new sub webs (mainly the documentation)
• Update references (mainly in documentation) so it references the correct web
• Modifications to install and upgrade documentation
• Over time: remove code that first checks topics in Main for preferences before checking topics in System (removing a layer of complexity)

Can this be done, or do I miss some important aspects?

-- ArnoldAtMallos - 01 Jan 2012

I think part of the issue is that Foswiki is installed in many environments with very different requirements for file system access. The current goal of installation is to allow a user to unzip/untar the distribution file and run configure. No file editing / copying required. The need to go copying files prior to having a usable system is fairly limited. Hosted sites using .htaccess files have to tailor them, and sites with unusual path requirements need to edit bin/LocalLib.cfg. And sites with non-standard perl (and windows users) need to run rewrite-shebang. But in many cases the file system does not have to be touched at all.

I for one would object to needing to copy files. The current install "just works."

That said, there have been many proposals over time to try to address this from various angles.

• UpgradeSafeCustomisationsOfSystemTopics
• AutomagicalUpdating
• AddDefaultTopicParameterToINCLUDE (use of include for system topic customization)
• ShipCommonlyTailoredTopicsInConfigure
• (And we've talked about building an "upgrade" package and installing core updates using the Extension installer.)

For Foswiki.org and for several sites I've run, the script attached to Support.HowDoIUpgradeSafelyACustomizedFoswikiInstallation works quite well. The script has evolved over time and the latest version handles the merges quite well, giving you a choice of "use old, merge, or use new".

The reason for the "reinstall" is that we don't have a way to remove obsolete files with an upgrade. The move from 1.0 to 1.1 removed quite a few files that can cause issues if present in 1.1 . Also LocalSite.cfg was restructured and "upgrades" get into trouble.

And as far as the size of the "upgrade" vs. "full" package. I had considered for 1.1.4 generating a simple delta - only include topics modified between 1.1.3 and 1.1.4. But that would leave the user upgrading from 1.1.2 without a path. So we would have to generate a delta for each release ... and the user would have to apply multiple delta's. The approach we've chosen is to remove the "frequently tailored" files from the upgrade package, and include everything else. If you are interested, see Development.BuildingARelease

To address some of your other points above:

Copying files to Main not maintained, not documented

This is really an evolving process, and is being addressed on topic at a time. We could probably do this faster. WebStatistics was partially addressed in 1.1.4 - with more significant changes in 1.1.5.

Copying DefaultPreferences to SitePreferences

this is not the intention. Most of the settings in DefaultPreferences are seldom used. Just copy the individual settings you need to change to SitePreferences.

Complicated settings taken from multiple locations

Actually this hierarchy of settings is indeed a complex but very important part of Foswiki to understand. It's much more than just those two locations. DefaultPreferences -> SitePreferences -> UserPreferences -> WebPreferences -> SubwebPreferences inherit from Web) -> TopicPreferences (and for older Plugins, there are settings in the PluginTopic that also fit in the hierarchy somewhere. ) $ Over time remove code that checks Main for preferences) The preferences hierarchy is extremely fundamental to many Foswiki applications. It is probably something that we can't touch.

Disciplined tracking ...

Indeed. We are a volunteer project and trying to get discipline is like herding cats. And as both a developer and the 1.1.4 release manager, I was cursing myself at times. And it gets more complex because of the Trunk version (1.2 / 2.0) code. Plugin development is typically done on trunk and then blocks of changes are checked into 1.1 under a non-descript "Sync from trunk" commit. As release manager, I have considerable less hair after trying to decode all that.

Distribution web and subwebs

That was discussed in a prior proposal to populate missing "commonly tailored" topics in configure from distribution subwebs or other storage locations.