Guidelines for moving TWiki^® extensions to Foswiki

Migration from TWiki to Foswiki

Foswiki and TWiki share a common heritage, and with the help of the TWikiCompatibilityPlugin, Foswiki is able to run many TWiki extensions without modifications. However, there are a number of reasons you should consider porting an extension to Foswiki:

1. Efficiency. TWiki extensions require a compatibility layer that sucks some performance.
2. Tools. Foswiki has a richer development environment than TWiki. Extension development is easier, and is getting easier all the time.
3. Automatic back-porting. Extensions built with the BuildContrib can take advantage of the twiki target to automatically back-port some extensions to TWiki. This significantly simplifies supporting extensions that must work in both systems.
4. Opportunity. Foswiki offers richer APIs that can often help you simplify your code.

Porting an extension to Foswiki isn't difficult, especially if you have used the published APIs. Foswiki has 1:1 equivalents for every function in the TWiki::Func API, and there are equivalents for most of the CSS and Javascript symbols you might have used.

The rest of this guide assumes you are porting a TWiki extension that already exists in the Foswiki repository. If you are porting a new extension the process is much the same, just ignore the subversion specifics.

The first thing to do before porting an extension is to check its status in StatusOfPortingExtensionsToFoswiki. If the author has indicated that they want to port it themselves, contact them before doing anything. If the extension is marked "help wanted" you can just go ahead.

The next thing to do is to verify that the source in the Foswiki repository matches the latest version from its download page on twiki.org. Be aware that in a few cases, a more recent version of the extension has been uploaded to twiki.org than exists in subversion. Before any other changes are made, note the release of the extension (or the TWiki subversion revision number) that corresponds to this latest version, and the corresponding Foswiki Subversion revision. Add a note to the Copyright file indicating both of these revisions. For example: Foswiki Subversion revision 299 contains the source for extension MyWonderfulPlugin taken from TWiki Subversion revision 4411.

Now you are ready to convert the extension to the Foswiki namespace. Use the excellent tools/twiki2foswiki Bash script to do this.

Here's what the script does, just in case anything goes wrong, or you want to do the steps manually!

• Find/Replace TWiki::Plugins with Foswiki::Plugins (eg, package TWiki::Plugins::FooBarPlugin;)
• Find/Replace $TWiki::Plugins with $Foswiki::Plugins (eg, TWiki::Plugins::VERSION)
• Find/Replace TWiki::Func with Foswiki::Func
• Find/Replace $TWiki::cfg variables with $Foswiki::cfg
• Rewrite all references to the TWiki web to System web, and TWiki lib to Foswiki
  • svn mv lib/TWiki lib/Foswiki
  • svn mv pub/TWiki pub/System
  • svn mv data/TWiki data/System
• Replace BaseUserMapping_### references with ProjectContributor
• Update MANIFEST files (if present) with new paths
• Update build.pl (if present) replacing TWiki with Foswiki

Once the automated steps are complete, there are a few manual steps:

• Review the code and find any illegal API calls. Replace them with the published API, or, if no published API exists, raise a feature request to get one added.
• Update your documentation according to the TWikiTerminologyMap.
• Rewrite all references to renamed topics listed in TopicNameMappingTable (eg, TWikiPreferences to SitePreferences)
• If you have any bin scripts:
  1. Change them to rest-handlers
  2. If that's not possible, things get a lot trickier. Contact CrawfordCurrie or GilmarSantosJr for advice.
• Edit the plugin topic, add the following form metadata:

%META:FORM{name="PackageForm"}% %META:FIELD{name="TopicClassification" attributes="" title="TopicClassification" value="PluginPackage"}% %META:FIELD{name="Compatibility" attributes="" title="[[Compatibility]]" value=""}% %META:FIELD{name="DemoUrl" attributes="" title="DemoUrl" value="http://"}% %META:FIELD{name="DevelopedInSVN" attributes="" title="DevelopedInSVN" value="Yes"}% %META:FIELD{name="ModificationPolicy" attributes="" title="ModificationPolicy" value="PleaseFeelFreeToModify"}% 

• Port and run any unit tests
• Add additional steps to this list!

If the extension wasn't built using the BuildContrib, then now is a good time to add the files it requires. Usually all that will be required are the MANIFEST, DEPENDENCIES and build.pl files, and some restructuring of the extension topic. See BuildContrib for more information on these files.

Now you should:

1. Verify that the extension works with the latest Foswiki version.
2. Verify that the extension has a maintainer who remains active; otherwise, mark the extension as unmaintained.
3. Verify that the extension has no security vulnerabilities, based on the SecurityVulnerabilityGuidelines and other known best practices of the Foswiki team.

Though these work items may feel like extra overhead, they will help ensure that all Foswiki extensions meet a baseline of usability and quality once they have been flagged as being migrated to Foswiki.

Suggestions

From the extension directory, some changes can be done in bulk by using sed such as:

• find . -iname "*.p[lm]" -exec sed -i~ -e "s/TWiki::/Foswiki::/g" {} \;

This will change all "TWiki::Plugins", "Func" and "cfg" at once, in the perl modules, and also build.pl You can have more than one -exec statement in find, just remember the \; to terminate each exec. (e.g. to move out backup files).

Bear in mind that the Foswiki::Func API is richer (has more functions, and more powerful functions) than TWiki::Func. It may pay dividends to review places where you had to "code through a keyhole" because TWiki didn't support what you wanted to do.

Back-porting to TWiki

If you want to support an extension in both Foswiki and TWiki, then you can use the BuildContrib to support semi-automatic porting of your Foswiki extension back to the TWiki namespace. Because Foswiki is richer and more powerful than TWiki it won't be possible to back-port all extensions, but many of the simpler extensions (and some of the more complex ones as well) can be ported this way, as long as they have stuck to the subset of the APIs that TWiki supports.

For example, to auto-port the GetYourPlugin to TWiki:

cd GetYourPlugin/lib/Foswiki/Plugins/GetYourPlugin
perl build.pl twiki

this will generate a parallel directory structure that contains sources that have been mapped to the TWiki namespace. It's not as powerful as twiki2foswiki though, because it doesn't touch documentation, CSS or Javascript files.

Once you have your parallel TWiki sources, you can build a TWiki release:

cd ../../TWiki/Plugins/GetYourPlugin
perl build.pl release

Do not check the TWiki sources in to the Foswiki repository unless you have no choice (for example, you are using JS or CSS that isn't mapped).

"TWiki" is a registered trademark of Peter Thoeny

---

Discussion

Proposal for restructured SVN

(Not yet accepted.)

The following procedure is proposed (updates, comments, and further feedback are welcome — hey, it's a wiki!):

1. All extensions are moved ( svn mv) into a Legacy folder.
   • A folder is chosen over a branch because most developers prefer this, and it avoids any duplication.
2. One by one, migrate each extension to Foswiki. This includes code modifications, closing any known security holes, testing in the latest Foswiki release, and identification of a maintainer.
3. In order to help co-ordinate changes, verify that modifications are maintainable, and ensure that the changes fit in with the current architecture as well as future plans, please co-ordinate your changes with the maintainer of the extension being modified.
   • Feel free to bring any discussion about your changes to a wider audience on the foswiki-discuss mailing list or the #foswiki IRC channel. Also use these avenues for guidance if the maintainer does not reply to your messages.
   • By providing guidance and consulting support for all changes being made to the extension, the maintainer can help ensure that the extension's implementation remains cohesive and able to meet current and future requirements.
   • If you are unable to reach the maintainer, or he/she no longer wishes to be the maintainer, then update the maintainer of the extension to None.
4. Once the migration is complete, move the extension from the Legacy folder into trunk/FoswikiExtension
   • If the maintainer has responded to your messages within a week, then you must co-ordinate the move with the maintainer.
5. When this process is completed (code is moved to trunk/ (see below), tests are successful), upload the code to foswiki.org using the BuildContrib tools
6. Extensions that are known to be useless (no longer needed, merged into core, plugins for dead technology, etc...) should be moved to an Obsolete branch.

N.B: Legacy is a place holder. The actual path / Subversion implementation should be discussed internally. Possibilities include branches/Legacy (suggested by OlivierRaginel).

• It has been said this is a bad and probably unnecessary thing, thus I'm moving it apart. But I still think we need some better taxonomy on extensions, as this confuses the hell out of me. Also, I really dislike having everything directly under trunk, thus a AddOn, Plugins, Extensions, WhateverYouCallIt folder would make my day, even if it has all in it, and we're left with just it and core in trunk.

1. Extra: (this is a personal wish) split up extensions by directory, and rename them (eg at random):
   • AutoCompleteAddOn => trunk/AddOn/AutoComplete
   • AutoCompletePlugin => trunk/Plugin/AutoComplete -- Check also why there are a add-on and a plugin, and maybe update one's topic
   • BatchUploadPlugin => trunk/Plugin/BatchUpload
   • BehaviourContrib => trunk/Contrib/Behaviour
   • DallasSkin => trunk/Skin/Dallas
   • ChildTopicsTag => trunk/Plugin/ChildTopics ???
   • ...
   • I would like to re-organize these, and I personally hate to have everything in trunk, but I can live with it

-- OlivierRaginel - 08 Dec 2008 - 15:15 % C O M M E N T %

Comments on Restructuring Proposal

---

I really advice against changing the overall structure of trunk.

pseudo-install and build contrib relies on the structure and spending time on this takes time away from the release.

Moving extensions to legacy that are not yet moved to Foswiki space is not a good idea either. Only move those that Crawford and Michael have asked to be spared.

Also notice that I have already transferred some of the non default to Foswiki space. The ones I heavily depend on myself. If they are moved my pseudo-install stops working. I do not have time to fight with that so please be careful what you do.

The method above for transferring a plugin is not wrong but it can be 99% scripted. Only the .txt file needs to be manually converted. I already did 90% of the doc work but I have stopped now that I cannot do it efficiently with global search/replaces.

There is also something wrong with the statements around policies. From the behaviour of a few developers Plugins are indeed owned and not at all community property.

My personal view on plugins has always been

• I fix bugs found in any plugin without asking if I know the fix. Or if an old bug has not been fixed by the author for a long time.
• I fix security issues any time.
• I do not change plugins with respect to changing their behaviour or doing enhancements without the consent of the author.

These 3 rules are also what Crawford has confirmed to follow.

With the Foswiki work I assumed we all agreed that we forked everything. I am sure this was the agreement in the beginning. With 200 SVN checkins and noone saying anything it cannot have been unnoticed. Then things changed.

BUT many of the extensions have developers that did not follow us to the fork. I assume that these are free territory. Even if I wanted to contact them I cannot because we do not know their email addresses.

With all the mail exchanges on the mailing lists we must assume that those plugin authors that have a problem with the rebranding have spoken up now or will within the next few days. Or else please speak up as you see this. But don't worry about me. I am no longer doing bulk rebranding. I cannot because I cannot practically steer clear of the non-free (not free as in in beer, but free to modify) extensions.

If you have a need to brand name change an extensions pop me an email (kenneth@lavrsen.dk) and I will help you.

-- KennethLavrsen - 08 Dec 2008 - 21:42