Version and Release Numbering (revisited)

This topic is about version and numbering of Foswiki releases and extensions, which is changing as of Foswiki versions 1.2 and 1.1.6.

This document defines the terms "Version" and "Release" as follows:

• VERSION: A formal Perl approved version number, dotted triplet: v1.2.3, or decimal: 2.3. Version numbers may be automated by the Foswiki build tools. It's possible to compare two version strings and definitively know which represents the newer version.
• RELEASE: A "human friendly" description of the release. Two release strings "might" be comparable, but there is no guarantee.

Version Numbers as of Foswiki 1.1.6 and 1.2

Foswiki has chosen to adopt the module version numbering used by Perl. See also CPAN:version

• Foswiki Releases will use "Dotted triplet" versions: v1.1.6, v1.2.0.
• Extensions can choose between simple decimal versions, 1.1, 2.3, or dotted triplets v1.2.1, v2.3.1.

The change for Foswiki version numbering is subtle - the "v" prefix is required to be a valid Perl version. The Extension version string has undergone a substantial change.

Alpha, Beta and RC (Release Candidate) versions

Perl version strings support an "alpha suffix" _nnn, which is written as v1.1_6 for a dotted triplet, or 1.14_1 for decimal versions. Written with the same components, the "_" alpha version will compare as newer than the same "." released version. So 1.1.6 will compare as newer than 1.1_6

The CPAN:version module states this as:

Note that "alpha" version objects (where the version string contains a trailing underscore segment) compare as less than the equivalent version without an underscore

A sequence of releases might look like:

Decimal	Dotted Triplets	Notes
1.1	v1.1.5	Starting release
1.1_1	v1.1.5_001	Test release (alpha or beta)
1.1_2	v1.1.5_002	Next test release (alpha or beta)
1.12	v1.1.6	released for production
1.1_3	v1.1.999_001	Next minor test release (alpha or beta)
1.13	v1.2.0	released for production

Developer rules of thumb

• Always "Declare" dotted triples using the following on a single line:

use version 0.77; our $VERSION = version->declare('v1.2.3_001');

• Decimal versions should be set with an assignment and should be quoted

our $VERSION = '1.2';  #Recommended

• When using "alpha" designations with Decimal versions, use the following on a single line.

use version 0.77; our $VERSION = version->parse('1.20_001');

• Be cautious when incrementing a decimal version. The results might be surprising, especially if comparisons are made between decimal and dotted triple versions are made.
• BuildContrib will refuse to build an extension that uses the $Rev or $Date keywords in the VERSION string.
• Add PatchFoswikiContrib as a dependency of any extension that uses the new version string format. See section on Backwards Compatibility.

Why use version->parse() and version->declare()

These forms create a "version object" and is able to correctly interpret and compare versions. Using a simple quoted string without declaring it as a version object:

• Can result in warnings or errors due to invalid numeric comparisons.
• Can result in unexpected results.

Converting between version formats

The $VERSION values are used to determine that required dependencies are installed, and whether or not an extension needs to be updated. It is critical to ensure that a new version always compares as newer than the prior version.

The following shows some conversions from decimal to normalized dotted

perl -Mversion -e 'print version->parse("4.44")->normal'   returns v4.440.0
perl -Mversion -e 'print version->parse("1.02")->normal'   returns v1.20.0
perl -Mversion -e 'print version->parse("1.9")->normal'    returns v1.900.0
perl -Mversion -e 'print version->parse("1.10")->normal'   returns v1.100.0  

Release strings as of Foswiki 1.1.6 and 1.2

$RELEASE will continue to be a "friendly human readable" version.

• Foswiki releases will use Foswiki 1.1.6 or for a beta, Foswiki 1.1.6_002-Beta1
• Extensions can choose any of 5 formats for $RELEASE,
  • A simple integer number (SVN revision string) Deprecated
  • dd mm yyyy formatted date
  • ISO yyyy-mm-dd formatted date
  • Decimal (quoted) '1.1' or alpha '1.1_004'
  • Dotted Triplet (quoted, v prefix optional) '1.2.4' or alpha '1.2.4_004'

Backwards Compatibility

It is critical to install the patch for Tasks.Item12225 for any version older than Foswiki 1.1.6.

• Users should install PatchFoswikiContrib. It will patch Foswiki::Configure::Dependency.pm and Foswiki::Configure::UIs::EXTENSIONS.pm
• Developers should add PatchFoswikiContrib as a dependency on any extension released with the new version string

ONLYIF ( $Foswiki::Plugins::VERSION < 2.3)
Foswiki::Contrib::PatchFoswikiContrib, >=1.0, perl, Required for old Foswiki versions

There are two issues with older versions of Foswiki resolved by this patch.

1. The first is confined to the Configure "Find Extensions" interface. Configure assigns the version of "HEAD" to the version for pseudo-installed extensions. This is not legal for comparison to true version objects. The patch changes this to use version '9999.99_999' to represent a pseudo-installed extension. Even though a normal production install will not use pseudo-install, the Extensions Installer compares for HEAD and crashes configure.
2. The 2nd is also related to extensions. Some older extensions might write a dependency on a specific SVN Revision of a topic. When that extension is installed, the comparison with the installed version string fails, and the dependency is re-installed. The patch causes dependencies written as SVN revisions to be satisfied by any extension using a decimal $VERSION.

If an extension using the new version strings is installed before the system is patched, PatchFoswikiContrib can be installed from the command line using perl PatchFoswikiContrib_installler , or by manually patching lib/Foswiki/Configure/UIs/EXTENSIONS.pm

Numbering prior to Foswiki 1.1.6 & 1.2

Foswiki supported 3 different release numbering schemes

• Core Foswiki releases were numbered as a "dotted triplet" (major).(minor).(patch). Version 1.0.10, 1.1.5, 1.2.0
• Foswiki extensions use two methods
  • $RELEASE is a relatively free-form human readable version, with any of 5 layouts. Dates, decimal versions, dotted triplet, etc.
  • $VERSIONis a machine generated string tied to the Subversion repository rev number and date of last file modification in the extension.
    • Example: 14671 (2012-04-27) Extension was built from repository at rev 14671, with last file changed date of 2012-04-27

With the impending conversion from Subversion (SVN) to git as the code repository, we need to move away from Subversion rev numbers. Also, none of these schemes were completely compatible with Perl's concept of version numbers.

Previous document on VERSION and RELEASE

This topic is about release numbering as it applies to extensions. For information about release numbering of Foswiki releases, see another topic.

It's clear from recent proposals on the foswiki-discuss mailing list that there is a lot of misunderstanding about the release numbering automation, even among developers. This topic is intended to clarify the current situation as at 3 Mar 2009, and ultimately be refactored into developer documentation.

First I will reiterate the point that only a small subset (20 out of 317) extensions are bundled in a release package. That makes them the exception rather than the rule, and problems with bundling them should not impact the other 297. I'll comment on Foswiki release builds later, but first let's first review how the versioning scheme is designed to work.

BuildContrib supports %$VERSION% and %$RELEASE% for a reason. Everyone has their own ideas about how to number versions, but the configure automation requires standardisation in order to work. %$RELEASE% is the manually-controlled, human, tired and emotional release number. %$VERSION% is the cold, mechanical, standardised version number. The idea is that automation handles %$VERSION% in a way that is as hard as possible for developers to f*** up. If you could guarantee making all contributors equally cold and mechanical, then we could dispense with %$VERSION% and use %$RELEASE% exclusively. I gave up trying to do that fairly early on.

OK, that's the theory, so how about the practice? There are three key points in the development and release cycle we have to consider: cause SVN is a bit stupid, and doesn't provide any way to identify a related set of sources that we want to find a common "most recent version" number for. So our $VERSION in the master perl module is the version of that file, and not the version of the whole extension_.

To work around this stupidity, BuildContrib uses SVN commands to enumerate the contents of MANIFEST and find the highest revision number of any of the sources (see Foswiki::Contrib::Build::_get_svn_version). When an extension is built, the %$VERSION% for the whole extension is calculated using this method. Therefore when the Extension.txt file is created during perl build.pl release, the %$VERSION% expanded in the .txt file is the highest SVN revision number of any file listed in MANIFEST, and the build time.

So, on to configure. The page that looks up extensions on Foswiki.org, Extensions.FastReport, reports the =%$VERSION%====== of the extension. At the same time, the version number of the locally installed code is looked up using =$Foswiki::Extension::RELEASE || $Foswiki::Extension::VERSION from the master perl module (see =Foswiki::Configure::UIs::Extensions::_getInstalledVersion). Obviously there's a huge disconnect here, because the =$RELEASE is manually generated, freeform, and unusable for mechanical comparisons, and as noted above the =$VERSION is the version of the file, not the extension (note that this disconnect has been made even worse by distro:309277b130ef, which has dropped the display of =$RELEASE).

Clearly we want configure to report the %$VERSION% of the extension as the "installed version". I didn't do this previously because it isn't as easy as you might think to look up an extension topic from configure and I had other priorities (and no help). Also %$VERSION% is unexpanded in a pseudo-install, but a simple bit of code could improve on this. See Tasks.Item4480, which has been open for many moons.

OK, so how about Foswiki builds? Foswiki is built using mostly the same code as extensions, so the same process applies. Kenneth asserts:

> Each time I build a new Foswiki release the plugin topics appear as
> updated even if they are not. Their plugin .pm file and their .txt file
> are changed just in one little detail - the SVN number.

I have to confess I haven't been tracking the changes in the tools/build.pl script, but I find this hard to believe. The extensions built into the release package are recursively built using BuildContrib, so the %$VERSION% expansion in the included packages will be built the same way. If a single file has changed in the extension, this will result in a change to the generated %$VERSION% for the extension - as it should; something has changed. If nothing has changed, then the %$VERSION% will be unchanged from when that extension was last built. If this is not the case, then it is evidence of a bug, not a reason to change the way version numbering works.

Relevant tasks: Tasks.Item1055 Tasks.Item4480 Tasks.Item604 Tasks.Item924

-- CrawfordCurrie - 03 Mar 2009

The most recent changes to configure were done to improve the UX on the Download Extensions page. The numbers on that page (installed version and latest version on server) need to be comparable. The sole use of "upgrade" buttons in the extensions table is to upgrade to the latest version. For that the numbering scheme must be the same. And to add that bit of 'intelligence' (you would expect) that if both versions are the same you don't see an upgrade button.

As (almost?) none of the extensions uses %$RELEASE% in the doc topic, so until that is done I need to use %$VERSION%.

But I would recommend that all extension writers add %$RELEASE% so that it can be read by FastReport and displayed in the Download Extensions page.

-- ArthurClemens - 03 Mar 2009

Reporting %$RELEASE% is fine, but is not a substitute for proper handling of %$VERSION%, which must drive the automation. Your change is OK, but incomplete; the code needs to be fixed as well, as described above.

-- CrawfordCurrie - 03 Mar 2009

As of today the configure is totally broken because of recent changes to release and the release versions have been mess since some of you started automating the revisions abusing SVN revisions.

The current way is confusing and inviting for error because you have to understand a lot of geek shit to make it work.

The way forward is to go back to what worked well.

• Manually updated revision number in X.Y.Z format in the .pm file
• Manually updated revision number in X.Y.X format in the .txt file

And with a date also.

The problem with the SVN number is that

• They contains no information about if the release is a major, minor, or little patch
• They say nothing about when so I cannot as a human tell if last update was recent or a long time ago before I last updated. This means that I have to check and compare each time with my installation. When I surf the Extensions web and I know I updated plugins in January 2008 I can easily as a human see if the different plugins are new.
• SVN numbers from TWiki are still in the histories so the sequence is broken now making the confusion total

-- KennethLavrsen - 04 Mar 2009

What's even worse for me with SVN revisions (apart the fact they're SVN specific), is that you can't use the ->VERSION method on a module, because it will throw (as of perl 5.9):

Version string '$Rev: 2242 (30 Jan 2009) $' contains invalid data; ignoring: '$Rev: 2242 (30 Jan 2009) $'

Therefore, if we want to stay close to being perl modules, and thus one day eventually publish some into CPAN, we have to use a standard naming scheme (major.minor[.revision])

Not to mention using the standard way to get the version of a module, and not some hack to use $Module::Name::VERSION;

A small piece of code to demonstrate what should be done, and what shouldn't, and why:

$ perl -Mstrict -wle 'package A; our $VERSION = 1.2; package B; our $VERSION = "1.2.3"; package C; our $VERSION = "\$Rev: 243\$"; package Test; print "$_: ".$_->VERSION for qw/A B C/'
A: 1.2
B: 1.2.3
Version string '$Rev: 243$' contains invalid data; ignoring: '$Rev: 243$' at -e line 1.
C: 0

-- OlivierRaginel - 04 Mar 2009

I'm still not getting my message through, even assuming the above was written before the IRC discussion this morning.

There are four sources of "version" information; one is manually supplied, and the other three are generated.

1. The $VERSION variable in the .pm file. This is normally automatically expanded when a module is checked in. For the purposes of configure this must be ignored, though as Olivier advises it may make sense to make it "perl compatible" for future migration to CPAN.
2. The $RELEASE string in the .pm file. This is a freeform string provided by the developer and cannot ever be trusted
3. %$VERSION%, which is generated in the .txt file when an extension is built by examining the versions of all the source files in the extension. This is the only sensible option for configure to use to determine whether an extension should be updated. It consists of the SVN version number, and (soon) the date of the most recent checkin.

(1) must be ignored; it is all but useless, (2) may be useful to present a "user friendly" version in configure though I strongly suspect it will be a source of confusion and error, (3) is the only thing we can trust for automation.

-- CrawfordCurrie - 04 Mar 2009

You do not seem to listen to our arguments either Crawford.

I do not want your automated SVN numbers because they lack major,minor,patch information. The few plugins where you only used these SVN numbers have been awful to look at when you need to decide if it is time to upgrade.

I want to use X.Y.Z versions in the plugins AND a date. And the version should be decided on by a HUMAN. And the date should be the date of the change of the version number.

And I want the date to be the date I changed the plugin. Not the day it was built. I want to be able to rebuild the same plugin later and get the same revision out of it. This is essential in the default plugins and equally important in any other plugin.

I cannot be such a big deal for you to update the revision. I think we are sacrifizing the voice of the customer just to make a few lazy developers happy. Most extension developers have been updating revision number nicely for years.

-- KennethLavrsen - 04 Mar 2009

Summary of above discussion. Added my flavor of must have and nice to have.

Our goal: making comparing extensions easy:

What we want to achieve:

1. MUST HAVE: Provide advice (in configure) when to upgrade an extension
2. MUST HAVE: A short term solution for release 1.0.4
3. NICE TO HAVE: Have a way to compare extension versions with svn revision numbers
4. NICE TO HAVE: Make version numbers 'perl compatible' for future migration to CPAN

How are we getting there:

1. Provide advice (in configure) when to upgrade an extension
   • Compare installed version with repository version: use the same numbers or strings
     • The version should be human generated, preferably major.minor[.revision]
     • The version should have a release date (human generated) to judge the recentness of a release
   • The only way to have a faultless automated process is by using %$VERSION%, which is generated in the .txt file when an extension is built. It displays the svn revision number
     • But a revision number will not say much to the end user. The revision number may be hidden though, and used under the hood only. But this may cause confusion if the human readable strings say "equal", while the revision numbers differ (suggesting "upgrade")
     • If we do use the revision number, we should use %$VERSION% of the extension as the "installed version", not our $VERSION that is file dependent
     • How much value adds a faultless automated process to a human generated process? If it is about comparing versions, we only need to compare the same strings
       • Developers may forget to update the version number and/or date.
         • We don't have that at this time
         • But we can add build support to facilitate this (or even require version updates) (Nice to Have)
2. A short term solution for release 1.0.4
   • Or we must re-build hundreds of extensions now
   • The repository information must be gathered by FastReport, so the information must be present in the extension topics
   • The only string available in the current extension topics is "Plugin Version", with a human readable string
3. Have a way to compare extension versions with svn revision numbers, in the interests of sane and literate development
   • We should use %$VERSION% of the extension as the "installed version", not our $VERSION that is file dependent: %$VERSION% expanded in the .txt file is the highest SVN revision number of any file listed in MANIFEST, and the build time
   • This requires developers to write %$VERSION% in their extension topic (most don't have that)
   • pseudo-install needs to be improved to expand that macro
4. Make version numbers 'perl compatible' for future migration to CPAN
   • The current version string is invalid when calling ->VERSION on it
   • We have to use a standard naming scheme (major.minor[.revision])
   • Example: our $VERSION = "1.2.3"

If our main goal is to be able to compare extensions, we are suffering from a lack of data to use. The extension topics simply do not carry enough information to compare extensions objectively.

These are the short term options:

1. Do not compare objectively, go back to the old comparison and let the user decide between 1405 (08 Jan 2009) (16 Dec 2008) and 03 Aug 2008 (1340)
2. Create a script to populate FastReport to carry the necessary information.
   • The script should read the .pm files and generate a comparable string that contains information from both %$VERSION% and =$RELEASE
   • The script should be used by Configure as well to get to the same information
3. ...

-- ArthurClemens - 04 Mar 2009

Arthur, you are mostly right in what you say. However I think we are all making this much more complicated than it needs to be. There is one important point that seems to be consistently missed (and I have not made clear enough):

1. BuildContrib computes %$VERSION%, but it is up to the extension developer whether they use it in the extension topic or not. I choose to write:

Version:

%$VERSION%

to take advantage of the automation. Another author may choose to write

Version:

1.2.3

and it will be equally valid. The magic is in the | Version: | which configure and FastReport use to scrape the topic. We agree that the main problem we have is a result of comparing apples with oranges. We agree that we need to scrape the Version string from the plugin topic for presentation in configure. The only problem, then, is deciding how to use this version identifier in a comparison. Well, it's hardly rocket science to make configure recognise both string formats, i.e. 1234 (3 May 2009) and 1.2.3, and compare appropriately. So, here's what I propose, in easy to swallow steps:

1. Fix configure to scrape the version from the locally installed topic (as we discussed on IRC)
   • At this point we have a short term fix. No other steps are required - yet
2. Make configure recognise the alternative 1.2.3 version format
3. Fix BuildContrib to use the SVN date in %$VERSION% rather than the build date
   • This only affects extensions that use %$VERSION%, and is not urgent

-- CrawfordCurrie - 04 Mar 2009

For me, there is also something else which got forgotten. We used to have 2 fields : %$VERSION% and %$RELEASE%.

For me, we should use one for automation ( %$VERSION% ) and the other one for display ( %$RELEASE% )

This way, we can make %$VERSION% perl compatible, and it should be incremented manually by the developer, and keep %$RELEASE% for display in configure, so that user can decide whether to upgrade or not. %$RELEASE% could be based on %$VERSION%, also the latest Subversion revision (and its date), taken out from the last modification done on the files within the MANIFEST of the plugin, just like it is now, to help the developer.

I do think we should standardise both, in order to avoid confusing our users, but as I'm no extension maintainer, I can't grasp how hard it might be.

We could also patch BuildContrib to deny uploading a new version without up'ing the version number, and prohibit uploading extensions without using BuildContrib.

Crawford, you made a very good job creating a framework to ease the developer's maintenance tasks, it's fair to have them use it, unless someone highly disagrees.

I think the politic used to be that we'd better have a mess than miss a contribution, and I'm all in favor of changing that. If people want to contribute, but don't want to use BuildContrib, then they should either ask for help, or give away the maintenance.

For our end users, it's far more important to have high quality in the extensions that they can install, and thus have a standard version scheme, than to have as many extensions as possible.

Just my 2 cents.

-- OlivierRaginel - 05 Mar 2009

OK, here's what I have done:

1. configure now scrapes the version number from the topic, and is tolerant of version expressed as SVN revision numbers, triples (1.2.3), ISO dates and dd Mmmm yyyy format dates.
2. BuildContrib now computes the SVN rev number and date correctly
3. BuildContrib now supports %$RELEASE% for expansion of $RELEASE in extension topics (this is the route for 1.2.3 to get in)
4. Fixed the extension topics that had grown extra dates, due to a misunderstanding of how it all works

-- CrawfordCurrie - 05 Mar 2009

I was pointed today to the perl "guidelines" for plugins: http://search.cpan.org/dist/version/lib/version.pod

We could use version.pm to generate proper perl versions out of Subversion keyword (but it was written for CVS, so it might need some "tuning" for Subversion).

Thanks Crawford for your work on this. It's the best we could do for now, and I think it's heading in the right direction.

-- OlivierRaginel - 11 Mar 2009