Feature Proposal: Cast in stone the topic revision numbering scheme used by the RCS-based stores

Motivation

All users are used to the existing 1..N sequential topic revision numbers. It is easy to understand.

Description and Documentation

All stores, regardless of the underlying implementation, must identify revisions using revision numbers that work like the RCS-based stores.

The revision numbers are per-topic. Different topics' revision numbers are independent.

The first revision of a topic is revision 1. Revision numbers are sequential and increasing.

Stores may use a different scheme for identifying revisions internally (e.g. timestamp, md5sum, increasing but non-sequential integers, etc), but the store interface (Foswiki::Store) must obey this constraint.

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

The following was relocated here from the below discussion, and was assembled from many comments and some IRC discussion.

Is there a common list of objectives we can agree upon?

1. User Presentation of revision info on an individual topic should remain a simple monotonic number starting at 1.
2. User Access to an individual topic should also be by simple monotonic number.
3. How a store actually implements the revisions can be left flexible as long as 1 and 2 are satisfied.
4. User Presentation of attachment revisions should also be preserved. (ie. view rev's of an extension zipfile)
5. Attachment history should follow the attachment if moved between topics / webs,. renames, etc.
6. It is highly desirable that we also capture the state of a topic in the context of the whole store, so that:
   • Search results reflect the returned results for a point in time.
   • Attached graphics render as show at that point in time.
   • INCLUDEs show the results at that point in time.
7. Accomplishing #6 might require some other revision scheme. Ie. a "point in time" is not necessarily the time a topic was saved.
   • eg. What were the release blockers on 9/15/2010 at 00:00 gmt
8. It should be made a requirement to track back to a topic revision + associated attachments given just a revision number or date. -- CrawfordCurrie
9. Migrating to a new Store should not cause a reset of existing revisions. A static link to SomeTopic?rev=5 should continue to return the same results.

-- GeorgeClark - 09 Dec 2010

-- Contributors: MichaelTempest - 09 Dec 2010

Discussion

I pointed out on QueryAcrossTopicRevisions that the syntax of revision numbers wasn't documented. That prompted a distro:494a1c1dba0c that, as far as I understand it, documented the revision numbers, insofar as the store is concerned. The RCS-based store's revision numbers comply with that "spec", but the RCS-based implementation constrains the revision numbers to start at 1 (for each topic) and to be sequential. There was a distro:2bc9d36f3a03 that changed the documentation to correspond to what the RCS-based store does. Heated statements about this issue have been made on IRC.

I am raising this feature proposal because (1) as far as I can tell, the revision numbering scheme was not actually documented, and (2) as far as I can tell, there is not unanimous agreement. We have use the RCS-based store's numbering because that has been the dominant store. It happens to be a user-friendly scheme, too. However, I don't think it has (yet) been cast in stone. This proposal is in tended to bring the issue out into the open where we can discuss it based on its merits, or lack thereof.

I have put my name down as committed developer, because it involves no implementation and no documentation, and because that allows me to set the date of commitment.

-- MichaelTempest - 09 Dec 2010

Note that many back-end stores will find it difficult (expensive) to map their internal non-sequential revisions to a linear, 1-based revision scheme. Note also that the use of revision trails that are unique per-topic makes changesets difficult to express. These mappings can, of course, be done, but at a cost.

-- CrawfordCurrie - 09 Dec 2010

I compiled a list of how other wikis do it, when it comes to linking to specific page versions. It's worth noting that only half the platforms (briefly) studied use sequential 1..N version numbers.

It sounds like each topic in a git, svn, etc. back-end would need to maintain a database of which GUID/revnos belong to it. We also assume topic histories would be linear.

-- PaulHarvey - 09 Dec 2010

If a 1...N rev number scheme will become a major headache for non-rcs stores, i.e. a git one, then I have serious concerns that this is a good idea, no matter what this means for the user interface (for instance as produced by HistoryPlugin). Added myself to ConcernRaisedBy until someone can at least outline what the consequences are for a git based store.

On the flip side: when the numbering scheme can basically be everything whatever the VCS underneath supports, well then we need a way to traverse revisions to at least do this: (a) show the prev rev given a current one (b) show the next rev (c) show first rev (d) show last rev. On a 1..N rev number scheme this is more of a numerical exercise: (a) is currentRev-1, (b) is currentRev+1, (c) is 1, (d) is N. As things are this is bound to break sooner or later so these need to be replaced with some api that abstracts the underneath maths...which we don't have either, afaik.

The other problem is the requiremtent to have per-topic rev numbers. That's bound to hurt for a non-rcs store as well, where changesets might cover changes in multiple places: e.g. topic+attachments, fixing backlinks, adding a web. These three operations can't be rolled back right now because we have a per-topic and per-attachment versioning. That's not a feature we should enforce.

-- MichaelDaum - 09 Dec 2010

If someone wants to create an extension that uses roman numbers in a none sequencial list, I do not care. Most sain people will ignore it as another academic study not worth wasting oxygen on.

The essential is that the shipped core/default storage uses a humanly friendly revision number scheme.

We have this today. We are not here to decide if we want to change how RCS works. Foswiki has a 1,2,3,4... revision scheme.

The thing I reverted was the change of documentation Crawford did where he added a theoretical possible geek-numbering scheme in documentation our end users read. That documentation change only added confusion and did no good to any end user. If we want to write notes for developers of alternative storage schemes in the comments of the code, go ahead. But the documentation meant for the end users must be clean.

This world is full of geek idiots that do not understand normal users. This world is full of people with Asberges Syndrome that are not aware of or care about their handicap and live in their distorted world where only math, geometrical shapes, and C-code makes sense. This world is full of useless shit software with badly implemented user interfaces. Luckily we also have companies like Apple that have accepted that computers are supposed to be used by normal people that are not computer programmers. And where user interfaces are designed by people that understands normal people. Finding bad examples of poor user interface does not justify that we should just go ahead and do the same shit.

There are no issues with any storage type of having a field that is the revision and use this for

• Displaying revisions to the end user
• Being the revision the user enters when he look up topics by revision

It is only what the end user sees that matters here. How the engine revision the files or the records in a database or in GIT is not important. You can use 50 digit hex numbers if you want. As long as the end user is presented with a simple number that starts at 1 and increases 1 at a time because this is how humans have counted for 1000s of years.

In a database store, the revisions simply becomes a field. In a GIT/SVN whatever store you need to store the human revision and write some simple glue that pairs the two. A bit of work but not very difficult.

If you want Foswiki to be used by people then you have to program for people. You are contributing to a product which is a collaboration platform for people. This is not a project that makes compilers or programming languages or a bug reporting tool for software programmers. If you want to make software with a UI for normal people then the starting point has to be people. And no normal people will ever understand that revisions may jump like 234, 5456, 33422, 90112. No normal human being will ever consider it natural or user friendly that the term "previous version" or "next version" is more a lottery event than a natural predictable sequence. Going 3 revisions back from revision 10 to revision 7 would be impossible without a list of revisions or taking one step at a time.

As a user you need all topics to have a sequencial revision and a date/time. You need both.

I do not understand why I even have to spend time explaining this.

Will this proposal prevent a GIT store? I can not see why? It is a statement of preferred user interface implementation and what the future core implementations should do. We should NOT add code to the core that prevents people from doing otherwise. If someone wants to make useless storage extensions then they should be allowed. Just don't ask for it to be shipped with the default Foswiki. If someone makes a GIT storage that still presents the revisions like the RCS does today, I have no problem how the actual revisions are handled under the hood.

-- KennethLavrsen - 09 Dec 2010

I am not sure that the average user really needs sequential numbers. What they want is "what did it used to look like" in some easy to understand format. And an easy way to step backwards and compare revisions. Maybe there are some use cases for sequential numbers, but I usually care less what the number actually is. I just want either - the "previous" (n-1) or the version "yesterday" at which point I step back one at a time 'till I find the one I want. And of course, presentation as a n > n-1 > n-2 ... is comfortable, and shouldn't change.

If we become too religious about rev. numbering, I suspect that we might make some forward progress toward change sets, and inter-related versions much more difficult to accomplish, so I raise an objection to this as well. You just implemented a plugin to try to address some of our "problem" - to capture the state of a page, and it's search results and cast them into stone. So you are forced to some form of html scraping to capture a topic, with it's search results, at a point in time. Attachments are an equally difficult problem. "Wouldn't it be nice..." if we could ask for a topic, with it's attachments and INCLUDEs and SEARCH results at a point in time. Maybe the "presentation" of the root topic versions is still a monotonic numbering, but I doubt that this type of snapshot linkage could be done with a purely sequential numbering internally. Since relationships between topics are unpredictable, If we cast this in stone we may be casting out future.

-- GeorgeClark - 09 Dec 2010

Some technical details that may have been missed; Foswiki::Store provides getRevisionHistory and getNextRevision which decouple the core from the specific representation of version numbers. Also note that version identifiers are already constrained - to be non-zero, positive, monotonically increasing, integers. Note also that any version in Foswiki is identified by a tuple (web, topic, version) which can be encoded into a GUID when expressing changesets.

-- CrawfordCurrie - 09 Dec 2010

If this feature proposal (which is not really proposing a feature) is rejected - then what?

You go and re-implement the confusing documentation I reverted yesterday? You start implementing a storage for 2.0 that jumps in revisions?

What is it you actually want?

I am very close to leaving this project now. I am losing the trust that Foswiki will continue being a nice product and I may be better off going back to TWiki and just be a passive downloader.

-- KennethLavrsen - 09 Dec 2010

A deeper problem here is that we are discussing solutions and not problems. I have seen this elsewhere on foswiki where debates often get heated as a result. I have been trained to run problem solving teams and from experience of actually running a number of these you need to start with problem storming and not brain storming. Problem storming eliminates a lot of ego and preconceived ideas about the real nature of the problem.

Defining the 'problem to solve' as an initial focal point is a very important but tricky first step.

If we could get all the key players into a room for a day or three I am sure we could really map out the whole problem (every incy wincy detail) and then analyse the problem space in a more qualitative and quantitative way. A further recognition of the problem into independent sub-problems may also be desirable.

Solutions are then best proposed by individuals who are now aware of the whole problem, and can suggest a better solution as a result. The solution need not cover all bases but the proposer already knows this. All the various solutions can be considered and the best solution or combination of solutions can be discerned.

As for this specific problem I have something to say, but alas I'm about to jump off the train! Hopefully, I'll be back very soon.

-- JulianLevens - 09 Dec 2010

I just have been reading up in my git book about tags. These seem to be the natural choice to encode the proposed rev scheme in a way users will be able to handle revisions. Still I doublt that implementing the proposed constraints is worth it. As Kenneth says, this isn't really an additional feature, is it? On the other side after some thinking about git I have no more problems with it either.

Kenneth could you please CutTheCrap? Thanks, makes it easier for all of us including you yourself.

-- MichaelDaum - 09 Dec 2010

[Summary moved up to top for better visibility]

-- GeorgeClark - 9 Dec 2010

Thank you, George, that is a good summary of the objectives, that I for one can subscribe to. I would like to add:

8. It should be made a requirement to track back to a topic revision + associated attachments given just a revision number or date.

This has been an extremely technical discussion, and very useful. I've thought hard about this now, and I think the user presentation of revision numbers is actually a selling point, something that is natural and.... just works. My concern was always that enforcing this regime makes other types of store interaction impossible; but I'm satisfied that intelligent use of tags and mapping can overcome any such problem.

This proposal covers points 1, 2, and 4 above. As such, I'm happy to support the proposal. The other points are equally relevant, and need to be carefully considered. 5, especially, needs attention as a feature, as it is a major failing of the current store.

-- CrawfordCurrie - 09 Dec 2010

okay - removed my objections as well. BTW, #5, "attachment history follows attachment" actually works in the current implementation. The history in the Attachment rcs file follows the attachment. What is broken is the Topic part of the attachment history, which is lost. As is the information that an attachment was moved/renamed, etc.

-- GeorgeClark - 09 Dec 2010

It was very clear to me that there was not agreement about whether revision numbers should be constrained, and if so, how they should be constrained. As far as I know, the feature proposal is the main mechanism we have for hashing out technical issues, so I created this proposal. It has prompted much discussion - mission accomplished.

I didn't know what the outcome would be. What if someone had pointed out that constraining revision numbers has some huge implication that really paints us into a corner e.g. by imposing a huge, unavoidable performance penalty? That could still happen - the clock only started today. I sincerely hope it doesn't. In the very least, this proposal will let us collectively count the cost of our current user-friendly revision numbers.

Is this proposal really about a "feature"? Think about large sets of topics, with topic histories, where topics contain references to various revisions of each other. I do think it would be a feature if we make it possible to migrate a set of topics like that from one store implementation to another (e.g. RCS-based to Mongo- or SQL-based store), without affecting revision numbers. At present, there is no guarantee that it would even be possible, let alone practical. This proposal, if accepted, would (I think) provide that guarantee of possibility.

-- MichaelTempest - 09 Dec 2010

I'm going to add an item 9 - new revision scheme must not cause a wholesale reset of current topic revisions. A static link to Sometopic?rev=5 must still show the same contents if the data is merged into a new store. And I'll merge Crawford's point 8 into the same table so they are in one place.

-- GeorgeClark - 09 Dec 2010

I am glad to see a simple, user friendly proposal emerging.

As I said earlier there were other problems being discussed which caused confusion. I could see the following:

1. Foswiki is a Wiki and has a standard interface. We are primarily concerned with alternative plug-in replacements for the current RCS based store.
   • Some discussion was around coming up with a scheme that would suit any possible future store, one size fits all - good luck with that one!
   • Nonetheless, I do see possibilities for non-standard webs and stores for example
     • No history - for a static web-site from a snapshot web you are developing elsewhere (no I'm not really sure about this one, but I needed an example)
     • A Wikipedia:Literate_Programming web which uses git as a backend. Yes it's geeky, but then it has a specific market. It will still have Foswiki features but some will work a little differently. How this store/web handles revisions in an excercise for some brave soul.
   • Non-standard webs and stores is not merely a technical discussion but also a marketing and project structure issue.
     • Should these indeed be sub-projects like apache arranges these things?
     • ...
2. Has the API/feature set of a minimum store been defined. The revision handling is just a sub-set of all of this, and all of this needs to be defined so that plug-compatible Foswiki standard Wiki like webs can be supported.

-- JulianLevens - 09 Dec 2010

Has the API/feature set of a minimum store been defined - that's what we draw closer to, every time we have one of these discussions. George's objectives will help us draw ever closer.

The Store API is currently defined in Foswiki::Store, but there are two other implementation levels in the system that we try to keep "pure" viz. Foswiki::Store::VC (which is a general interface to a Store built on a version control system, such as SVN, RCS or (maybe) git) and Foswiki::Meta which could - theoretically - be swapped out for an implementation that sits closer to the store implementation.

There are a couple of big holes we are well aware of; first, there is no way to use different store implementations for different webs. For example, you might want to have a "docs" web that was mainly attachments using an S3 based store, which other more dynamic webs use a DBI store. Second (and definitely more future-looking), there is no clean way for stores to interact. There's a tacit assumption in the store object methods that if they are passed a store object, then that object has the same implementation as the current object. That makes moving content between store implementations tricky, to say the least.

-- CrawfordCurrie - 10 Dec 2010

its 6 months later - I think we might be able to accept this by the 14 day rule

note that if we define 1..N as the rule, core changes need to be made, as atm, any topic without a ,v file has a rev of 0 - which imo is bad all round - especially in light of QueryAcrossTopicRevisions and LoadDifferentTopicVersions, where users should be able to 'know' where revs start.

-- SvenDowideit - 12 May 2011

In my view the current rev 0 we see is a bug. It should have been 1. And first edit done then 2. Someone changed that recently. It used to be 1. And yes it is for sure accepted.

-- KennethLavrsen - 12 May 2011

Check Tasks.Item10307. Nobody proposed a better solution at the time... and I think avoiding distinction between an initial rev and a first rev, isn't as easy as it sounds. Also see Tasks.Item9969

-- PaulHarvey - 12 May 2011

I clearly stated I wanted the rev displayed to be 1 if there is no ,v file. And the task was then closed ignoring my statement

I guess I have to raise a new bug item then. To the end user the ,v files existance is pointless and invisible. The end user does not know about RCS and ,v files. They just see a topic with a history. And in this case it is the one and only revision. So it must be 1. The rev 0 makes no sense to an end user. We distribute topics with a rev 1 in the TOPICINFO so why does the code display the rev as 0??

-- KennethLavrsen - 12 May 2011

no bug needed - this feature is the task that formalises it. before this feature, there was nothing defining '0' as wrong - and now that we have this formal declaration, we are able to write tests and constraints to make it so.

course, it'll be a cuid hell for someone that is going to work on a non-linear revision system - but that's not happened yet so 'meh'

-- SvenDowideit - 12 May 2011

Actually, it has happened yet. In the deep and distant I implemented the "Subversive" store which had non-linear versioning, and resulted in me making several changes in the core to support that (since reverted as we have zeroed in on the true way). The only solution I could come up with was a map from linear to non-linear versions, which worried the hell out of me as it's a single point of failure that could easily result in a catastrophic mess. I think any such store would need a "rescue" script that could reproduce a linear revision stream from whatever the version control system holds.

-- CrawfordCurrie - 13 May 2011

The r0 is not the topic revision, it's the topic history. See a topic like System/MainFeatures. At the bottom of the topic are:

... | History: r0 | ...
Topic revision: r1 - 09 Jan 2009 - 12:00:00 - [[../../../Main/ProjectContributor][ProjectContributor]]

I agree that there is still an issue here - but in fixing the history corruption bug (Initial save results in a new r1 without any record of the changes) I couldn't see how to deal with the passing of revision number between meta and store. IIRC the revision number is used for multiple purposes. We probably need API changes so that a revision number test isn't used as both a flag and a pure rev. number. My memory of this is a bit vague now.

The r0 display is cosmetic, while the loss of the rev. when saving a topic is true data corruption. The only recovery is to restore a modified topic from backup or distribution files.

-- GeorgeClark - 14 May 2011

Note. Changed the form to make this documentation. Related topics were: QueryAcrossTopicRevisions LoadDifferentTopicVersions

-- Main.GeorgeClark - 19 Nov 2015 - 16:03