PlainFileStoreContrib

I just checked in a first version of this simple extension. It's an implementation of a Foswiki store using nothing more that plain files. It extends the existing VC handler concept, so shares some overheads with the RCS stores, but is a lot simpler.

I've tested against the standard (trunk) test set. The contrib should work fine with Release01x01, though I haven't tried it.

I'm interested in any feedback.

-- CrawfordCurrie - 23 Mar 2012

Awesome! I have been meaning to try this myself. Will try to test in the next month

-- PaulHarvey - 24 Mar 2012

I'm keen to use this contrib to learn how to go about doing a proper git store. All our stuff is in git right now, just to ease synchronization and staging of large bulk-edits & imports; although it's possible I'm just addicted to git and need professional help. I wrote some of this up in BestPracticeTip29.

I wanted to start by working to make META:TOPICINFO work out-of-band from the .txt files. Are you okay with that? Would you prefer to diffs first? Well, there's no hurry anyway, I doubt I'll get a chance to do all this any time soon, although my current usage of git is proving to be a workflow bottleneck, and I can't go back to rsync at this point.

-- PaulHarvey - 24 Mar 2012

can you make the OOB TOPICINFO separable so that we can use this in both styles of serialisation?

Actually, can you do this by making different serialization selections? using the Serialisation classes I built into trunk (I hope its not only in store2..)

-- SvenDowideit - 27 Mar 2012

Paul, sure, I think that's a great idea. I always indended this as a back-to-basics experimental platform. My next step was going to be to collapse the individual file handlers into a single 'Store' (i.e remove the dependency on VC) but taking META out of band is a great way to go. And Sven, I thought you had made separable serialisation a feature of the core store?

-- CrawfordCurrie - 27 Mar 2012

Yep, it's all there - Foswiki::Serialise - it was RestPlugin (and MongoDBPlugin) that needed it.

-- PaulHarvey - 27 Mar 2012

To make this store more acceptable, we require rock-solid conversion from existing RCS stores. So what are the requirements for this?

• Extract all revisions of a topic/attachment into a plainfile folder. This folder can co-exist with the ,v (the ,v will be ignored if PlainFileStoreContrib is the selected store provider)
• Check, and possibly correct, META:TOPICINFO in each topic revision?
• Ensure modification dates are set correctly from the RCS metadata.
• Handle files with missing ,v's gracefully.

Because some files are revision-count greedy, they need to be excluded from the conversion. I'm thinking mainly of WebStatistics here, but other files may be involved.

• Support a mechanism to limit the number of revisions that are retained.
• May need a mechanism to filter certain topics/webs completely.

The reverse transformation:

• Reproduce correct checkin dates (from file modification dates on the rev files)

-- CrawfordCurrie - 25 Apr 2012

How would you go about correcting the TOPICINFO in each topic rev? What would be wrong with them?

In our wiki, I've culled all topics that I know of that had many hundreds -> thousands of revs.

Leaving the ,v files hanging around makes me nervous, but I can live with it (I can move them out myself if I want to).

I'm just seeing a future admin do a Foswiki upgrade or re-configure, without realising (or being confused by) the topics quietly having a trucated/ancient history if the store ever accidentally got flipped back to RCS.

-- PaulHarvey - 26 Apr 2012 - 00:36

Re: TOPICINFO I don't know. On reflection, I will probably leave them alone.

The thousands-of-revs topics will always be an issue for PlainFileStore, so much so that I think a pruning mechanism is de rigeur.

Leaving the ,v files around is a poor-mans backup mechanism. I'm always nervous of deleting stuff. If god had meant us to delete stuff, he wouldn't have given us ...... actually, maybe deleting them is the best option.

-- CrawfordCurrie - 26 Apr 2012

Oh, I'd never delete ,v files - I just want them somewhere where they won't cause confusion a find ...| xargs... incantation should do the trick. Or just a full backup & delete.

-- PaulHarvey - 26 Apr 2012