Feature Proposal: Abstract stores to distinguish between handling attachments and their META (was: Handling attachments is distinct to Store-ing them, so abstract out into Vaults)

Motivation

While developing VDBI, I needed to write code to physically store the attachment in the file-system. I ended up looking at the attachment code in PlainFileStore and in RcsWrap /Lite. It's clear that I should abstract these out, an admin could then choose VDBI with an Rcs style Vault which diffs between revisions or a PlainFileVault that stores each revision in it's entirety.

This will also simplify writing a store as a fair piece of work can be offloaded to an existing Vault.

Description and Documentation

At this time I wish to maintain a tight focus and initially abstract out a Vault from PlainFileStoreContrib and offer this back to CrawfordCurrie to update the same. This will help to better understand the required Api and the division of labour between a Store and a Vault.

Future vaults could support DBMS backed attachments or the cloud. Multiple vaults could even be supported. However, It is not the intent of this proposal. The writing of a Vault manager or Vault listener will need further work and proposal.

As per StoreAs1stClassExtension I do wonder about Vaults also becoming an extension type or just calling them an AttachmentStore but again that's another proposal.

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: JulianLevens - 30 May 2012

Discussion

in the store 2.0 refactor, a foswiki will be able to use several store backends at the same time. And to head towards the point where topics and attachments are the same thing - a stored item, I tried to set things up so that a web is not tied to one store, but rather that all stores get asked (in turn) if they have a resource.

So basically, store2.0 does not differentiate between topic and attachment, except by the legacy conventions - so your vault kinda duplicates this work, but with less flexibility.

-- SvenDowideit - 31 May 2012

From my FW 1.1/1.2 and VDBI perspective I'll end up copying code from RcsLite /RcsWrap, ho hum.

From a Store 2.0 perspective, do you imagine Stores offering Topic services or Attachment services?

I have so many questions about Store 2.0, I started looking at your Github stuff, but decided that I needed to stay focused on VDBI and FW 1.1/1.2.

When that's done I need to badger you about Store 2.0.

Until then I can only look forward to Foswiki 2.0 with Store 2.0, Attachments 2.0 and more contentiously CharacterSet 2.0.

-- JulianLevens - 31 May 2012

> all stores get asked (in turn) if they have a resource

Doesn't sound right as it means: the slowest takes it all down, and even: when one store is currently unreachable (cloud), the whole thing goes down?

Sven, did you outline these kind of features you subsume with store2 somewhere? Can't find it. And SimplifyTheStoreMetaSemantics doesn't seem to be talking about it whereas MultiStoreRefactor seems to be stalled and in need of an overhaul with regards to your store2 ideas.

Reading MultiStoreRefactor, it seems you envision a per web store. That sounds better as a design decision.

Slicing it along topics-vs-attachments as Julian proposes makes sense too, yet is quite a different beast. Databases vary a lot implementing blobs. So a database store is probably doing better keeping those large binaries out of its tables and only store their metadata. Doesn't sound too hard to do conceptually.

There are however some potential storage services whose interface allow to operate quite differently, with foswiki only orchestrating the interactions of a user's browser with that store, as it is the case for cmis. In these kind of setups, foswiki as a storage vault is largely out of the loop, just visualizing what is stored somewhere else, like in an alfresco dms.

-- MichaelDaum - 31 May 2012

Michael, it doesn't sound right because it is not right. Why would you keep asking for a stored item after you have one?

re-reading SimplifyTheStoreMetaSemantics, yes, I've not set the address->store mapping.

What you need to understand, is that I am designing store2.0 to cater for both these possible implementations. Its already capable of both (other than the bugs :)).

Store2.0 is the abstraction layer between foswiki-core and multiple backends (in both directions..)

Hardcoding to one store per web is a terrible design decision. its slow, cumbersome and needlessly limited ().

Julian, Just to reassure you a little, it is easier for you to work on the the pre-store2 implementation of VDBI, because the unit tests pass, and the bugs are ironed out. When store2.0 comes, the conversion of your existing code will be relatively easy - I'm trying not to change the inner Store API as much at this point - its more that the middleware is getting a major revamp.

fundametally, store2 is an address->item service, where the item can be any type that foswiki's core understands, and can be located anywhere that a loaded store backend understands - cache, db, external resource, whatever.

-- SvenDowideit - 31 May 2012

I've just re-read this and I think I now see why our thoughts did not seem to connect. Basically, I get the idea that an attachment is an object to store just like a topic. So Store2 offers a topic to Versatile which promptly stores it away. OTOH, if Versatile were offered an attachment it would say "pass".

However, that's not quite refined enough. There is both the attachment and separately it's META to consider. If store2 handled an attachment as follows:

1. Offer attachment to stores until a store accepts responsibility for it
   • Physically store the attachment (but not META)
   • Return attachment META to Store2
2. Store2 then offers the attachment META to other stores to see who will store this

That would be fine. I am under the, maybe mistaken, impression that at the moment an attachment is offered by Store2 with it's META

The reverse process where a store has been queried (on META attachment details) to provide enough info about the attachment back to create links for upload etc would also be required. (Roughly the reverse of the above).

An SQL based backend will be great for handling attachmnet META, but rather poor with storing the attachment itself. If Store2 does or will offer this refinement then a vault is just a type of store (or part thereof).

This then leads me to how do I support attachments in Versatile before (the refined) Store2? One issue is that when I look at the store API. It assumes attachment with META and does not separate out these concerns.

Some stores might want to store attachment & META simultaneously (ATOMIC commits), but that will not be the common case.

-- JulianLevens - 05 Oct 2012

yup, either we're still not connecting, or you're a tad wrong.

in the current store, and attachment (the file itself) is quite separate from that attachment's meta info - which is (sadly) stored in the topic.

However, from the replacement Meta code I've been working on over the ages, its not that hard to separate it out. For working on what will be 1.2.0, it would be similar toa combination of the serialiser extraction that I hope to commit next week and the MongoDBMeta class - something that retreives the data for META:FILEATTACHMENT when its needed from the 'other' source.

in store2 - well, thats dead atm, i've not touched it in ~10 months, and won't look over there until after 1.2 is out.

The shorter takeaway is that there is more to Store implementations than just the files in lib/Foswiki/Store - its convoluted, because it wasn't coded to be pluggable, we just keep trying to make it moreso.

-- SvenDowideit - 05 Oct 2012

Ok, I've rejected my own proposal.

When I add attachments into Versatile and look at the code/API more carefully I'll hopefully see more clearly what you mean. It has not been obvious that there was already a separation of file/META concerns.

Some of the store API docs are somewhat sparse, I'll add my notes to flesh them out.

-- JulianLevens - 20 Dec 2012

Commenting is disabled while not logged in