You are here: Foswiki>Development Web>RestPlugin (03 Dec 2010, SvenDowideit)Edit Attach

JSON+REST API for Topics & Webs via browser Javascript

Background

RethinkingTopicInteraction work needs a solid REST API of some sort.
At work we sorely need a decent REST API for shiny web browser javascript to do basic CRUD (Create, Read, Update, Delete - and move/rename), without a stock-standard JS developer having to:
- Make use of the venerable /bin/save and /bin/view scripts which weren't built for AJAX. Consulting CommandAndCGIScripts, requires a JS dev to notice the minor inconsistencies in options between each.
- Write their own javascript to do validation token getter/builder on sites using strikeone
- Write their own AjaxHelper topics just get/set topic metadata; we already have a proliferation of URLPARAM naming conventions, query string concatenation recipes and so-on... our 'ajax-ified' wikiapps are getting completely hairy

Requirements

By POSTing, PUTting or GETting JSON with Javascript in a web browser to a particular /bin/script or /bin/rest/Handler, Foswiki needs a standard way to -

Create, Read, Update and 'Delete' (rename/move) topics (CRUD)
Create, Read, Update and 'Delete' (rename/move) attachments (CRUD)
Get the complete list of attachments on a topic, probably with the meta on each
Get the complete list of topics in a web
- probably with some meta on each item (date modified/author/parent/form etc) (version 1 probably doesn't need this)
Create, 'Delete', Rename (move) webs
Get the complete list of: all webs, just root webs, or subwebs of a given root web
- Topic count against each would be nice (version 1 probably doesn't need this)
Javascript-side validation token handling provided by the plugin
- Access specific versions of topics and attachments (version 1 probably doesn't need this)

Required, but the first version can live without

Well thought-out renaming of backlinks (ideally the js dev needs enough information in a response to a rename command to be informative about setting/reporting scope of backlinks scanned, renames that couldn't be done due to locking/permissions issues, progress indication for lengthy rename operations, etc)
Fixing Foswiki's REST API properly. Where to start... LoginManager vs BasicAuth shenanigans, probably.
A standard API for doing QuerySearch queries (but it would be nice...)
Fixing backlinks in rename so that this is 'sane' (admit it, the way we handle renames with backlinks at the moment is nuts)

Discussion

this is a duplicate of the already partially existent RestPlugin - in which I made QUERY (despite being told it couldn't be done), and even provide mime type to allow requests for json, xml, html, tml....

and it even does GET & PUT....

yes, now it need to re-written to take advantage of foswiki - ie, lots of code to be removed..

-- SvenDowideit - 12 Oct 2010

Ok, re-named this topic to use RestPlugin as the basis for the work.

-- PaulHarvey - 12 Oct 2010

HijaxPlugin already enables much of what you want (if I understand correctly what you want

using foswiki over REST with returned content clearly defined via JSON). It can also be used as a wrapper around RenderPlugin calls to handle error responses.

-- DavidPatterson - 12 Oct 2010

Just last week I was also wondering why we don't have a full REST plugin, so I started rolling my own methods.

-- ArthurClemens - 12 Oct 2010

David, I dont' think I've reviewed HijaxPlugin since you first checked it in, I'll have another look. Arthur, it would be great if you could share your thoughts on methods that you need.

Obviously we're all going to want to do special handlers - but hopefully Foswiki can ship with an api that can manipulate topics in a standard way over rest for browser javascript. EugenMayer pointed me to drupal entities api which apparently has a rest interface component.

-- PaulHarvey - 13 Oct 2010

It would be good to come to a foswiki remote api also useful to base a soap service on, not only rest. See for example Confluence's remote api specification. Quite feature complete.

-- MichaelDaum - 15 Oct 2010

Updated the brainstorm with what I personally think I need from Foswiki

The main point is that we have too many ways of doing it (working with topics in AJAX, that is). There aren't many well-documented recipes for a js dev to go to, especially when it comes to saving data back to the wiki.

-- PaulHarvey - 29 Oct 2010

to get the ball rolling on arguing over how to lay out the URI's and responses, I thought I'd toss this at the wall:

Sven's first thoughts for adding REST API

The URI of any element in foswiki needs to be definable

I'm proposing to use a URI like:

http://server.com/foswiki/bin/query.pl/SOMESearchQuery/elementalias[.mediatype]

the [.mediatype] is optional, and can be set as the proper list of accepted types in the request header.

for example:

http://x61/query/Development.FeatureRequest/topic.html - return the WysiwygPlugin rendered version of the topic (this is afterall primarily a programatic interface)
http://x61/query/Development.FeatureRequest/web.json - returns the Development web's object (whatever that is)
http://x61/query/Development.FeatureRequest/info.xml - the topic's META info - date, author, rev
http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor ~ '*SvenDowideit*'/topic.json?fields(web,topic,Summary,State) -

because each item that is returned needs to be uniquely addressable, I want to define the return as a 'hash of items, where the key is the uri of the item'

eg, the reply for

http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor ~ '*SvenDowideit*'/topic.json?fields(topic,Summary,State)

{
   'http://x61/query/Tasks/Item1234' : {
                                       'topic': 'Item1234',
                                       'Summary': 'Someone changed the way that SEARCH does something',
                                       'State': 'broken by design'
                                    }
   'http://x61/query/Tasks/Item7901' : {
                                       'topic': 'Item7901',
                                       'Summary': 'Add more icons',
                                       'State': 'Pretty'
                                    }
   'http://x61/query/Tasks/Item8432' : {
                                       'topic': 'Item8432',
                                       'Summary': 'DebugLogPlugin',
                                       'State': 'Strange'
                                    }
}

allowing the client to PATCH to any one of the URI's, or to PATCH the lot in one go.

*******************************FILTERING/PAGING/WHATEVER. can we add webtopic to query, so that we can have paging based on the last web.topic name - and limit(20) ... ala distinct.

HTTP verbs

If you can GET a set of elements in a non-lossy form (tml, json, xml), or a convertable form (ie, html from WysiwygPlugin, rather than normal view) you should (depending on your ACL's) be able to use that addressing, and incomng stream of information to modify that set of elements.

Interestingly, the PUT / POST difference means that AUTOINC00 or 10X can work with a POST asis, but not for PUT as that requires it to map to a unique URI.

If we presume to be like google, we can use PATCH as they do...

GET

retrives a web/topic/attachment resource

PATCH

patches an existing resource, updating only the elements that are sent in the body

POST

creates a resource (by default), or if action=rename renames the resource

DELETE

makes the resource go away, puts it behind the couch

PUT

uri elements

prefix url so the API is versioned

we should really hedge our bets and prefix the query with a 'API version' eg. http://x61/query/v1/Development.FeatureRequest/topic.html

specifying what object to return (this is basically missing in SEARCH, and )

looking at google, they seem to end all requests in /full - feels like a good match for the way the QUERY macro already works (plus webs and topics)

/webs - for a set of 'web objects'
/topics - for a set of 'topic objects'
/revisions - meta data about a requested set of revisions - including URI to request that revision
/attachments - for a set of 'attachment objects' - our meta, including a URI to request the attachment
/form ....
/fields - maybe
/preferences
/info
/parent
/moved

when requesting sets, it becomes necessary for each item in the set to have an address - so we need to either have the set be a hash where the key is the address that can be used to request/update that item, or to add the address into the element. I prefer to use a hash - do we (yet) have any topic meta arrays where we have defined duplicates to be both legal and useable (rather than undefined on update?)

specify subset fields to return

?fields=element(listofitems)

ordering.... and limiting and ....

=?orderby=um, like...

output format

mediatype in the HTTP header (ie, content negotiation)
request parameter contenttype=application/json,text/csv,application/msword (CommandAndCGIScripts has it as a scalar, so mmmm - might drop this)
as an 'filename like extension' eg http://x61/query/web.json could return a set of web objects.

We need to differentiate between rendered text and raw tml markup - I'm guessing the correct MIME type would be text/vnd.tml and text/vnd.rawtml (we need something different for raw=debug...

addressing

Query syntax isn't yet expressive enough, but we should use it and drive its further development

longhand == full query syntax

Yes, a pig wrt encoding and decoding, but if we're building queries with javascript..

http://x61/query/d2n('20 Oct 2010') < info.date AND info.date < d2n('1 Nov 2010') AND SomeField = 'save'/topic.json - set of topics who's last revision was made between 10/10/10 and 01/11/10 that have a field SomeField=save
http://x61/query/d2n('20 Oct 2010') < info.date AND info.date < d2n('1 Nov 2010') AND SomeField = 'save'/topic.json?fields=topic(web, topic, info.rev, info.author, info.date) - subset of the topic - what changed?

specific single elements

attachments

the alias.contenttype suffix might allow us to use server side converters - one day?

http://x61/query/Development/FeatureRequest/attachment.json - returns and allows modification of the topic's attachment meta data -
http://x61/query/Development/FeatureRequest/nameof_png_attachment/attachment.png same as viewfile to the nameof_png_attachment.png - and allows GET&POST
http://x61/query/Development/FeatureRequest/nameof_png_attachment/attachment.json would allow GET&POST, DELETE etc on the attachment's meta data (er, unless we clash with an attachement ending in json, ? TODO - perhaps the specific attachment accessors should contain the source file ext and then the content-type allows attempts at conversion?

short cuts for the longhand addressing

Clearly, it would be best if the current view addressing still worked, so there's less surprise for someone familiar with it.

so the following are identical

http://x61/query/Development.FeatureRequest/topic.html
http://x61/query/Development.FeatureRequest

BUT /topic.html probably should be the WysiwygPlugin version of the topic

more examples

*NOTE: need to change this / resolve the issue wrt nested webs (see below)* Where we know the address

http://x61/query/System/webs.json - =System' web's object
http://x61/query/System/topics.json - all the topics in the System web
- ok, so this points out a need for paging / iterator / something
http://x61/query/System/topics.json?fields=topic(name) - names of all the topics in the System web
http://x61/query/System/topics.json?fields=topic(name,FIELD.SomeField) - names of all the topics in the System web, paired with a formfield (name, value, and whatever else belongs in the FIELD object..)
http://x61/query/System.ProjectLogos/attachments.html - the attachment table for ProjectLogos

obscure corners

Nested web implied WebHome

Someone once allowed foswiki to have both a topic and a web with the same URL name (the only difference being a trailing slash)

this means that http://x61/Development/Stuff does not equal http://x61/Development/Stuff/

case sensitivty

http://x61/Development/Stuff does not equal http://x61/Development/stuff

though this may change one day

-- SvenDowideit - 01 Nov 2010

Cool, this does look really nifty. It seems like a lot of problems to solve... Okay, some observations:

v1 doesn't need full CRUD on webs to be useful. Just a way of listing topics, subwebs and bit of meta for a given web.
QuerySearch in the URL - could this cause more trouble than it's worth? Detection of double-(de/en)coding; dealing with '?', '&', '%', ';' - Eg. for regex queries and content values. Might it be easier to:
- Support full URL path for getting/saving whole topics, attachments;
- For queries - encode the entire query into a dedicated url param
HTTP PATCH - I personally have no problem with using it, as long as we support PUT for those who can't (bad proxies? strict webserver configuration?)
Media types - an extremely useful thing, but v1 doesn't need all of it to be useful. Also, I would rather use HTTP accept headers, aka content negotiation - this is the way (most) other APIs on the web work (it's really the only documented way to get this using only HTTP spec); but certainly there are a couple of services we deal with which also support the pseudo-filename-extension method (when you specify the resource without an extension, it uses your HTTP accept headers to determine the destination eg. .html, .xml, .json etc. for the subsequent 303 redirect)

-- PaulHarvey - 01 Nov 2010

in some ways doing the same thing for all supported 'objects' is easier than coding them bit by bit, but excellent - moving, deleting and creating webs is actually a multi-part operation, so it may be easier to stub those.
- the thing is that the 'specific item' URI is a simple, and specific case of the Query URI, so its easier to just write one code path
first up, encoding and decoding isn't difficult, and as this is primarily a programmatic interface, i don't see it as a problem.
ok, fallback to PUT basically means its pointless to write PATCH - will take your point and use it to write less code
media types and content negotiation are pretty much the same side of the same coin - I only added the urlparam and .json style extension to simplify debugging (it's basically no extra code)

-- SvenDowideit - 02 Nov 2010

Next to existing topic data it would be necessary to store other kinds of (json) data, for example .../topic/put/mylist or .../topic/put/myhash.

These might be stored in the topic meta data, as long as putting and reading is using a verb syntax.

-- ArthurClemens - 02 Nov 2010

This would require PATCH verb to work; PATCH to 'topic'/META:FIELD[name='myhash']

-- PaulHarvey - 03 Nov 2010

I am sure I must be missing something, here, so please bear with me

Between TMCE and WysiwygPlugin, there are already ways to read the WysiwygPlugin version of a topic (although we don't use it at present, so I don't know that it still works), to save Wysiwyggy content to a topic, and to convert between HTML and TML. That is four operations, of which three are in regular use (only the first is not). This proposal addresses the first but not the other three. If the first does not work, then I would rather fix it than provide a completely separate interface.

Also: what should these do?

http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor ~ '*SvenDowideit*'/topic.html
http://x61/query/Development.FeatureRequest/web.html
http://x61/query/Development.FeatureRequest/info.html

-- MichaelTempest - 03 Nov 2010

Relevant work and observations

The FilesysVirtualPlugin (part of the WebDAV suite, not released open-source yet, sorry) has a plug-in architecture that supports the recovery of resources in a range of formats, using file extensions as the representational selector. For example, Topic.html and Topic.json reveal the HTML and JSON representations of a topic, respectively. This is very similar to the way Rails does it.
A clear and unambiguous specification of error handling is required. The current Foswiki "rest" script suffers by not having this built-in, forcing developers to invent their own meta-protocol for handling errors. The WysiwygPlugin has a fairly advanced model using HTTP status codes, but it's a one-trick pony. A general purpose REST protocol requires a general purpose error handling model (and Javascript classes tailored for it)
Most of the discussion above relates to resource read. I am rather more concerned about the problem of resource write. For example, say I have a REST resource that represents a topic, and that topic has an attachment. If I write the resource with new JSON that doesn't have the attachment mentioned in it, what happens to the attachment on the server?
While I appreciate the need to maintain implementation-agnosticism at this point, I am somewhat concerned that as it stands, each REST request requires a rebuild of state on the server side, which is an expensive process. I think this proposal has the potential to bring the hardiest server to its knees unless state can be maintained more efficiently on the server (for example, a transactional model).

-- CrawfordCurrie - 03 Nov 2010

first up, Michael, yes, the html view is a duplication of the existing WysiwygPlugin response - it is just one view into the data, and the point of the programatic REST API is to provide one consistent API to GET/SET/PATCH/DELETE using what is currently considered a good resource based style.

it is just going to be an API to the existing code.

http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor ~ '*SvenDowideit*'/topic.html
- this one can do with some discussion - imo it would be some variation on the TemplatingSearchResultSummaries - as it sets us up for using the API to request rendered sections - useful for distributing load, or simply defered rendering..
http://x61/query/Development.FeatureRequest/web.html
- I'm not entirely sure what a 'web' object is, but essentially, a html rendering of one = perhaps giving information like favicon, colour, other WebPreferences.. but using the sectional html view
http://x61/query/Development.FeatureRequest/info.html
- like attachments.html, this would be a html rendered partial element that could be requested dynamically and pasted into the right place

ok, Crawfords q's

yup FilesysVirtualPlugin can do some of this, but until it is open source, its totally irrelevant to us
Error handling - agreed, I'll have to spend some time on that ASAP
the discussion above from my POV, is about all CRUD operations, not read - the point is that the URI is the identifier for all operations, the HTTP request type is the verb, and the payload is the data. If you POST a topic.json, without the attachment data it should remove the attachment (though I'm not 100% sure we should do this, that would be the convention). Most existing foswiki operations use the PATCH style...
rebuilding state is a fact of life, we do it every change operation already, do there's little downside to adding more usecase appropriate APIs. However, my intention is that the json view is the fast path - as it is a very close mapping to the Meta object.
- though I must say, i'd like to remove some of the un-used arrays, and convert things like the attachment array into a hash..

-- SvenDowideit - 03 Nov 2010

I can see the usefulness of http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor ~ '*SvenDowideit*'/topic.html as you have explained it, but that interpretation appears to be at odds with "/topic.html probably should be the WysiwygPlugin version of the topic". Either that, or ".html" is magically context-dependent, which might not be a good idea.

Also, CRUD is fine, but WysiwygPlugin has two commonly-used REST interfaces (HTML2TML and TML2HTML) that are not CRUD. Are these excluded from this discussion precisely because they are not CRUD? Is it an oversight? Or have we simply not got there yet?

-- MichaelTempest - 04 Nov 2010

Hi Michael, WysiwygPlugin provides a useful translation service - and TinyMCEPlugin gives you a way to to work on a topic's main text. I would hope that one day, my EditorAPI hack which secretly translates html-in-formfields to tml-in-formfields, could actually think about using RestPlugin directly - especially for in-line edits from a view.

-- PaulHarvey - 04 Nov 2010

At its root, the 'real' REST API should provide a single consistent way to GET,SET,MODIFY addressable resources. That means duplicating existing ways into the data (given that the entire thing is a duplication of all the GET&POST param RPC's we currently have.

I'm not sure i agree that /topic.tml in a scalar context (ie, Main.SvenDowideit/topic.html) being a WysiwygPlugin view of the topic is inconsistent with making the list context use a summary list ala WebSearch, but thats the point - i'm not sure.

mind you, you are foccusing strongly on one specific mimetype, when it is largely there for consistency - its pretty odd providing an API that can do HTTP content negotiation, by then arbitarily doesn't provide text/html.

-- SvenDowideit - 05 Nov 2010

This still appears to me as a tangled ball of threads. I am sure that one day, we will get a jersey out of it. In the mean-time, I tugged on the one thread I have a vague understanding about

I have no problem with providing an alternative API. However, the API should then be flexible enough to deal with existing use-cases, or else those use-cases should be excluded from the outset. I asked about the tml<->html conversions because I wasn't sure about the intended scope of the API. Those conversions are performed with reference to an addressable object (a topic) but they don't set, get or modify the addressable object.

Here is another example by which I intend to illustrate the same issue: Suppose I had a RandomQuotePlugin that provides %RANDOMQUOTE{SomeTopic}% that picks a quotation at random. The quotations are stored in tables in topics. You specify the topic and the plugin picks a random quote from the first table in that topic. Now I add a REST handler that does the same thing. Does this hypothetical rest handler fit into the scope of this discussion, and if so, how?

There is a use-case for wanting a non-WysiwygPlugin view of a topic in scalar context - browser-side %INCLUDE (e.g. Mystuff.TakesALongTimeToRender/topic.html ). Perhaps the WysiwygPlugin view of a topic should be a different mime-type, or accessed as (say) topic.editable.html

Something else to consider: the WysiwygPlugin view might turn out to be many views. We are already at a place where WysiwygPlugin emits HTML that is intended for editing within TMCE. Before that, it was tailored for Kupu. We are now considering supporting various different editors which may have differing capabilities, which means the WysiwygPlugin conversions may have to be configurable, which means passing parameters. Does that affect this API?

-- MichaelTempest - 05 Nov 2010

We lack a coherent javascript API to do basic CRUD from the browser. So that's the basic goal of this plugin. The idea is that we should be able to also do basic CRUD on any 'TOM' (QuerySearch ?) addressable element of a topic. Given that tables aren't yet covered, your plugin example would not be able to use this.

As for WYSIWG, I share your concerns - but I think it's possibly too early to worry too much just yet. Wysiwyg isn't the primary target for this service just yet, but I hope we could consolidate everything after we get some experience with it. Maybe there will always be a case for WYSIWYG to do its own thing - but I do hope that RestPlugin could take over listing/saving attachments, and most other topic-related operations required on an edit screen.

Because really, getting/setting QuerySearch addressable elements of a topic should be easy to do.

-- PaulHarvey - 06 Nov 2010

Yes, I agree, a basic CRUD API should be the goal. When you are using Rails (or any other REST-based framework) you tend to build apps using CRUD as long as possible, but you often have to flip to providing your own services that don't fit the CRUD model, in the way we did in WysiwygPlugin. So, we need 2 things:

A Javascript CRUD API
A framework for the extension of the API for non-CRUD calls.

BTW my recent work on the EditRowPlugin is pushing the boandaries, as it is addressing elements that are not addressable using the query API (specifically, elements in TML tables). I'm torn between completing that work with a roll-my-own API, or trying to extend the query model to support addressing those elements. I don't have a problem with doing that for REST, but addressing table elements in SEARCH is hairy.....

More thoughts in HowToAddressNonMETAmetadata

-- CrawfordCurrie - 06 Nov 2010

Have a look at backbone.js, a lightweight MVC framework. It includes a RESTful JSON interface connector. I am using this (small, 2K) library for another unrelated project.

The documentation on REST is at http://documentcloud.github.com/backbone/#Sync

-- ArthurClemens - 06 Nov 2010

Discovered Development.AddMetaSetActionToEditAndSave, where it seems many of the same ideas are repeated here

-- PaulHarvey - 07 Nov 2010

BIG TODO: both Error handling and authentication (sessions, you name it) are undefined atm, and will need work.

I'm starting implementation adding Foswiki::UI::query (in the plugin) in the hope that this allows us to define the authentication better

-- SvenDowideit - 12 Nov 2010

Also, what about request validation (StrikeOne)? It should be hidden behind the JS API, IMHO.

-- CrawfordCurrie - 02 Dec 2010

Again, HijaxPlugin with its hijax.js offers error handling, authentication requests and strikeone validation on most of Foswiki's actions via the foswiki.HijaxPlugin.serverAction({...}) interface (a practically 1:1 drop in for jQuery.ajax({...})). This is without having to duplicate anything. As a wrapper around RenderPlugin calls, you can perform any macro call you want with all this protection.

Yes, it's a hack, but until RestPlugin is ready, you can already start building applications on this sort of interface.

Still some bugs in my working version on IE and Chrome where I'm trying to execute the onSubmit js on forms before executing the action over ajax. And upload is missing.

-- DavidPatterson - 02 Dec 2010

given that one way to use the RestPlugin is to use curl:

curl -X PATCH -H "Content-Type:text/json" -d '{"_text": "set the topic text to something"}' http://x61/f/bin/query/Main/SvenDowideit/topic.json

can be used to send a JSON payload that will change only the topic text of my user topic (ignoring authentication and request validation info)

I like the restriction that the HTTP payload is only used for resource information, and so am working towards having both auth and strikeone info site in the HTTP Header.

yes, there should be a Perl and javascript class to abstract the sending of requests - but curl is a wonderful touchstone for simplicity of resource oriented actions

-- SvenDowideit - 02 Dec 2010

Sorry about my previous post, really egocentric. What I meant to say was, Why are you putting effort into another plugin rather than fixing the core rest implementation? You're talking about duplication of the core functions (actions, error handling, authentication etc.), wouldn't it be better to upgrade the rest interface to map to the core functions from within the core? Hijax already maps to the actions and Render already maps to the macros. Perhaps I don't get what you're doing (quite likely), but it seems to me that you're just creating an alternative. Isn't the best solution to fix what's broken rather than implement another workaround, and all of you core devs seem to agree that the rest implementation is broken. Then we wouldn't need the likes of HP or RP.

-- DavidPatterson - 02 Dec 2010

ok, we need to be clearer in our writings. the existing RestHandler work is not broken, its just mis-named. it should be called RPCHandler - as that is what it implements - a way to call methods that then may or may not affect the resources.

What I'm working on, is a new 'script' query (though I am willing to rename, i don't find myself caring about the name) that is trying to implement a CRUD style REST API - a different view into the data.

Mapping core functions is basically what I'm doing - in that this API gives access to what are essentially methods on Meta - using the HTTP verbs (ie, what REST means)

-- SvenDowideit - 03 Dec 2010

BasicForm edit

TopicClassification	BrainStorming
TopicSummary	We need a plugin to manipulate webs, topics via REST + JSON
InterestedParties	PaulHarvey, SvenDowideit, CrawfordCurrie