This is a big change as one of the parts of the HTTP model is the small “uniform interface”, where there are very few things you can do to web resources. GET is the most common, to retrieve a resource representation. Then there is PUT to update a resource, and DELETE to delete it. Then there is POST, which tends to cover everything else you might want to do. The problem with that is that discovering the interface for POST is difficult, as is knowing exactly what it will do. (There are a few other verbs too).

PATCH is much more straightforward. PUT updates an entire resource with a new version, while PATCH just makes an amendment to a resource. For some types of resource, the entire resource may be large, so that just sending differences will save bandwidth. Also, sending the full resource may unnecessarily make the changes sequential, for example append operations where the order of the operations is not significant. One example given is a log file, where many processes may be adding entries, and if they had to retrieve the whole log, append a new entry and write it back there would be a lot of extra traffic, and a chance of either lost updates or processes having to retry if the resource was modified during this process. Clearly a PATCH operation here that does an append would make sense. I am not sure that is actually a very good example though, as you would almost certainly create a resource for each log entry, rather than one for the whole lot, but clearly other similar patterns exist.

HTTP is not a filesystem

When it was new, people tended to treat HTTP like a filesystem. After all that was the common model for storage, and web servers generally stored web pages as files, so they tended to e treated much like that, with filename extensions annd index files, and WEBDAV was created to try to make the web usable as a filesystem protocol. This model does not really work very well however, as it does not model the things you can and can’t do with HTTP. The methods are one example; updating entire resources at once means they tend to be small units, rather than, for example, log files. File systems generally struggle to store millions of tiny files without wasting a lot of space, and without becoming slower. The web resource does not have to support full Unix filesystem semantics (a topic that oddly Wikipedia seems to be missing an entry on! May have to rectify that), and supports a much simpler updte model.

Maybe the easist way of thinking about HTTP is to see every URL is a small Model View Controller (MVC) system. The model is an abstract resource, which we can see through GET, which retieves views. There can be multiple views, as a request can ask for different media types and encodings for a single resource. The controllers are the media types supported by PUT, which are usually the same as those for GET, but need not be; because both the view and the controller need to represent the whole resource state, they do tend to be quite complex representations, such as XML documents. PATCH however is also a controller, but a more interesting one in many ways, as it can send just state changes to the model, which tends to be how many MVC systems work.

Another thing that PATCH enables is resources that hide some of their state. A resource could only support PATCH and not PUT so that state modifications were only changes. If the state returned by GET is not the complete state, the resource could hide parts of the model. An example could be a voting method that records but does not reveal who has voted, only returning totals, which accepts votes as PATCH requests.

Server side scripting

One PATCH format that makes a lot of sense is actually to use executable code, rather than say diff files. There is no reason why you should not send the server a PATCH request that is some Javascript to modify the DOM of an HTML resource which can be executed serverside, or an XSL transform to modify and XML object. Sending code is an efficient way of making changes to a resource, and can be executed in a sandbox like the browser sandbox. This will be another driving factor for Javascript on the server side, as it is well suited for embedding like this, and already has a DOM model for transforms.

All these changes take us further away from the filesystem model. Web resources will more and more combine some storage with some computation, including ability to execute code in a contolled way. Smart resources will become more common, over dumb storage only resources.

In other news

Next in line for HTTP, hopefully, is the Link header which adds a new header for legacy document formats that do not include a native hyperlinking capability. This will allow relationships between these documents to be included in the retrieved resource, such as a link to metadata or other related resources. The HTTP replacement for the file system model is getting serious.

SPARQL DESCRIBE is a somewhat controversial feature of SPARQL that returns an RDF graph “around” the query, including all the referenced URIs and some sort of domain specific context. It is less well defined than CONSTRUCT, that also extracts a graph.

But there are not many guidelines for how to actually write useful interfaces for search type functionality for REST web service APIs.

A lot of this applies to other collection type interfaces, collection in the Atom sense, which is essentially a predefined filter rather than one you can submit a query to. Resource collections are a fundamental feature of any real world system, being able to take a set of resources and manipulate it as one. Query languages are generally an efficient way of defining interesting collections.

I do not specifically refer to SPARQL here. Like many of the W3C recommendations, it is XML document oriented in its specification, not HTTP oriented, and so does not try to address the types of issues here. I have some issues with the W3C approach, sidestepping implementation issues; indeed the whole RDF as Prolog wrapped in XML does not seem very productive, and I think a more pragmatic approach from practise will work better, as with some of the other W3C initiatives. Items like DESCRIBE imply much more domain knowledge, and it is unclear what interoperability would mean.

Whats the problem with query languages and REST?

The main issues are that a SQL style SELECT should map to an HTTP GET, but instead we are passing a query with another verb in it. One method of avoiding this would be to work with predefined refinement “queries” where you could walk through paths of filters from a resource that represents everything. Imagine a faceted navigation system drilling down to construct what becomes a filter query through a selection of refine links (where you might end up with a hierarchical URL in the end not a query document). This might work better in a system with a finite vocabulary, although it could still be navigated via forms potentially. However this starts to work less well with complex nested queries with sorting and limits, and subterms, as the number of opotions is large, and it would be complex to make a comprehensive set of link types to make them understandable (you might just end up tokenizing SQL in links which does not seem elegant).

REST search and GET queries

First off, for standard read only GET queries, there is one model that is really useful and is too rarely used. For some reason most search queries in “REST” frameworks involve submitting a query as a POST request because the query string would be too long for a GET request. This however generally does not create a resource, it just returns a response, or a temporary response resource. CMIS for example works along these lines. However the sane model along these lines is to submit a query document with POST (or PUT) and have it persistent until DELETEd, and be able to requery as needed. Also, it should be possible to create query resources with unbound variables, which can be used by a system like prepared queries in SQL databases; in particular the creation of these should allow a system to work out perhaps if it needs additional indexes based on the queries that have been created. These can then be evaluated by using query strings to bind the parameters. For some reason you rarely if ever see a query specification for a REST API that has unbound variables in the query, thus missing out on the ability to use this pattern.

Note that the query itself is a resource, so we can for example DELETE the prepared query eg http://example.com/query. There are also resources corresponding to the result sets eg http://example/com/query?q=red.

As well as unbound variables for constants, another missing feature from most query languages (probably because they were not designed for HTTP transport) is the lack of ability to specify a URL that corresponds to a subexpression. This can be considered as another extension of unbound variables; you can also see this as the extension of the labguage to support SQL style views that can themselves be queried. Like a view, you could either use the definition or the current materialized version for performing further queries.

REST and update queries

Although it rarely seems to be done, there is no particular reason that once you have a query resource like the ones defined above, you should not be able to DELETE http://example/com/query?q=red to delete all red objects, or PUT a new set of red objects to replace the existing ones. Most collection oriented protocols, such as Atom, do not allow PUT and DELETE methods on collections, only on the member resources. POST is allowed in Atom, to define collection membership; this only makes sense where collection membership cannot be defined by visible properties of a resource; the instruction create this item with whatever properties are needed to put this item in this query result is not necessarily something that works for any domain. Certainly one can think of models in which allowing PUT and DELETE on search results resources would be useful, although there might be issues with paging and large document sizes on some types of resource. PATCH would be a useful addition for doing the equivalent of SQL UPDATE, in order to reduce the document sizes, rather than having to PUT back everything.

There may be other domain issues, and in some systems consistency issues with mass deletion and insertion that mean that actions on individual items not collections are more efficient of course, but extending the uniform HTTP interface to collections should certainly not be ruled out for many domains.

It started with an idea I had, in an innocent sort of way. I thought if I looked at the JCR specs for a bit I might find some kind of way of building a non Java interface with them. You know, maybe there might be a nice REST architecture waiting to get out. But of course there is no such thing. It is an application definition. There are not even that many ways of implementing it, other than choosing your object persistence method to be a database, file, or something else.

The REST architecture is notionally provided by another layer, such as Apache Sling, but Sling is in no way a REST layer, it is a URL dispatcher and scripting and application layer with which some REST style applications can be developed. With that you end up with a pretty heavyweight development framework, indeed together you have much of Day’s CMS offering in effect, rather than a lightweight REST repository solution.

I had a look at CMIS again. Fielding once laid into CMIS for not being REST and you can see why, although some improvements have been made since that. Although resources are discoverable through hypertext, there is a fair amount of semantics that needs to be known to understand what a type or a checkout means, and the search queries are obviously just RPC wrappers. It is not too bad though, but unfortunately the data model does not map well onto web content management right now for obvious historical document management reasons. Fixable? I think it serves a part‎icular purpose well and should probably not be forced into anything else, as we need it to succeed in its field.

Day claims that JCR is not a Java standard in an odd way, that you can implement the API in another language. Thats a strange argument to make, especially as the types are defined as Java types, and standards without interoperability are pretty vague. Without some sort of wire format or ABI this is meaningless outside the JVM world. People are making JCR like repositories in PHP but outside any standards process, so in the end this just becomes a PHP repository project; Typo3 seems to be building another, also closely aligned to JCR.

The problem with these efforts is that it is not helping the balkanization of web CMS, which is already fragmented by language and API, which is ridiculous in an industry that is about the web. The web has an architecture (REST) and an API (HTTP). Building web content management on Java APIs or PHP APIs or .NET is a legacy way of thinking; it is acceptable for document management given its role in existing enterprise architectures, but it is not going to work if we want to get widespread acceptance in web development; in the short term it is the easy path, it is what people are used to, but a forward thinking industry needs to look at defragmenting the landscape and building future proof tools.

The odd thing is that a web content repository alone surely lends itself to a simple REST architecture. Content is after all lots of small resources with relations. Hypertext. It is pretty much in presentation a fairly dumb web application, although with a fair amount going on behind the scenes. It takes content, relates it to other content, and serves it back, with authentication and versioning. Everything else is in other system layers, transforming it and so on. Not simple, but well defined; lower level than JCR + Sling say

So we need to work on a web content repository model, as a community. Process wise, it makes sense for this to sit in an organization like AIIM, as a content management based industry body. It may well be that what ends up coming out of this is more standardized architectures and semantics and open source implementations rather than the tighter prescriptions of JCR and CMIS; I have some ideas along these lines that I need to code up. I have had some discussions and there is a degree of interest in some sort of solution; who is interested? Or is infrastructure dead, everything ust wants interfaces?

Only solutions that produce partial results when partially implemented can succeed. The network is littered with ideas that would have worked had everybody adopted them. Evolvable systems begin partially working right away and then grow, rather than needing to be perfected and frozen. Clay Shirky

I came across that quote from Clay Shirky as I was writing this which seemed very relevant.

Jan Algermissen added the two following very good points:

the use of REST brings the decentralization aspects of the Web into the enterprise world and allows designers and developers of networked systems to create and evolve their components without the resource and time consuming effort of bringing them into a single room to discuss APIs. In geographically distributed organizations this might sometimes even be impossible.

the use of REST facilitates fragmented change. It deliberately provides room for evolution (e.g. format extensions) that can bypass the main organizational control. Evolutions that turn out to be valuable can flow back into the main line while others silently die. No harm to ongoing communication between systems is done by either of those.

These are really very important points. You cannot really undersell the success of the web, and the web took off with its own architecture despite attempts to force other architectures on it. The decentralization and the ability to reuse and mashup services to make new things is part of that architecture. The idea of introducing REST to the enterprise is to help the enterprise leverage the agility of the web architecture, and to make building new enterprise applications cheaper and easier.

Benjamin Carlyle made the good point that you can view REST as an evolution of SOA, as better easier to consume services. This does not quite fit with my view of history, but I do agree that it is a good way to sell it now as “SOA on steroids” as he says. He linked to useful sounding book coming out later this year on SOA and REST. He examines seven SOA objectives from Thomas Erl’s book, Principles of Service Design:

1. Increased Intrinsic Interoperability

   Made better with a uniform interface – Interoperability stops being an “easy to plug together at design time” thing to “easy to plug together at run-time”

2. Increased Federation

   This is not an explicit target of REST, to get everyone using the same service. However, it is compatible with REST. It also occurs in practice on the Web through market forces.

   While SOA emphasises reuse of services, REST emphasises reuse of components (including services), media types, and connectors.

3. Increased Vendor Diversity Options

   More or less the same for SOA and REST, however REST is probably still less mature in the enterprise space. Time will change this.

4. Increased Business and Technology Alignment

   ie, we build services that the business wants, and can change in pace with business objectives

   Again REST doesn’t particularly target specific valuable services, but is compatible with the notion

   REST is more capable of incremental upgrade than a classical SOA, especially “dynamic evolution”.

5. Increased ROI

   Classically realised in SOA by reuse of services, which again is compatible with REST

   Again, a uniform interface produces wins on this front when compared to classical SOA

6. Increased Organizational Agility

   Classically realised by being able to build new applications quickly by reusing existing services

   Again, compatible with REST and again a uniform interface produces improved performance

7. Reduced IT Burden

   Few legacy systems doing more means less maintenance and related support activities – again compatible with REST.

   REST makes maintenance easier with its focus on dynamic evolvability

Darrel Miller has obviously been through some maintenance pain. The web architecture was always designed to not have centralization, and that obviously mean that things are incrementally upgradeable.

For me the business case for REST is around client/server coupling and maintenance effort. My experiences with SOAP API’s on the Microsoft platform were that if you sneezed on the server API that defined the contract you had to recompile and redeploy the clients. And then there is versioning. In the WCF world versioning is a potential nightmare, with changing URLs, XML namespaces on service contracts, message contracts and data contracts.

I believe that REST’s use of hypermedia and conneg make maintenance, deployment and versioning much easier, which translates to lower costs to the business in the maintenance phase of an application.

Bill Burke responded to Darrel saying how he has been using it.

Also, I think interoperability is much easier to achieve with REST because the ubiquity of HTTP. This has a huge positive side-effect for integration as well. My gut tells me that SOA Governance could really reap the benefits of a RESTful architecture as well, but I don’t have any details yet on how.

All this plus Darrel’s comment on the fragility of SOAP stacks sold me on the idea of REST which is why I’ve been really pushing it hard at Red Hat and all our middleware projects.

A very useful set of responses, thanks to everyone. Will be writing more on this soon.

I went to a meeting a while back with a company starting to move to a web service based, internal API based architecture, and there was a minute where the CTO said (more or less) “does anyone know of any let or hindrance to this being a SOAP API?”. Like those moments at weddings, no one spoke out. But I do object. SOAP is not the backbone of a happy architecture, but we do not have the strength to say no right now.

So here is the beginning of the shout out. The enterprise architecture needs to be a resource oriented architecture (ROA) not a service oriented architecture (SOA). The enterprise needs to move from SOAP to REST.

Web developers know this. When given the choice, 9 out of 20 cats prefer to use REST. It is more productive. It does not involve programs to generate huge chunks of useless code. It involves hypertext (HATEOS) not opaque documents that are mappings of database schemas.

What are the resources in the enterprise? Start with customers. A CRM system is a clear example of something that needs to be modeled as resources. You need API calls that can find products a customer has bought, support tickets, all the core data that you would need to build customer service portals, internal support applications. What is your customer API?

Building this framework can be incremental. You need either tools that already provide REST APIs, which is becoming easier, or a web application development framework. This is the convergence point between application frameworks and web content management. The big issues are the domination of legacy applications with bolt on APIs that may have moved from SQL to CORBA to SOAP over the years. You can however build a REST API over at least parts of these applications to open them up into the enterprise.

OK thats the beginning. Apart from cheaper application development where is the real value?

Resources are the first part of the REST architecture, and resources are much easier to work with than services, as they share uniform semantics, and they are addressable, two keys to making application design simple. The big bit though is the (harder to understand) HATEOS, Hypertext as the Engine of Application State. What this involves is moving the vague and very expensive field of business logic from code (often code that is not even owned or understood by the enterprise, as it has been embodied into code written into systems from suppliers) into hypertext documents.

A concrete example might help here, based on another recent consulting example. A professional body has a set of different membership levels, exams, rules, CPD requirements and so on. These are resources, described states, and hypertext links of state transitions. This is the core of the business logic of the organization, embodied in documents. It can be used to build the membership application, as it describes for example the actions open to a member at the present time, as well as providing potentially a browsable structure, as well as a formal structure that can be used to ask questions about membership (what are the routes to becoming X, for example).

You can view this as a move of business logic to a declarative rather than imperative form. Things do not have to be stored in documents in the underlying storage, although the interface is document and hypertext based. Hypertext makes things human browsable, and declarative makes them computer browsable, and HATEOS makes them discoverable. Business logic becomes content rather than data, rather than there being tables of parameters that the business logic black box feeds off, the states themselves become resources that can be discovered, addressed, and reasoned with.

As states become resources in a REST model, because of the statelessness of web applications, both states and state transitions become first class objects. Constructing ad-hoc queries about state changes should become easy (what percentage of customers renew their contracts, say). The first class objects need to be the ones that are meaningful for the business.

There is more to this, it needs expanding. Perhaps we need a manifesto. The software architecture of the web is going to make huge changes to the architecture of other realms, more than most people realize it seems. It is inevitable as things get re-architected around the web that more areas will be affected; also there are the benefits of scalability and reliability that are being created for the web. If anyone has any other business case references for REST please post in comments.

(Image from this software architecture overview).