Standards

• repository
• feeds
• CMIS
• JCR
• terminology, ways of thinking, industry model

Standardization has really started to affect the content management industry. The industry was very immature, a bit of a landgrab, and not very customer focussed. This has changed rapidly, with the wide adoption of the JCR standards, but particularly with process around CMIS. What is being set now is the model of the industry for the next five years, what the customers expect and what the products will deliver. Setting the agenda matters, and now is the opportunity to participate.

CMS as a platform

• build applications on a content platform
• API driven development
• SOA
• embed code everywhere in domain level scripting languages

A content management system is at last becoming less of a product that lets you do some stuff and more of a platform for working with content and building content centered applications and a service oriented world. Pervasive invasion of scripting languages such as Javascript into this is coming. The web programming model of pervasive agile scripting and rich REST APIs is going to be the norm, not large scale Java programming or application specific templating languages.

Co-opetition and community

• collaboration on standards, infrastructure
• open source as community
• twitter, blogs, enterprise 2.0
• end of NIH
• customers are community too

In the last year especially the landscape of content management as a community has changed. First through the standards processes, particularly CMIS and JCR, and then through social media, particularly twitter, as well as via events and blogs, there is now a growing cross vendor technical content management community, particularly with the open source players, and joint projects, for example with CMIS. This is in addition to the developer communities that are strongest around the open source products, although the .net products are trying hard to build around the Microsoft developer relations model. And of course the community of customers, who are becoming more vocal.

Rich content

• richer xhtml and xml
• enhanced metadata; richer metadata in other formats
• constraints not just validation
• RDF and semantic web, linked data
• relations and IA expressed in metadata
• enhancement via deeply integrated search
• document management, DAM and WCM converge
• richer presentation layers, richer APIs
• Flash is dead, plugins are dead, HTML5 is winning faster than anyone thought

As we have moved from document management, where the focus was on whole documents, to web content management, which is more component and assembly based, there has been a gradual push to do more with the documents. Standardized rich document semantics are after all one of the main advantages of web documents. It is taking a while but making use of the potential here is beginning to happen, now we have Google indexing rich snippets and even Tesco using RDFa. There is a lot more standardization work to do here.

In the front end the aim of this backend information enhancement is to build richer interfaces more easily, and to enhance findability, search and navigation, as well as to enable repurposing, richer APIs, and linked data. Authoring is the biggest challenge, as the majority of users need to be given interfaces that are independent of the IA, simple to use, but support generation and modification of complex data structures.

SAAS and the service business

• cloud
• internal delivery in a SAAS way
• devops
• APIs and standardization forced by SAAS
• changes to customer service model

Software as a service models are winning because no one wants to buy software as a product any more. I will cover more of this in another article I have been working on for a bit, but the main point is that enterprise software is a paid big ticket product is dead. The replacements are open source software and SAAS. These are not alternatives though, as people want the open source software delivered as a service, albeit maybe a more commoditized one if there are multiple providers, and many of the SAAS products delivered will be largely built of open source components by companies that run a mixed model. Microsoft is going headlong into cloud in a way that redefines what the operating system is. Even purchased software will be delivered in internal clouds.

This changes both how code as written and administered, with the web operations joining up into rolling delivery and creating the emerging field of devops. Developers need to understand operations and how to build code for this environment.

The service business as a business is different from the product business. Open source companies have got that better than product based vendors, but the less there is lockin the more key these changes become. The success of the customer using the services becomes the key business driver.

Performance and scaling, real time

• cloud has pushed scale up out of picture
• scale out transparently
• new technologies beyond RDBMS that fit CMS
• dynamic generation becoming the norm; Google pushing the performance thing; the industry norm of 100ms will fall
• real time becomes more important – dynamic updates, forget crawling, Google is going push
• backend: queuing (0MQ, AMQP)
• frontend: websockets, XMPP, long polling

Just buying big hardware for scale up is really becoming difficult; the web vibe has always been to scale horizontally on commodity hardware. There is a lot of development around scale out technologies such as NoSQL which fit into the WCM data models, which are those of the web after all.

As well as scaling for volume, latency and real time are becoming key. Google’s time to crawl has been falling rapidly, to a few or less, but is moving to real time with push updates. Twitter has really pushed the boundaries of expectation for real time. Behind the scenes there are a lot of technologies for efficiently pushing around notifications and events both at the backend and on the frontend. Real time is going to become increasingly pervasive.

Page generation times will need to fall; the standard industry benchmark of 100ms per component will probably need to be halved; overall total times under 1s will become the norm.

Security

• web increasingly hostile
• every bug is a potential security issue
• security focussed on fewer areas, push into the OS not out to applications

I read the excellent Linux Weekly News every week, and every week there are security exploits for many pieces of software; one that really struck me recently was the major exploit against the CMS e107. What happened here was the a group of crackers found a serious security flaw in the CMS, which they began attacking systematically. When the patch was released however, they already had control of the developer’s website via the flaw, so they replaced the patched version of the code with a version with a backdoor. Hacked websites are a vital part of the underground online crime scene, and a content management system is a high value target. Expect much more of this, and be prepared.

Narrowing the security into fewer points of vulnerability, sandboxing, using every available facet of the operating system’s security layers; make the most of processes, permissions, everything that you get there; I wrote more about this in an earlier post on emerging trends. File format parsing is another area of vulnerability that is common.

It is war out there on the internet, and many people underestimate or ignore the issues, and too many programmers do not code defensively by habit.

Summary

It is an exciting time in web content management right now; the industry is growing up beyond its beginnings as a way of getting web sites up, towards being the core of the broader content management industry. The choices made now will shape the industry; the next generation of products will be a big step forward forr the industry.

The data model for content management

I have another draft post on this subject in more detail, which I am working on as parrt of my REST modelling in content management work, but I will outline some of the types of data relations that are important. I will be quite abstract here, if you want more concrete examples you will have to wait for the other post: database models like the ones we are talking about here are more easily understood in the abstract I think.

First we our unit of modeling. This in itself is the first issue. Content management tends to deal with, at the conceptual level, something that looks like a document. It may be a fragment, in the sense that it is say a page component (asset if you use that terminology) rather than a whole item, but the unit for the user to edit and which is usually versioned is a structured object itself. The processing model tends to treat it as almost of binary blob, except that certain properties can be extracted, such as metadata, links in HTML and so forth, but it is stored as an item rather than decomposed further.

OK, so we have a piece of content and some attributes extracted from it as one basic model. This corresponds pretty much to the JCR data model for example. There are variations; sometimes people do not store metadata in the file formats, as historically many file formats had poor support for arbitrary structured metadata, although that is largely obsolete now, and the advantages of actually storing metadata and relations substantially within documents are high. External storage does not change the model much, just complicates processing and storage. Another variant, often seen in document management systems is to be able to have multiple ’streams’ ie several document variants rolled into one, for example a video and a still from it. You can however from the modelling point of view regard these as anotehr compound document format kept together because conceptually they are a bundle of content; you might distribute them as a zip file if you havent got any other suitable container format.

So now we have a storage model where we have a blob, with rich media operations on it, and extracted structural and metadata information. There is also versioning to consider, but let us ignore that and treat it either as part of the blob, or as a new document with some relation to the old ones, those being the two core versioning models, this does not really affect anything else.

There are two kinds of metadata, although they are more similar than they appear, properties and relations. Properties are the standard attributes (this picture depicts sheep), while relations join two items in the repository (this is a cropped version of this other picture). Although this distinction seems clear, in the end richer information architectures demand that everything becomes a relation, so I can browse a sheep node and find all the sheep items, turning every attribute value of any significance into a node with relations instead. Pure attribute values are only left for the less interesting properties (this PDF file is 176k in size).

They are also less interesting from a relational versus non relational storage point of view, although there is one important point, which is the dense versus sparse question, so let us take a look at this. Most real world attributes are sparse, that is most attributes aare not set on most items. In the relational model we have a row for our item, and columns for all the attributes, so we are saying most are NULL. (I was brought up on matrix algorithms and still think in terms of sparse versus dense matrices as this is exactly the same problem, and matrices represent graphs anyway). Storing huge mainly null tables is not very efficient, so there are two common practices in relational mapping of attributes in content management systems. First is to define a type based system, where a particular type of content item is defined to have certain attributes (or at least fewer NULLs!), and each set of that type therefore can have its own table which is assumed to have fewer NULL values. Mixins, sets of properties that live across types can potentially be added to this model, as can inheritance schemes, but the basic idea is one table per type. This gives a nice simple direct database programming model, and causes a complete nightmare if you ever want to change the schema, for example add an attribute, as for any large database most DBMSs will effectively shutdown the system while a schema change takes place, as schema changes require pretty much all locks. Silverstripe is one example of a content management system built like this; there are many others.

The alternative is the entity attribute value (EAV) model (terrible Wikipedia article, please fix), where rather than a direct mapping of the attributes to relations, you indirectly map, creating a table that joins entites, attributes and values; this table of course looks just like RDF triples. Doing this though loses everything that makes a relational database useful: constraints, typing, query optimization. It adds an extra layer of logical schema above the physical schema which the database layer does not understand. This is a pretty common relational mapping for content management systems, as it allows full flexibility in defining and redefining attributes. To implement well it needs a large mid layer to manage the constraints, provide an API layer, generate efficient queries, effectively to manage the logical layer to physical layer map. The Drupal CCK is an example of this model.

Of course this is not to say that neither of the two relational models do not work. The direct mapping works well with simple, unchanging content types in small websites, for example, or in models where attributes are not very sparse, or the sparseness is worth the overhead, and changing the schema is rare. EAV works well too, if managed carefully; it helps if the type of queries required on the model are not too complex.

Once you add relations as well as attributes, the already difficult mapping layer gets harder; you add another set of operations (recursion to handle tree structures) that the relational model does not handle well, so you may need to add more into the mapping layer. The promise of NoSQL is that you can bypass this for these types of applications, and program directly to a database model that handles sparse attributes and relations natively. But how much do the NoSQL databases get you? You can argue that if you are already looking at EAV, then you are already not getting much from a relational database, and you are building a modeling layer on top of it, so dropping that and going for something that maps the logical data layer directly does make sense from a development point of view. Whether that really helps performance is less clear; much of the original work for NoSQL has come out of huge scaling, big problems, not actually providing efficient solutions to the types of data mapping problem we are seeing here on a medium scale; of course for huge sites there may be benefits.

The types of NoSQL database vary in their level of support for attributes and relations as they are used in content management. Document oriented databases do not give you much more than retrieval of content items; associative ones give key value type attribute lookups; graph databases should let you query relations directly, expressing the types of queries that are needed for information architecture problems directly, in principle. Examples I am thinking of are things like tag clouds, which is simple to express as a graph problem as it is simple a count of the number of edges from a set of nodes. Indeed most information architecture problems look like graph problems, and also like OLAP processing operations which also do not work well on relational databases. And of course one of the things that NoSQL has shared with OLAP is the use of denormalization; you can use simpler models if you denormalize data to match the queries you will be using, rather than assuming that the types of query you will use can necessarily be optimized and made efficient by a general purpose system.

Denormalization is not without its difficulties, although arguably it could become a tool embedded in databases like indexes are now. One of the issues with NoSQL is most of the database systems leave denormalization to the user: you need to use it because joins are not available, but you have to manage that yourself. Building an infrastructure to explicitly manage denormalization as a first class database item akin to an index might be interesting. So that gives us a first issue, as in any NoSQL system except a graph database we will either need to denormalize or compose queries to get the results we want.

So I think there are four realistic models for content management backends going forward:

1. The direct relational model for small systems with simple data models, rare attribute changes, little or no use of relations.
2. EAV models wrapped in a content modeling layer; JCR is an example of this, hiding the underlying SQL layer very well, and indeed allowing it to be replaced with another underlying storage model potentially; I am sure someone is testing a Neo4J backend somewhere. This is where most production solutions are at now.
3. Direct, nondenormalized graph database backends, with the raw content stored in a document store. Cuts out a special purpose middle level by mapping the domain more directly. As Emil Neo says, it may not scale right up as far as the othe NoSQL technologies, but it cuts complexity of implementation; there are also issues about whether all the kinds of queries required are available efficiently. I think this will be the sweet spot in a few years once the products mature and we see more open source activity in the field. Of course RDF based solutions, for example using SPARQL fall into this category too, and the maturity of products around these technologies will help drive this category as well as the NoSQL models.
4. Big, denormalized systems, probably with software support for managing the denormalization, and using underlying simple but scalable technologies like key-value stores. These already exist in large scale web applications, but may remain niche if the development effort remains high. If frameworks for modelling more easily on these turn up they may trickle down for performance reasons even on smaller datasets; a key value store runs fine on a relational database backend, although the types of processing required probably means a specialized backend is useful.

Note that the Lily CMS which there was a talk about fits very much into the fourth option above; this is where the NoSQL technologies have perhaps seen most use, but I think there will be a lot of work in order to build a CMS like this now, in particular in terms of tools to support denormalization strategies that are needed. The outlined approach sounded much like the outlines I have been thinking about for this type of model, although I would focus more on tooling for denormalized queries and less on scaling other parts like full text search right now. It will be interesting to follow the progress of this project.

We are at an interesting juncture, where it looks like there are some options that will let us do domain modelling in a way that corresponds more directly to the domain, but there are a lot of interesting challenges on the way.

My argument was mainly that storing flat, unstructured data was not enough for most content projects, and the difficult questions of structure needed to be addressed. Daniel’s third part addresses how they do this. Actually when I first looked through the NPR content after reading the first two articles I could not find any content that had inline HTML, clearly was an accident should have read more, as it is used.

The actual NPR process is interesting. In particular it shows the amount of care and attention in curating content that is needed to keep it reusable, repurposable and valuable. Semantic content requires you understand the range of meanings that it encodes and how to work with them, and transform them, and quality control them. And that means you need to know every tag and attribute that is going into the system and what that means for every output you are or may using.

One of the outputs that many people forget is plain text, and NPR is very clear on the writing style that is necessary for writing for HTML and plain text. Everything must make sense in the text form; links should be additional information that adds to the text not necessary to understand it. And of course no “click here”. Text output for other devices may vary between the expressiveness of HTML and that of plain text. Text that reads sanely also helps screen readers and other assistive technologies make the content understandable.

The key points here are that content markup must be

1. Valid. Processing is likely to be inaccurate without valid content, and tools will be more limited in how the process it, or will fail unexpectedly. Best fix this at the beginning of the pipeline.
2. Meaningful. You need the markup to mean what the author intended, so look at interface usability and training.
3. Accepted. You do not have to accept all valid XHTML, say. For a start, XML is an extensible language! You can choose a whole range of markup for a story, from the very minimal, to marking up each person and place involved, or more.
4. Stored. The marked up text must be stored; the NPR decomposition is plain text plus plus normalized markup which may work for some systems; storing marked up HTML without output transforms may work out better for others.
5. Processed. You need to handle each kind of markup for all output mechanisms, so they need to be introduced in a controlled way, although this should not be difficult. Changing markup is something that may need to happen.

I think the third part is the interesting one. Information architecture with websites often stops at the content level, missing out on this information microarchitecture of the textual content itself, leaving this to authors without enough guidance to build a consistent structure to maximise the long term content value.

Other links are here: Jon Marks, Irina Guseva, Ian Truscott; other participants Adriaan Bloem, Andrew Liles, Philippe Parker (first use of Wave over GotoMeeting?)

Well it was fun. Technical difficulties, lost sync and crashed a few browsers, some people lost whole machines though. Safari coped better than Firefox. It took a while to realize what was happening here, hey but this is in beta!

As a brainstorming tool at worked pretty well. I thought it scaled pretty well. The named cursors indicate who is in the bit you are in, but for brainstorming you can look, write another point, move, continue, not edit much. After half an hour of getting to bulletted lists, a bit of moving around the heavy writing started (after a discussion at the top in our proxy process section; we should have split the thing up a bit).

There is a great tendancy to write temporary notes about the discussion and then just delete them. Which feels odd, data and metadata together of course. The editing process was odd, you would find orphaned bits, move things, try to join stuff up to make it flow, while it was all changing around you. Pretty chaotic. Bits that no one expanded into prose got junked (quite a good edit method, as they couldnt stand up themselves).

Here is the “finished” article… which cannot be attributed to anyone individually of course… the subject was chosen about 10 minutes in, just as something people would have something they could easily contribute into this situation, there are some good points in there though!

Things We Hate About Content Management

- By The Motley Crew

It was a lovely Friday morning/afternoon, and we were Waving. The experiment initiated by McBoof (yes, that one) brought together 6 CMS folks from around the world. The event gathered together analysts, journalists, vendors, system integrators to Wave on a topic that was decided at that very moment. We had one hour (in between conference calls and other job thingys) to pick a topic and Wave it.

A little collab on what exactly to Wave about later, we decided to do “a mindmap of things we find annoying in CMSs.” To up the ante, we also decided to take the original bullet points (deemed “too easy”) and convert the whole thing to prose. Was the tool given really up to the task? Were our minds flexible enough to wrap around this kind of realtime collaboration?

In the beginning — we blame the tool — we were Drowning, not Waving. We (almost) didn’t fight about edits. We almost didn’t step on each other’s toes. All in all, it turned out to be a fun and productive collaborative exercise. Read on to see for yourself.

Cosmetic Issues

There really should be a CMS UI fashion police. As there should be a Magic Quadrant for shoes and handbags. Why? Well, there’s a couple of issues.

For instance, sloppy, non-designed design. You know the kind of thing that has not been thought about and reworked and made to feel right. The sort of thing coders do if you don’t force them. But at the same time, over-designed interfaces can be just as bad: the designers and developers really need to be on speaking terms.

When building a system that works, you can’t have the development team in the basement on a sustenance of Jolt coding away into the night, and the designers in the penthouse in turtleneck sweaters sipping espressos. Too many CMS designs end up being programmer vs. end-user friendly. And this is not the best way to charm away those marketing and web content folks.

Developers and designers need to talk to each other and essentially, both should talk to users – not just eat your own dogfood – but listen to what dogs like to eat. A developer or UI designer are not content editors, marketers or knowledge and information workers.

Some vendors say that the agonizingly and depressingly black UI backgrounds are hip and modern. Well, they are not, really. Who told you that? Especially if you add a Star Trek theme to it and sprinkle in some stars and cosmic swirls, because if Apple does it, it must be cool right? Not pointing any fingers, but I would quit if I were a content manager having to spend my 9-5 staring into the “black hole” of some of the CMS UIs that are out there on the market.

Even pop-ups seem less annoying when compared to dark UIs. Which brings us onto…

Interface Issues

Interfaces need a comfortable lived in feel. Content management is something people work with every day, it is their interface to their job. You meet people who hate the interface, and that makes their work a heap of pain. I have seen people who describe the 44 clicks it takes to insert an image. You have a responsibility to these people, to make them love the content and make the tool disappear.

We all hate it when the interface does something on its own that ruins your context. E.g. a page refresh, or in Wave the jumping around of the scrolled window in some cases Or the lack of an easy way to bookmark, so you can reference someone to the content. Remember people will be collaborating and need to send links around. Make sure the UI is a proper web application with URLs. And why do tasks that are easy to describe and often repeated in exactly the same way still take more than a few clicks? (Or maybe even dozens of clicks.) With bonus points for forcing users to use dialogs or tabs to enter mandatory information. Remember people do not have all the information in the right order.

Also, we need sane conflict merges. Check in and check out is too extreme for most uses. But people want to edit offline still. Of course Wave doesn’t have an offline: Google thinks this problem is going away, it’s real time so there are never conflicts (that’s defined in the XML protocol; it’s quite interesting if you are that way geeky). Does Google have the right answer here? Well, the Motley Crew is struggling here, and some browsers lost sync during this experiment.

“Power users” (those who use it all day long) of CMSs needed to have a “Desktop” experience. What does Desktop Experience mean? Well, it doesn’t really have to be on the desktop — these days it is perfectly possible to get very close to a hitherto Desktop experience in a browser or similar. these are qualities: very low latency from action to response, no page refreshes, modal and modal-less dialog boxes as appropriate, “push” notification.

Architectural Issues

Architectural issues of the wave overtook any architectural issues of Content Management Systems. The fact that we authored this entire article in a single blip didn’t help, and slowed everything down enormously. McBoof learned the hard way that he really need a new laptop and spent most of the session giving his machine CPR. Next time we’ll do each paragraph in its own blip to stop FireFox going down like a Led Zeppelin.

Monolithic systems. Build it out of pieces that the client can not use all of. Obviously your pieces may work together better, but there should be components. Do not try to reinvent all kinds of wheel. “Best of breed,” though, is just another weasel marketing idea, as if systems are pinnacles not about meeting requirements.

Marketeers are adroit at using the term Best Practice to position Their Way as the only way that a particular matter can be solved. (Many of us live in that netherland of having to pedal that point of view, but it is a falsehood that the careful buyer should try to see through.) I think this devalues genuine best practice, vendors should cite references

Most often a marketeer’s Best Practice view is the only one they subscribe to as their product development has paddled up the wrong stream and cannot or won’t reverse their architectural design (probably because of the cost of doing so). This intransigence most often causes a product to doom itself. (Think of IBM and The Mainframe Is The Only Way To Do Serious Business).

Who really still believes that there is a place in this world for Flash or Java Applet based Rich Text Editors? TinyMCE, FCKeditor and others are filling the gap left by Ektron when they bit the hand that feeds and entered the CMS market. Ephox is trying to spread, but I find it difficult to come up with an excuse to use an Applet over HTML with javascript these days. Stick with the standard.

Business Issues

Where you are buying into something that you may very well need to change or integrate with there is strong benefit in considering Open Source. Open Source used to frighten commercial software companies but we have come along way on that road to understand that commercial organisation can operate in an Open Source world and benefit. This does not necessarily mean that their prized system needs to be fully opened up, but taking the spirit of it to mean that you are completely open to people seeing and learning from your code how it operates.

Exactly what you need to see opened up varies. In a CMS there may be a subsystem that stores the content or one that allows a Rich Text Editor. These arguably don’t need to be opened up, but when a CMS ships with modules for, for example, an RSS feed widget, calendaring tool, prebuilt webforms, users who then want a variation on this module can benefit from seeing how the “pros” did it, they can then use it as a starting point for their own different implementation.

We really don’t need vendors that pay lip service to the buzzwords. When they think the new CMS buzzword “engagement” is just a screenshot of Google Analytics. Or when they add an image picker and call it DAM. And a cross-over between WCM and ECM? Don’t think WCM is like ECM and it’s about organizing content, not about effectively communicating with the audience. And don’t think that if you organize the content, you can automatically communicate effectively.

Completely different, but equally frustrating, is procurement (and the procedures that go with it.) Procurement folk don’t recognise the importance of user adoption to the success of the project — of the black background and all the UI issues pointed out previously. If a CMS is procured according to procedure, the selection is a success to them. But those same rules are often a recipe for ignoring what the users really need.

At the same time, budgets that aren’t transparent are an issue – customer and vendor should be able to have a sensible grown up conversation. As a customer, of course you want good value, but how cheap are you? But to vendors: many licensing models don’t make any sense, and force you to do stupid things. People are scared to have that conversation – the best architectural fit first I say, lets figure out an appropriate license around that.

Conclusion

So much hatred rolled up into a tight little ball of anti-CMS rage. Who would have expected it from such a respected bunch of CMS folk. We hate the designs, the interfaces, the architectures and the business. Time for a beer/wine? Wave good bye!

It started with an idea I had, in an innocent sort of way. I thought if I looked at the JCR specs for a bit I might find some kind of way of building a non Java interface with them. You know, maybe there might be a nice REST architecture waiting to get out. But of course there is no such thing. It is an application definition. There are not even that many ways of implementing it, other than choosing your object persistence method to be a database, file, or something else.

The REST architecture is notionally provided by another layer, such as Apache Sling, but Sling is in no way a REST layer, it is a URL dispatcher and scripting and application layer with which some REST style applications can be developed. With that you end up with a pretty heavyweight development framework, indeed together you have much of Day’s CMS offering in effect, rather than a lightweight REST repository solution.

I had a look at CMIS again. Fielding once laid into CMIS for not being REST and you can see why, although some improvements have been made since that. Although resources are discoverable through hypertext, there is a fair amount of semantics that needs to be known to understand what a type or a checkout means, and the search queries are obviously just RPC wrappers. It is not too bad though, but unfortunately the data model does not map well onto web content management right now for obvious historical document management reasons. Fixable? I think it serves a part‎icular purpose well and should probably not be forced into anything else, as we need it to succeed in its field.

Day claims that JCR is not a Java standard in an odd way, that you can implement the API in another language. Thats a strange argument to make, especially as the types are defined as Java types, and standards without interoperability are pretty vague. Without some sort of wire format or ABI this is meaningless outside the JVM world. People are making JCR like repositories in PHP but outside any standards process, so in the end this just becomes a PHP repository project; Typo3 seems to be building another, also closely aligned to JCR.

The problem with these efforts is that it is not helping the balkanization of web CMS, which is already fragmented by language and API, which is ridiculous in an industry that is about the web. The web has an architecture (REST) and an API (HTTP). Building web content management on Java APIs or PHP APIs or .NET is a legacy way of thinking; it is acceptable for document management given its role in existing enterprise architectures, but it is not going to work if we want to get widespread acceptance in web development; in the short term it is the easy path, it is what people are used to, but a forward thinking industry needs to look at defragmenting the landscape and building future proof tools.

The odd thing is that a web content repository alone surely lends itself to a simple REST architecture. Content is after all lots of small resources with relations. Hypertext. It is pretty much in presentation a fairly dumb web application, although with a fair amount going on behind the scenes. It takes content, relates it to other content, and serves it back, with authentication and versioning. Everything else is in other system layers, transforming it and so on. Not simple, but well defined; lower level than JCR + Sling say

So we need to work on a web content repository model, as a community. Process wise, it makes sense for this to sit in an organization like AIIM, as a content management based industry body. It may well be that what ends up coming out of this is more standardized architectures and semantics and open source implementations rather than the tighter prescriptions of JCR and CMIS; I have some ideas along these lines that I need to code up. I have had some discussions and there is a degree of interest in some sort of solution; who is interested? Or is infrastructure dead, everything ust wants interfaces?

“Spent a week in a dusty library waiting for some words to jump at me” Camera Obscura.

Kas Thomas in his contribution to Julian Wraith’s popular thread on the future of content management really managed to make me disagree, the first of the posts that has!

The theme of this blog (yes it has got one) is that content management is changing as the web way of working starts to infiltrate the enterprise. And the web way of metadata is not what the old document oriented way was.

Kas says “Keeping knowledge about a file separate from the file itself is a hugely important concept.”

Looking at that point first. Look at any newish document format, from PDF/A to EXIF via the HTML meta tag, even a Word document and you will find embedded metadata. Remember that documents are emailed around, generally get lost. Metadata is the dog collar with a name and phone number on, saying version me and send me home. This trend is not going away, documents are becoming self contained.

Kas then continues “A file’s metadata becomes its interface to the outside world. It’s like a service descriptor.” That is simply not the way the web works. Resources use self describing formats like HTML for core data, and then all the other important metadata is linking information. It used to be that you had a blob file type and a program that could understand it, that was the basis for the desktop architecture, but that is not the case any more, even on the desktop you have a choice of applications that can understand a given file type, and your choice depends on what you can do with them. The web architecture goes much further, and resources become fully self-describing, a browser can understand all the web as every resource carries its own description, and bundled code to help you interpret it. A web page is its own service descriptor, and defines application state through hyperlinks. The web architecture has never had service descriptions.

There is one vital part of metadata that is not kept with a file, that is the link. Kas says “content, on the whole, is becoming richer, less structured”, missing out completely on the big picture that content is being structured by the imposition of links onto it, by its transformation into a hypertext, that creates a much richer structure than the individual documents have in themselves. Documents contain metadata about other documents in the form of links. The semantic web project is an attempt to add further richness to this structure. “What does the trend toward richer, less structured content mean for management of content?” well that means that content management is going to be about managing those links and relations between items, a lot more than it is now, when it came from a background of just managing documents, each an isolated item.

In a way that does come back to “Keeping knowledge about a file separate from the file itself” but not at all in the way Kas was trying to argue. Now time to link this to his argument to create a structured discussion…

6f82f1d2683dc522545efe863e5d2b73

Stéphane Croisier in his post in the thread says

There is currently an unclear separation between applications frameworks and content infrastructure. But at the end of the day everything is content and every application has first to deal with content items rather than with processes, states, UI components or other application oriented paradigms.

In my general work in content management I think this is one of the things that has become very clear, and that “unclear separation” is very apparent. First content; it is a very good start, and every project needs to be grounded in content, and in the structure of content, the architecture of content and the user IA. However, the processes, states, UI and other parts of the web application are beginning to dominate projects. In the Gartner terminology we are building CEVAs (Content Enabled Vertical Applications that is), as integration, process, e-commerce, CRM parts of the project start to dominate the requirements over the purely content based parts.

It seems that most of the contributors to Julian Wraith’s future of content management thread who mention it see content management moving to a clear split between repositories (Common Content Information Infrastructure as Stéphane calls them) and applications and content management systems and CEVAs implemented on top of these.

I don’t think we can yet see what the successful content infrastructure stack will be; as I said in my earlier post there are technical decisions that have to be made that there is not yet agreement on (except between me and Peter Monks!) and the existing putative standards (CMIS and JCR) do not extend far enough to take a position on. But we can see that this is the way things are going. Quite clearly the standards for the infrastructure will be open, and most implmentations will be open source. There will be some vendors who do not embrace standards, but they will need to be the few large ones or they will lose out. Infrastructure environments remember (think Linux, Apache) are mainly open source, although there is scope for proprietary layers at the very high end (think Amazon, Google).

At the application layer, as Stéphane says, everything is a mashup, content from different systems, content from other APIs, this is the we application layer. It needs to be content aware, very much so, but it needs to be an application development environment. This is where most people will see the value added in the content management business, although in fact the value here is in implementation, design and integration services, not the technology itself. Application development environments no longer make a lot of money, and again they are dominated by open source (think Java, Eclipse, JBoss, Django).

Once you take out content infrastructure and application development, and the other tools like search, workflow, there is a core of tools for working with content, to support reuse, refactoring, cleaning, import and export, that one might call a Content Workbench. There is a lot of potential value if these types of tools are the value added end of the business, as they can differentiate vendors and add value. Interfaces for merging changes and so on would be part of this type of toolkit. This is the stuff where good UX means timesaving for content workers, but it is difficult to build on a customized per-project basis, so this still offers value from a particular vendor.

Overall then we see a picture where the monolithic CMS starts to break apart into infrastructure, application and toolkit layers, that can perhaps gradually be mixed and matched together to build content applications. We are just seeing the beginnings of this now.

6f82f1d2683dc522545efe863e5d2b73

I went to a meeting a while back with a company starting to move to a web service based, internal API based architecture, and there was a minute where the CTO said (more or less) “does anyone know of any let or hindrance to this being a SOAP API?”. Like those moments at weddings, no one spoke out. But I do object. SOAP is not the backbone of a happy architecture, but we do not have the strength to say no right now.

So here is the beginning of the shout out. The enterprise architecture needs to be a resource oriented architecture (ROA) not a service oriented architecture (SOA). The enterprise needs to move from SOAP to REST.

Web developers know this. When given the choice, 9 out of 20 cats prefer to use REST. It is more productive. It does not involve programs to generate huge chunks of useless code. It involves hypertext (HATEOS) not opaque documents that are mappings of database schemas.

What are the resources in the enterprise? Start with customers. A CRM system is a clear example of something that needs to be modeled as resources. You need API calls that can find products a customer has bought, support tickets, all the core data that you would need to build customer service portals, internal support applications. What is your customer API?

Building this framework can be incremental. You need either tools that already provide REST APIs, which is becoming easier, or a web application development framework. This is the convergence point between application frameworks and web content management. The big issues are the domination of legacy applications with bolt on APIs that may have moved from SQL to CORBA to SOAP over the years. You can however build a REST API over at least parts of these applications to open them up into the enterprise.

OK thats the beginning. Apart from cheaper application development where is the real value?

Resources are the first part of the REST architecture, and resources are much easier to work with than services, as they share uniform semantics, and they are addressable, two keys to making application design simple. The big bit though is the (harder to understand) HATEOS, Hypertext as the Engine of Application State. What this involves is moving the vague and very expensive field of business logic from code (often code that is not even owned or understood by the enterprise, as it has been embodied into code written into systems from suppliers) into hypertext documents.

A concrete example might help here, based on another recent consulting example. A professional body has a set of different membership levels, exams, rules, CPD requirements and so on. These are resources, described states, and hypertext links of state transitions. This is the core of the business logic of the organization, embodied in documents. It can be used to build the membership application, as it describes for example the actions open to a member at the present time, as well as providing potentially a browsable structure, as well as a formal structure that can be used to ask questions about membership (what are the routes to becoming X, for example).

You can view this as a move of business logic to a declarative rather than imperative form. Things do not have to be stored in documents in the underlying storage, although the interface is document and hypertext based. Hypertext makes things human browsable, and declarative makes them computer browsable, and HATEOS makes them discoverable. Business logic becomes content rather than data, rather than there being tables of parameters that the business logic black box feeds off, the states themselves become resources that can be discovered, addressed, and reasoned with.

As states become resources in a REST model, because of the statelessness of web applications, both states and state transitions become first class objects. Constructing ad-hoc queries about state changes should become easy (what percentage of customers renew their contracts, say). The first class objects need to be the ones that are meaningful for the business.

There is more to this, it needs expanding. Perhaps we need a manifesto. The software architecture of the web is going to make huge changes to the architecture of other realms, more than most people realize it seems. It is inevitable as things get re-architected around the web that more areas will be affected; also there are the benefits of scalability and reliability that are being created for the web. If anyone has any other business case references for REST please post in comments.

(Image from this software architecture overview).

Although the vast majority (but not all) of open source content management systems are continually trying to reinvent the blog, we are talking about internet infrastructure here, and the future of content is going to be open source, like the rest of internet development. I also believe that long term the web project will overwhelm the legacy areas of document management, although it may take some time. Hypertext, the web architecture, XML, HTML, and all those standards are here to stay and to dominate long term. Content management will also become pervasive long term, as the blogging projects show, as the right tools make content management a natural part of workflow. Content management succeeds when it replaces the file and folder paradigm with a content-led paradigm.

In my conversation the other day with Jon I was arguing that although we agree on many of the technical issues there are real decisions that need to be made about what needs to be built to get the the content management future. Below are some of my lists of differences. Generally, I think the future of content management is going for the left hand one of the pairs, although some are not clear yet. I have probably missed a lot of the things to determine, but it is a start.

Architecture – API differences

These may cause API and other more significant differences, though some may not matter (eg git can read svn repos, but not vice versa).

• REST vs SOAP
• REST vs Java native interfaces
• distributed version control (git) vs file based (SVN)
• compositional vs monolithic
• structured content vs files
• relations vs metadata
• web (hypertext) content vs documents
• URIs vs referential integrity
• web applications with content management vs content management systems

Architecture – performance differences

These could have different implementations with different performance characteristics potentially. These are basically IA differences to a large extent, so they do depend on the type of problem being modelled and the modelling process. Models and performance are linked though, and the best we can do is to make parts of this pluggable so that a range of performance characteristics can be used.

• unstructured vs structured
• sparse vs dense
• untyped vs typed
• NoSQL vs RDBMS
• permission hierarchy vs permission graph
• scaleable vs local

Development process

This is key to getting the product to where you want it to be.

• open source vs proprietary
• API driven vs UX driven
• ubiquitous content management vs isolated systems
• agile vs monolithic

Architecture – usage differences

These could potentially just come down to the ways or tools with which components are joined together, maybe they do not affect architecture per se.

• social media vs controlled content
• programming languages (Javascript, XSLT) vs templating systems

6f82f1d2683dc522545efe863e5d2b73

In content management the old saying has not really been relevant. There have not really been many standards. Then one day JSR-170 turned up, a Java content management standard.

Then CMIS.

Then the other day, JSR-283, ostensibly the simple successor to JSR-170 suddenly split into two, one part the data model and the other part the Java API. Clearly leaving room for a non Java API track. Not being on the standards group I do not know when this happened, but it does smell like a bit of reaction to CMIS. And indeed Roy Fielding from Day did react to CMIS recently in a not happy way.

Previously most of us outside the Java CMS world looked on the Java standard as being a pure Java move. Many standards attempts now are about strategic commercial interests. Actually though within the Java world it seemed to be quite well accepted, and Magnolia and Day could live in the same world. Deeper down though, is it an attempt to define ranges of functionality within product ranges? Although the specification does not correspond to the implementation necessarily, both constrain each other, especially in some parts of these specifications.

CMIS was a different matter. A different set of vendors, a similar model, a different set of APIs (a non RESTful REST that Fielding could lay into!). But the biggest criticism of JSR-170 was simply the J; anything could look credible by being more inclusive. Hence I think the current changes in JSR-283.

The CMIS draft states

The JSR standard requires a particular type of implementation for ECM repositories: Whereas CMIS restricts itself to specifying only generic/universal concepts for ECM constructs like Documents and Object Types that could be layered on most existing ECM implementation, the JSR standard requires a highly-specific & feature-completion implementation of a repository. This structure may not be appropriate for many types of applications, or efficiently layered on existing ECM repositories.

This is core to Fielding’s substantive criticism – it is just trying to model folders and files, and a WEBDAV interface does that fine. JSR defines abstract item and property trees, that the CMIS players feel don’t fit with their content models. The CMIS draft mentions webdav, but says it misses out on types and queries, locking, and not http interfaces(!).

Jon Marks http://jonontech.com/2009/04/09/cmis-is-xpath-just-a-bit-too-tricksy/ points to the XPATH/SQL distinction. XPATH too is a bit modern for the CMIS vendors. The XPATH expressions refer to the XML representation of the property tree, which if it does not actually correspond at all to your internal models or implementation method is quite a lot of work to implement.

There is some truth in the implementation specific parts of the criticism of JSR, in that I am not aware of any implementation of the JSR standards that does not use a JSR native content repository as the base implementation (eg Jackrabbit). Maybe I have missed one. Partially that is because it is a strong model for a moderns CMS, with excellent open and closed source implementations. Partly also that not many vendors have yet explored related but dissimilar frameworks (it is not clear that an RDF model would fit well for example as properties are first class; though a mapping may be possible the semantics of an object may end up differing). The other area in which differences will probably become apparent are the versioning models, though these will not necessarily be exposed through interfaces if there is a big mismatch.

In an abstract sense the differences in the two models seem small. In CMIS documents have a (single) body and then additional properties. In JSR objects simply have properties, the “content” is simply anther property. Although that sounds subtle and easy to switch models – just add a type property and a content property to documents, not to folders, there are more and more model differences the further you go. Locking for example. And the rules about folders not being versioned while files are, and the primacy of the document folder containment relationship. Another difference in CMIS is the large list of optional facilities, such as the query types available, and whether checked out copies or versioned copies of documents are accessible through the query mechanisms.

Overall there is a difference in what the “generic/universal concepts for ECM constructs” are. Two formalized models is actually a useful start to help classify the CMS models that exist. A web services interface to JSR will help that model expand outside the Java only area. Islands of interoperation or at least data transfer can only help the industry as a whole.