Now this has been something have been thinking about recently, and there are people who are moving big parts of their systems to just be built on search, such as the Guardian API which is served from Apache Solr. In this case though, the search engine is still not the system of record for the core data, which is still the Oracle based CMS which did not scale up enough to serve the API. There was some discussion at the talk about search engines that do support persistence (the D in ACID databases), something Lucene used to have a bad reputation for. My view here though is that, while actually making fsync work properly is a good thing, and you should not buy software that cannot recover from crashes, persistence involves a lot more than this now, such as replication, audit, versioning and access control. Building this directly into search products is a mistake. Another issue is that search engines are denormalized, and data stores of record should really be normalized to a large extent, to minimise the amount of data to be replicated.

There are two approaches that should work instead, however.

The first is more or less the current approach, to use the search engine as an index to a persistent store. I really like this approach if we follow it to its logical conclusion, which is that the persistent store in this type of application architecture should not be a relational database, but it should be a document store, that is a file, an HTTP resource, a document in a NoSQL document database, or an object in a replicated cloud storage system like S3. Modularize the database application, and split the persistence function from the index function. The persistence function provides the durability, versioning and audit and access control, with replication, backup. This can update the search index, and potentially any other types of index, such as a graph database for querying relationships, potentially even a relational database if that is the best way of querying some aspects of the data.

Obviously there is a potential consistency issue, if updates from the document store happen slowly, so potentially there is an eventual consistency model. Historically search was a bad offender here, as dynamic updates were not the norm and everything was batched into nightly updates, but that is going away and dynamic updates are more normal for search indexes. In principle you can have more consistency, especially in an architecture where there are fixed releases that can be consistently indexed, rather than distributed rolling updates, you choose your architecture and take your choice. Small consistency lags rarely matter in a lot of applications.

So you end up with an architecture with a well defined persistence layer that is not a relational database, and a set of indexes appropriate to the application, almost certainly including a full text search engine, but perhaps a graph engine too. Maybe you run consistency checks on your indexes for peace of mind.

The second approach is to see that search engines were some of the original NoSQL data stores, building custom storage and indexing engines, because they had such difficult problems. Indeed Google’s BigTable, and so the ancestry of a lot of NoSQL products came from search. However the search engines around now have not yet refactored themselves on top of the NoSQL engines that have emerged from this work, although this is starting with Lucandra which is Lucene persisted in Cassandra, which looks promising, offering seamless replication and distribution, and HBasene, an HBase Lucene backend. These make a huge amount of sense to me, as if you are developing sophisticated search algorithms, not having to build the whole index and persistence layer as well is a big advantage, as well as the scale out potential. Of course this approach does not conflict with the first one, in fact you could choose a NoSQL backend that is aimed more at read performance than persistence, and at storing small index values fast. The hard bits with this are that the search engines have specifically customised their data storage for the particular use cases, and reworking this onto a more general backend has few apparent advantages; as you can see from the examples above, most of these changes have come from people already using the backends in question and who want a single database to manage all their data requirements, particularly once they are working with high availability and replication. Software modularity really is not at the right level yet is it, I blame object oriented programming for this lack of reusability.

Anyway, back to the main point. For applications like content management, an architecture based on a content store that deals with persistence, versioning, access control, replication, with a set of indexes based on search engine techniques, graph databases, and anything else your applications needs. Ideally the indexes are all based on a common set of low level primitives so the backend can be swapped out or shared between the search store and other application specific indexing requirements, so there is a single low level indexing infrastructure that can be available as a common scalable service, with different implementations available. This type of architecture is quite buildable now, and is certainly used in quite a few applications, and I think it will become much more widespread, particularly in the cloud where it seems more natural, certainly for many types of application that fit into a document type model, such as content based applications.

Virtualization

Virtualization mainly started as a way to deal with issues in running multiple services on Windows, due to compatibility issues. This has always been much less of an issue with Linux applications, due to the scale of supporting libraries packaged by distributions. It is still an issue though, for security reasons (apache without suexec for shared hosting still exists, bypassing OS based multi tenancy security, a model that should have gone years ago). KVM, which uses Linux as a hypervisor and uses the hardware virtualization capabilities of newer hardware as now in the Linux kernel, and supported in Redhat Enterprise Linux. I suspect this will gradually overtake Xen and VMWare in areas where only Linux is of interest, due to the built in kernel support; however lighter weight solutions for the security issues such as containers will probably take off instead for many applications where running multiple kernels is unnecessary.

Containers

Linux now has a full container model called LXC, similar in principle to BSD jails and Solaris zones. It arrived a bit gradually as a set of patches to namespace various parts of the system such as the process ID space, so a container has its own init process with ID 1 and can have the same IDs as other containers (this also is needed for process migration). There is also a network namespace, so each container has its own loopback device, and independently named network devices (that can for example be bridged back to the host). There is also a read only bind mount which can be used to safely export libraries and binaries to multiple containers with updates done centrally if required; otherwise the container can be managed as a standalone system just sharing the kernel. This environemnt provides a level of secure isolation between containers that solutions such as chroot never had. Processes in containers can be seen from the container host so obviously this needs to be well secured. Because containers do not need hardware support and are very lightweight I think they will grow rarpidly in popularity; they can also run within a virtual machine guest for process isolation inn a virtual environment. Ubuntu 10.04 will have full support; earlier versions do work.

Capabilities

The old high risk ways of setuid binaries (with broad permissions) are going at last, replaced by a fine grained capabilities system. In principle this means you can drop root capabilities completely, making root an unpriviledged user. There is a good summary article on this and another on trying to remove root access. It seems that we will not see pure capabilities based Linux distributions for a while, and will have setuid binaries in general purpose systems, but there is no reason why single application sandboxes should not drop root capabilities in their init process and just use capabilities set in the file system. Fedora seems the furthest ahead in trying this out as a full distribution, and hopefully this will move ahead, adding another security layer in addition to SELinux.

Sandboxing

Privilege separation in network applications has been around for a while, but it is starting to spread, with the best example being the Chrome security model. The thing that has really started to change is treating all complex bits of code, such as HTML rendering in Chrome, as potentially hostile as they are likely to be buggy. There is a lot to do to get good security thinking pervasive in application design, but having some well thought out examples is a good start. Currently Linux Chrome seems to offer a choice of sandboxing methods of varying effectiveness from a suid helper to using seccomp

SELinux

SELinux has been available in Linux, providing a Mandatory Access Control framework for ten years now, but it has taken that long for it to get really widespread use, mainly pushed by RedHat. Gradually it is extending to other applications, such as mod_selinux for Apache that runs web applications in appropriate security contexts; Postgres SELinux extensions are also available. We are getting to a point when OS security mechanisms can and will be used as they provide the types of security hooks that modern applications need, after a period where we have had applications inventing their own security mechanisms because the OS did not provide the right ones.

Physicalization

There was an interesting new buzzword this year: physicalization. Yes just when you tought virtualization was an important new trend, along comes the opposite. What is the idea?

A two socket 8 core server with 16GB RAM and multiple ethernet ports divided into four virtual servers is actually quite expensive compared to four commodity low end boxes. There is a server premium built into the chip manufacture profit model for a start, and also a volume issue.

The price arbitrage is fairly compelling, although the other costs (disks, motherboards, networking) add up and reduce the saving. The example systems are things like SGI’s Microslice – yes SGI, that name from the past! This offers dual core but single CPU systems, but with ECC, for significantly lower price and power consumption than typical two way servers, and potentially more throughput per $, for some workloads.

There are even some suggestions that for Linux workloads non x86 architectures (eg ARM) might be competitive for applications that scale out effectively to multiple machines, although I think the risk of introducing these would be high, and there would need to be a big buyer.

Cloud

The big coming trend as the world comes out of recession is that cloud computing platforms are cheap, very cheap, compared to in house server provision. Some estimates put it at 20% of cost now, falling to 10% this year. Part of this is economies of scale, part is standardized components and architectural options, and economies of scale in administration. Part of it may be untrue, as there certainly do not appear to be good figures. What is clear is that the SAAS model is compelling for many kinds of product, and fits in with a general movement to charge software as an expense not an investment. There is a lot of hype, and a lot of people have seen the cloud idea before under different names, but the web has produced a viable delivery mechanism, and the uniformity of hosting environments like EC2 cuts costs. Costs such as upgrades are much lower in a SAAS environment too; although the architecture of this software needs to be different to support that.

Availability

The last year or so, high availability programming has reached out into awareness a bit. The Erlang model has become better known, bringing more awareness of the base elements for building reliable systems such as process supervision. We are starting to see other implementations, such as Akka. This is a great move, as availability needs to move from being a sysadmin and maintenance issue to being a coding issue; for too long effective handling of failure has been ignored by programmers.

Locks

As applications start to scale to more threads on multicore CPUs, locking becomes more of an issue. Lock-free algorithms are one interesting answer that has emerged that can work well for some algorithms. Getting past the scaling issues as architectures get more cores needs innovation in lots of areas such as this. Locks are definitely in the sequential areas that limit scaling through Amdahl’s law.

Summary

Software architecture is at an interesting point; the principles of web architecture and the security mindset are gradually feeding into tools and infrastructure and becoming more widespread, and delivery is also changing. Scalable, available and secure systems are the aim.

Although the vast majority (but not all) of open source content management systems are continually trying to reinvent the blog, we are talking about internet infrastructure here, and the future of content is going to be open source, like the rest of internet development. I also believe that long term the web project will overwhelm the legacy areas of document management, although it may take some time. Hypertext, the web architecture, XML, HTML, and all those standards are here to stay and to dominate long term. Content management will also become pervasive long term, as the blogging projects show, as the right tools make content management a natural part of workflow. Content management succeeds when it replaces the file and folder paradigm with a content-led paradigm.

In my conversation the other day with Jon I was arguing that although we agree on many of the technical issues there are real decisions that need to be made about what needs to be built to get the the content management future. Below are some of my lists of differences. Generally, I think the future of content management is going for the left hand one of the pairs, although some are not clear yet. I have probably missed a lot of the things to determine, but it is a start.

Architecture – API differences

These may cause API and other more significant differences, though some may not matter (eg git can read svn repos, but not vice versa).

• REST vs SOAP
• REST vs Java native interfaces
• distributed version control (git) vs file based (SVN)
• compositional vs monolithic
• structured content vs files
• relations vs metadata
• web (hypertext) content vs documents
• URIs vs referential integrity
• web applications with content management vs content management systems

Architecture – performance differences

These could have different implementations with different performance characteristics potentially. These are basically IA differences to a large extent, so they do depend on the type of problem being modelled and the modelling process. Models and performance are linked though, and the best we can do is to make parts of this pluggable so that a range of performance characteristics can be used.

• unstructured vs structured
• sparse vs dense
• untyped vs typed
• NoSQL vs RDBMS
• permission hierarchy vs permission graph
• scaleable vs local

Development process

This is key to getting the product to where you want it to be.

• open source vs proprietary
• API driven vs UX driven
• ubiquitous content management vs isolated systems
• agile vs monolithic

Architecture – usage differences

These could potentially just come down to the ways or tools with which components are joined together, maybe they do not affect architecture per se.

• social media vs controlled content
• programming languages (Javascript, XSLT) vs templating systems

6f82f1d2683dc522545efe863e5d2b73

For anyone who has missed this there is a survey from Richard Jones at last.fm of some of the main software projects out there – although missing some like Tokyo Cabinet – there is a lot in this area going on in Japan.

Scaling has suddenly become something that a lot of people have to do, and they are working in an open source world, so many of the things they build et released out into the world. Many of these projects come out of the big web sites, or are products for building scaling services like Amazon’s SimpleDB. Scaling a relational database above the point where replication alone works gets difficult, and replication only improves read performance while leaving a write bottleneck. Sharding is the usual approach, partitioning the data across multiple machines, but it removes the ability to perform joins (as these now cross machines). Once you lose that you have lost the theoretical basis of the relational model.

The key value stores that exist are more or less based on building persistent distributed high performance versions of basic data structures. Two dominate, the simple hash table, which is just a pure key value store, as with MemcacheDB which is built on Berkeley DB which is perhaps the oldest persistent key value store, but with a network interface not just a language API. These just give very fast lookup and storage of values associated with a key. Thats it, one single value. The other form is the B-tree based form, which generally add the ability to do range queries, so with careful selection of keys you can return multiple values from tanges, as in CouchDB. Pretty simple then.

Although relational databases are basically built on B-trees, there is a huge gap for most applications in using structure at this level. Normalisation has gone, so there goes that part of the storage design process. Some applications or parts of applications map easily, and these easy parts are where these technologies are getting most use now. Some people are using denormalisation now with database backends (Flickr stores data twice in some cases). So far there is no data model, so it is expensive custom work in each situation to map these calable storage models.

As a side note, there used to be a lot of discussion about integrating SQL into programming languages as a native data type. But the only language the relational model really works with is Prolog as that has a matching native structure. Key value stores match much better to native structures, but mainly because of the weaker expressive structure.

The relational database did well because the model had the right balance between implementation for people and for computers to optimize much of it. That is generally a hard balance. Key value stores are putting all the work back on the programmers right now. Some projects are starting to work on the next stages – CouchDB views get a few things right that other frameworks do not look at, but there is a lot of work to do here.