As I said in the talk, D3 is really great to use, as it is about code not configuration, and lets you really do what you want. It is purely based on working with data and the DOM, without other abstractions layers. I hightly recommend it!

One thing I mentioned in the talk was using Google spreadsheets for storing arbitrary data that doesn’t already have somewhere to live. I have found this worked out quite well with clients who want an easy “CMS” for data that they can update and feed straight through to graphs and maps etc on the website. A few people asked about this, so I thought I would give a quick guide.

I have made a public Google spreadsheet for these examples. Will probably get messed up as it is public!

The easy way to use this is just to use CSV, as once you make the spreadsheet public, the interface will show you the CSV URL (of the first sheet only, regardless of the setting). In this case the link is https://docs.google.com/spreadsheet/pub?hl=en_US&hl=en_US&key=0Agr9u3lfdOGHdHZoc1dVYXhDdl9GSk5SbVRBSGZYRFE&single=true&gid=0&output=csv which gave me the following:

date,colour,value,genre,french
2011-01-01,red,2,rock,roches
2011-01-02,blue,1,grunge,grunge
2011-01-03,orange,66,pop,pop
2011-01-04,green,4,metal,métallique
2011-01-05,pink,6,jazz,le jazz
2011-01-06,brown,5,punk,Punk
2011-01-07,purple,33,rap,Rap
2011-01-08,black,12,reggae,reggae
2011-01-09,white,11,alternative,alternatives
2011-01-10,gray,5,dance,danse
2011-01-11,violet,14,country,pays
2011-01-12,yellow,9,blues,le blues
2011-01-13,indigo,10,funk,funk

This is clearly usable in D3, but it does have cross domain issues in most browsers, so you would normally have to proxy it back to your domain if you are calling via Ajax.

The options in the “publish to the web” menu for giving you URLs are HTML, CSV, text, PDF, XLS or ODS, none of which are what we really want, which is JSONP. However there is a JSONP option, it is just a very different URL, which is designed to be got through the API. The problem with the API is all the calls are authenticated so it is a real pain to just extract a (public!) URL. After hacking around with the Python client example code, I found a (messy) workaround however.

Every spreadsheet on Google has an ID, and each sheet within it also has an ID. These do not correspond to the keys in the public access links. You can get the spreadsheet ID from within the Javascript API available for spreadsheets, as SpreadsheetApp.getActiveSpreadsheet().getId(). You cannot get the sheet ID, but the first sheet you create always has the ID od6, the next od7, and then some other values that do not seem to change. So to write a little function that gives you the spreadsheet JSONP URL, do the following:

<

ol>

function getURL() {
  return "http://spreadsheets.google.com/feeds/list/" + SpreadsheetApp.getActiveSpreadsheet().getId() + "/" + "od6/public/values/?alt=json-in-script&callback=";
}

If we open that URL with a callback of test we get back:

// API callback
test(
{ "encoding" : "UTF-8",
  "feed" : { "author" : [ { "email" : { "$t" : "justin@specialbusservice.com" },
            "name" : { "$t" : "justin" }
          } ],
      "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
            "term" : "http://schemas.google.com/spreadsheets/2006#list"
          } ],
      "entry" : [ { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: red, value: 2, genre: rock, french: roches",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "red" },
            "gsx$date" : { "$t" : "1/1/2011" },
            "gsx$french" : { "$t" : "roches" },
            "gsx$genre" : { "$t" : "rock" },
            "gsx$value" : { "$t" : "2" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cokwr" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cokwr",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/1/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: blue, value: 1, genre: grunge, french: grunge",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "blue" },
            "gsx$date" : { "$t" : "1/2/2011" },
            "gsx$french" : { "$t" : "grunge" },
            "gsx$genre" : { "$t" : "grunge" },
            "gsx$value" : { "$t" : "1" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cpzh4" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cpzh4",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/2/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: orange, value: 66, genre: pop, french: pop",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "orange" },
            "gsx$date" : { "$t" : "1/3/2011" },
            "gsx$french" : { "$t" : "pop" },
            "gsx$genre" : { "$t" : "pop" },
            "gsx$value" : { "$t" : "66" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cre1l" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cre1l",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/3/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: green, value: 4, genre: metal, french: métallique",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "green" },
            "gsx$date" : { "$t" : "1/4/2011" },
            "gsx$french" : { "$t" : "métallique" },
            "gsx$genre" : { "$t" : "metal" },
            "gsx$value" : { "$t" : "4" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/chk2m" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/chk2m",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/4/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: pink, value: 6, genre: jazz, french: le jazz",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "pink" },
            "gsx$date" : { "$t" : "1/5/2011" },
            "gsx$french" : { "$t" : "le jazz" },
            "gsx$genre" : { "$t" : "jazz" },
            "gsx$value" : { "$t" : "6" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ciyn3" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ciyn3",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/5/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: brown, value: 5, genre: punk, french: Punk",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "brown" },
            "gsx$date" : { "$t" : "1/6/2011" },
            "gsx$french" : { "$t" : "Punk" },
            "gsx$genre" : { "$t" : "punk" },
            "gsx$value" : { "$t" : "5" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ckd7g" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ckd7g",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/6/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: purple, value: 33, genre: rap, french: Rap",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "purple" },
            "gsx$date" : { "$t" : "1/7/2011" },
            "gsx$french" : { "$t" : "Rap" },
            "gsx$genre" : { "$t" : "rap" },
            "gsx$value" : { "$t" : "33" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/clrrx" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/clrrx",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/7/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: black, value: 12, genre: reggae, french: reggae",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "black" },
            "gsx$date" : { "$t" : "1/8/2011" },
            "gsx$french" : { "$t" : "reggae" },
            "gsx$genre" : { "$t" : "reggae" },
            "gsx$value" : { "$t" : "12" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cyevm" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cyevm",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/8/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: white, value: 11, genre: alternative, french: alternatives",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "white" },
            "gsx$date" : { "$t" : "1/9/2011" },
            "gsx$french" : { "$t" : "alternatives" },
            "gsx$genre" : { "$t" : "alternative" },
            "gsx$value" : { "$t" : "11" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cztg3" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cztg3",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/9/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: gray, value: 5, genre: dance, french: danse",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "gray" },
            "gsx$date" : { "$t" : "1/10/2011" },
            "gsx$french" : { "$t" : "danse" },
            "gsx$genre" : { "$t" : "dance" },
            "gsx$value" : { "$t" : "5" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d180g" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d180g",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/10/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: violet, value: 14, genre: country, french: pays",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "violet" },
            "gsx$date" : { "$t" : "1/11/2011" },
            "gsx$french" : { "$t" : "pays" },
            "gsx$genre" : { "$t" : "country" },
            "gsx$value" : { "$t" : "14" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d2mkx" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d2mkx",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/11/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: yellow, value: 9, genre: blues, french: le blues",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "yellow" },
            "gsx$date" : { "$t" : "1/12/2011" },
            "gsx$french" : { "$t" : "le blues" },
            "gsx$genre" : { "$t" : "blues" },
            "gsx$value" : { "$t" : "9" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cssly" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cssly",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/12/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: indigo, value: 10, genre: funk, french: funk",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "indigo" },
            "gsx$date" : { "$t" : "1/13/2011" },
            "gsx$french" : { "$t" : "funk" },
            "gsx$genre" : { "$t" : "funk" },
            "gsx$value" : { "$t" : "10" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cu76f" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cu76f",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/13/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          }
        ],
      "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values" },
      "link" : [ { "href" : "https://spreadsheets.google.com/pub?key=tvhsWUaxCv_FJNRmTAHfXDQ",
            "rel" : "alternate",
            "type" : "text/html"
          },
          { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values",
            "rel" : "http://schemas.google.com/g/2005#feed",
            "type" : "application/atom+xml"
          },
          { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values?alt=json-in-script",
            "rel" : "self",
            "type" : "application/atom+xml"
          }
        ],
      "openSearch$startIndex" : { "$t" : "1" },
      "openSearch$totalResults" : { "$t" : "13" },
      "title" : { "$t" : "data",
          "type" : "text"
        },
      "updated" : { "$t" : "2012-01-29T15:22:57.368Z" },
      "xmlns" : "http://www.w3.org/2005/Atom",
      "xmlns$gsx" : "http://schemas.google.com/spreadsheets/2006/extended",
      "xmlns$openSearch" : "http://a9.com/-/spec/opensearchrss/1.0/"
    },
  "version" : "1.0"
}
);

Now you may think that is odd JSON, but it is a literal JSON conversion of ATOM, which is why it has a lot of apparent junk. The bit we want is the array of objects in feed.entry which has among other entries

[         { ...
            "gsx$colour" : { "$t" : "red" },
            "gsx$date" : { "$t" : "1/1/2011" },
            "gsx$french" : { "$t" : "roches" },
            "gsx$genre" : { "$t" : "rock" },
            "gsx$value" : { "$t" : "2" },...
          }, ...
]

Bonkers, you may think. But it does work, and what you get is a reliable easy to use http addressible tabular data source. Really, the whole of Google docs is like this, half genius half biscuit. It could be a great service, but almost seems to be wrapped up in legacy already. It could be so much better, but improvement seems slow; the work seems to have gone into the actual editing side not the backend, which is also saddled with Excel compatibility goals. A lot of scope for competition in this space.

Anyway, I hope that helps. I really do think for many applications it is a great backend system, if you hide the complexity from the end users!

I actually had native IPv6 ADSL many years ago, with the now defunct Black Cat Networks, although at some point ipv6 routing just stopped working, and actually apart from being able to ping6 my friend who had the same setup it was not terrible useful, although I did test all my code and make sure it was all ipv6 ready. But now there are actually compelling reasons for IPv6, like the fact it is getting harder to get IPv4 addresses. In particular I have been spending some time on Linux containers (lxc), and if you want to run network isolated containers they need addresses, and if you are doing this on many cloud services you cannot get extra IPv4 addresses, so either you use internal NAT, yuk, or use IPv6. Also the first steps towards Amazon web services supporting IPv6 came out the other day, with ELB supporting IPv6 termination, although there is not yet support for IPv6 addresses on EC2 instances directly, although you can add tunnels; see my answer on Quora for the current status.

So how do you go about getting yourself IPv6 enabled? First this blog, which is currently hosted on Slicehost, which does not yet have native IPv6 support, although they will have later in the year, as part of their full merger into Rackspace. I keep wondering about moving to Linode as it is cheaper, although the way they have set up IPv6 seems odd to me.

Stage one is signing up with someone who will give you an IPv6 tunnel to an IPv4 address. I would recommend using Hurricane Electric, as they have global tunnel endpoints and a good service. Sign up here, and you can have 5 free tunnels. It is very simple, just provide the IPv4 endpoint, and your details, choose an endpoint near your server, and you will get given an IPv6 /64 address and some helpful instructions as to how to configure it based on your OS. They had an endpoint in Dallas where my Slicehost is, which has a ping time of 1ms, which is nice. I used the modern style ip setup, which I just added to /etc/rc.local so it is recreated on reboot:

modprobe ipv6 
ip tunnel add he-ipv6 mode sit remote 216.218.224.42 local 67.23.6.148 ttl 255
ip link set he-ipv6 up
ip addr add 2001:470:1f0e:6e6::2/64 dev he-ipv6
ip route add ::/0 dev he-ipv6

Very easy, no authentication or anything as it is all fixed by IP address. My IP is 2001:470:1f0e:6e6::2. Whats the /64 you ask? Thats a block of 18,446,744,073,709,551,616 IP addresses, minus the first and last which are not used. That is generally the smallest globally routable allocation, and should be find for an organization to allocate to their users and devices, including virtual devices and so on; it also makes it easy to autoconfigure devices, as they can use their MAC addresses or random numbers to pick an address within the allocated block, so configuration like DHCP is not generally needed.

Next we need to configure DNS records for IPv6, that is AAAA records in addition to the A records that give IPv4 addresses. If your DNS provider does not support AAAA records it is probably time to get a new one! I use Zerigo for my Blog domain, and it was very easy to add records pointing at 2001:470:1f0e:6e6::2.

After that you need to make sure your services are listening on IPv6. netstat lists IPv4 only listening sockets as *:*, while IPv6 sockets, which can listen on IPv4 addresses too as there is a legacy mapping, are listed as [::]:* instead. In my case, I had to change Nginx to listen on IPv6 too, with the config being listen [::]:80; instead of listen :80;. Restart the web server, and then you can use the IPv6 test site validator to check for you that everything is up and ready, and get your badge:

All very well, but you really want to test it yourself don’t you. And of course you have other software you need to test on IPv6, so you want to set up your desktop machine with IPv6 too. Time to make another tunnel. Now this is one of those cases where maybe you do want to try this at home. Or at least tell your sysadmin before setting it up in the office. With IPv6 machines are directly addressable on the internet, and the tunnel probably bypasses any firewalling. WIth all the IPv6 addresses, portscanning is really difficult, unlike IPv4, but you should treat machines as connected and keep them secure, which of course you do at home, but judging from the amount of IE6 around does not happen at work so much.

Now my home connection is ADSL with a not very good modem, that I have been meaning to replace with a proper one. The main issue is I don’t really know how stable my IP is, so just going to have to experiment. If I had a UK server I could make my own tunnel I suppose, but will see how I manage with a Hurricane Electric tunnel which I will have to recreate if the IPv4 address changes, possibly with a new IPv6 address too. My guess is the IPv4 address will be pretty stable, it was the same over a box reset just now. Also Hurricane Electric have an API call to change your tunnel address! They also have an endpoint in London, 20ms away from me.

First issue was that Hurricane Electric complained it could not ping my IP address. I looked through the config options on the O2 ADSL box (really a Speedtouch WL780) and there was nothing, so the internet found the answer. telnet to the box, login as SuperUser with password O2Br0ad64nd and type service system ifadd name=PING_RESPONDER group=wan, then saveall to write the config to flash and all will be well. There is a manual for the router online.

Next realization though was that I need to get the ADSL NAT to forward the encapsulated IPv6 traffic to my Linux box, as the router has no ability to do IPv6 itself. At some point I will move the termination to my nice ultra low power dual core 1GHz ARM Trimslice box. So back to the telnet and some helpful guides, all written in Dutch, thanks Chrome for the translations. Make sure you don’t translate the router commands! Replace YOURNATIP with the NAT address of your IPv6 endpoint.

expr add name=IPv6to4_prot_41 type=serv proto=41
firewall rule add chain=forward_host_service index=10 name=map_41 serv=IPv6to4_prot_41 log=enabled state=enabled action=accept
nat tmpladd group=wan type=nat outside_addr=0.0.0.1 inside_addr=YOURNATIP protocol=6to4 weight=50
saveall

Then set up the Hurricane Electric tunnel on the endpoint, using the NAT address:

modprobe ipv6
ip tunnel add he-ipv6 mode sit remote 216.66.80.26 local 192.168.1.77 ttl 255
ip link set he-ipv6 up
ip addr add 2001:470:1f08:1962::2/64 dev he-ipv6
ip route add ::/0 dev he-ipv6

Now you should be able to ping6 inbound, and outbound to your IPv6 address, and reach ipv6.google.com for example. Going to an IPv6 test site such as http://test-ipv6.com/ finds the IPv4 and IPv6 addresses. It does point out that the name server I am using does not have any IPv6 support though, which could cause issues with IPv6 only sites with IPv6 only name servers. Not going to fix that for the moment though. So we can browse the web on IPv6, at least, although it is quite hard to tell if you actually are. You can check your web server logs, IPv4 addresses are listed in IPv6 notation as ::ffff:188.220.243.64, while the real IPv6 we are browsing from are given in full, so we can see that Firefox for example will use IPv6 addresses by default when available.

One more thing is the rest of the machines on the network at home (or in the office!). We have our /64 allocation, so lets use it. All you need to do is run a server that will reply to the stateless autoconfiguration requests, which is radvd on Linux:

apt-get install radvd
sysctl -w net.ipv6.conf.all.forwarding=1

Set the sysctl setting in /etc/sysctl.conf too so it is persistent. This stops you receiving IPv6 router advertisements and lets you forward IPv6. Then create an /etc/radvd.conf for the interface you want to broadcast on, yes mine is br0 as I have an bridge setup for my VMs.

 interface br0
{
  AdvSendAdvert on;
  MaxRtrAdvInterval 4;
  prefix 2001:470:1f09:1962::/64
  {
    AdvRouterAddr on;
  };
};   

Now you can start radvd and set up the service to be auto restarted. And then by magic any computers in the vicinity that do not have IPv6 disabled will automagically get an IPv6 address, based on their MAC address and your prefix. Magic! My Linux netbook got one, but I could not ping anything. First issue was that my bridge (or in your case your network interface) did not have an IPv6 address, so I added one with ip addr add 2001:470:1f09:1962::2/64 dev br0. Now I could ping6 that, but no external routes worked. It turns out that IPv6 forwarding is all very well, but you need explicit routing tables too. So add ip -6 route add 2000::/3 via 2001:470:1f08:1962::1, where 2000::/3 is all global routes, and the 2001:470:1f08:1962::1 is the other end of your IP6 tunnel. Sorted! Well kind of, both the machines, Linux and Mac, that I am testing with are constantly losing routing and ability to ping6 the gateway. Odd, the gateway machine is fine. OK fixed, see this forum page, actually the link address is not the allocated routing address. 2001:470:1f08:1962::/64 is the network of the point to point link, but my routed network is the one after, that is 2001:470:1f09:1962::/64. This is listed when you login to Hurricane Electric again. I have changed the configs above now. Everything just working as expected on all the computers.

Note that you should be able to do all this with a Mac OSX machine or Windows machine as the router or server, the setup will be a little different, but Hurricane Electric should give you the help, along with your OS vendor. As for your routers, you may have to read some Dutch too!

So there you are, a quick guide to getting started with IPv6. Happy protocol upgrades!

Note: I have now moved my blog to Hetzner which has native IPv6 availability, which makes life much simpler. If you are selecting a new provider, always choose one that supports IPv6! Rackspace/Slicehost my former provider seems rather behind the times.

General purpose scripting languages have been with us for a long time, since Rexx was started in 1979 and Tcl in 1988, followed by Lua in 1993; depending on what you define as a scripting language, as you might include IBM’s Job Control Language, or shells such as ksh. Indeed there is some confusion in the Wikipedia article about what they really are, and the term has been used for all sorts of languages.

A scripting language is a language that is designed to either control an external program or set of programs, providing an easy way to construct compositions of the components being provided by the library or toolkit, or the opposite construction, to be embedded inside another language to control parts of the application in a dynamic way, like Javascript controls rendering in a browser. These are normally referred to as extending and embedding.

The reasons for the two different ways vary, with extending being for use cases like shell scripting where there is a large library of programs with a uniform interface which can be composed to perform a multitude of tasks. The performance of the shell script language is not particularly important is it is mostly just building system level compositions, such as pipes, or making simple conditions and loops. Other examples though are more complex, and the successful general purpose scripting languages have a full set of programming language types and constructs, including first class functions, inheritance and so on; languages like shell scripts and Tcl which only have a single string type staying limited to smaller areas. Essentially these become domain specific languages (DSLs) for working in a paarticular domain. Structuring your code so that it acts as a set of libraries for the domain, while embedding these in a scripting language that does the plumbing is a good way of creating a flexible design, and indeed if you cannot refactor your application as independent libraries with a scripting glue then it is probably not very well designed.

The other way round, embedding, is sometimes unfairly seen as a bad idea, but in the right situations it makes a lot of sense, for example for embedding a query language, or for avoiding server round trips by coordinating a set of commands, such as the way databases embed store procedure languages, or the recent embedding of Lua in Redis.

Why not do everything in one language? The original reasons were that “real” programming languages were statically typed, compiled, and had terrible string handling (yes C, we are looking at you, a language which once had gets), while scripting languages had garbage collection, dynamic everything, interpreted environments with friendly errors, simple string libraries, and were extremely slow, maybe 100 times slower than C. They also used to have fairly poor module structuring, and other facilities for programming in the large. This has not stopped people building large projects using largely scripting languages (Vignette in Tcl being an early example), particularly with the LAMP stack which started off as a simple glue between a web server and a database, but has grown to much larger applications.

What has changed is a gradual convergence, as some of the more friendly features of scripting languages, such as garbage collection and better string libraries started to move into mainstream languages with Java, and the JIT compiler that really started gaining popularity with the JVM has recently been seriously applied to scripting languages, in particular Javascript, Lua and Python, which are making a bid for serious performance. LuaJIT now performs similarly or better than Java in many benchmarks, while PyPy and Javascript are rapidly getting within a small factor of Java. This does not mean that there are not still many places where static memory allocation and the low level guarantees of C are not useful, such as in database design and so on, and of course there are large libraries of existing, well tested software.

Foreign functions

Another gradual change is the development of FFI (foreign function interface) libraries. The original open source libffi has been around since 1996, and Python has had ctypes for a long time too, but there have been issues with these bindings. While they are relatively easy to construct compared to writing a full C binding, they are messy in some languages, although the Ruby ones are fairly readable, a binding to puts in libc being defined with:

module Foo
  extend FFI::Library
  ffi_lib FFI::Library::LIBC
  attach_function :puts, [ :string ], :int
end

The second problem after syntax, was performance. Most ffi bindings were very slow compared to native C bindings, by a factor that was significant for most use cases. However this has started to change too, first with the Rubinius bindings from a few years back, which have a very small overhead, of just a function call which performs the necessary type conversion, and then more recently with the LuaJIT FFI library which not only has a usable syntax, as it has most of a C header file parser built in, it also is natively understood by the JIT compiler so it can generate code that has no overhead at all, actually less than the standard C Lua bindings.

So the following simple program that mainly executes a fast (virtual) system call:

local ffi = require "ffi"

ffi.cdef[[
struct timeval {
  long tv_sec;
  long tv_usec;
};
int gettimeofday(struct timeval *tv, void *tz);
]]

local tv = ffi.new("struct timeval")

for i = 1,100000000 do
  ffi.C.gettimeofday(tv, nil)
  if tv.tv_usec == 0 then print "." end
end

generates the following assembly for the inner loop, where the call r12 is a direct call to the libc syscall wrapper:

->LOOP:
394cffd0  mov rdi, [rsp+0x8]
394cffd5  xor esi, esi
394cffd7  call r12
394cffda  cmp qword [rbx+0x10], +0x00
394cffdf  jz 0x394c0024 ->5
394cffe5  add ebp, +0x01
394cffe8  cmp ebp, 0x05f5e100
394cffee  jle 0x394cffd0    ->LOOP
394cfff0  jmp 0x394c0028    ->6

And that runs marginally faster than the C equivalent, showing the advantages of an FFI library that is natively understood by the JIT compiler, as well as of course the analysis that goes behind making sure this is a valid optimisation, such as being able to register allocate the loop variable. It is also nice to see a scripting language generating nice assembler! LuaJIT is currently the fastest dynamic language available by a large margin.

So are there any disadvantages to a well designed FFI interface? Well, having been using the LuaJIT one for a while, the issues are mostly with how some C code is written. A lot of C code is not well written, well encapsualted code. The ABI may depend on all sorts of conditions, not just the architecture, but also the build options of the program, all wrapped in #ifdef conditionals. Macros are used a lot, sometimes generating code for runtime, which of course then has to be reimplemented in the scripting language. The preprocessor is overused, with people rarely using enum instead, and C enums have odd semantics, as they are always cast ints. Scripting languages rareky if ever use stack allocation, while C libraries assume that it is often the norm, so libraries do not hide their internal structures with void * pointers that they heap allocate, which would often be much easier. Also C libraries have historically been written to support old versions of C, so without variable length arrays, and without the sized integer types such as uint32 etc. So it can get messy to interface with, requiring a lot of testing and extra code wrappers to make things work well. Best to stick to well designed code if at all possible.

Structuring scriptable programs

The easiest way to write code that is friendly to being scripted is if most of the code is structured as a set of libraries, exposing clean operations and with clear semantics for allocation and deallocation; a good way to write code anyway. For maximum portability, C is easier to interface with than C++, due to name mangling and C++ exceptions, as well as the fact that you may be interfacing to a language that does not really do object orientation in the way you might use it in C++. C++ exceptions have marginal support in FFI interfaces, and explicit error handling at the external interfaces is more easily usable. Don’t use thread local return values, like errno either. Callbacks in the same scripting VM may also be a problem in some environments, that is calls from the scripting language to C then back to a script callback. Generally owning the event loop in a library is annoying, and this is a case of that to some extent, it is easier in that case if you just call into user code, as in Node.js. These are generally sensible organisational principles for code anyway, so it should be possible to script any well written code.

Note I didnt really mention Java and .Net here. Most people largely only interface within their own runtime. While there are JCM and .Net versions of most programming languages, they are less well supported in general, and especially in the Java case very slow, often similar to the native interpreter, or a little faster but not as fast as native JIT compilers; although there have been recent improvements, the JVM is not currently friendly to dynamic languages. It is of course possible to expose a C API to Java code, through JNI, as it works both ways, to allow Java to be scripted from a non JVM scripting language, although it seems to be less common, and it also has heavier performance costs than the usual C to non-managed scripting language boundary. Of course the big advantage of staying within these frameworks is that calling other languages say within the JVM is very easy, as the runtime understands the calling conventions, so interoperability is very simple, so many Java programs offer Rhino scripting say, but it will be slow. Instead there are statically typed but more appropriate languages for DSLs that can be used, such as Scala or Clojure.

For calling into user code, for example as in the Redis scripting API, or the older but similar example of database stored procedure languages, another case where more standard scripting languages are now available such as Lua in Postgres, the aim is to make the exposed operations easy to work with, and to allow use of native language features, such as how iterators work, and to use coroutines if that is appropriate, as for example in the Lua embedding in Nginx which uses coroutines so that apparently synchronous code can be run asynchronously. The use cases are many, one to improve APIs so that you minimise round trips and moving what would be client side operations to the server, sometimes to implement atomic operations that could not be specified over an API in a performant way, as with stored procedures in a database. Another use is the Node.js case, to embed a very well known and usable language in some low level code (the asynchronous native library), an environment that would otherwise only be availabe in C. Many large programs are actually structured largely as scripting language layers, for example over 40% of Adobe Lightroom code is written in Lua, and much of Firefox is written in Javascript.

Summarising

There has been a huge effort in making scripting languages perform well. Performing well while interfacing in a really easy way to external code is another big step to making scripting a default part of the design of the majority of large scale code, as well as to help integrate the huge installed and tested codebases already out there. Scripting languages generally do not have an imperative to avoid externally programmed core code, unlike say Java, for compatibility reasons, or Go for simplicity reasons. This makes them excellent glue languages, combined with dynamic typing that tends to allow easy modification. If scripting languages were as fast as say C, there are still people who prefer to use statically typed languages with more deterministic compilation and potentially runtime guarantees, and some sorts of libraries are more likely to be available on some languages than others, skewing the choices. But if you are writing complex memory management code in C++, it could be time to switch parts of the code with unclear lifetimes to a scripting language. Mixing languages has never been easier, or more compelling.

Now this has been something have been thinking about recently, and there are people who are moving big parts of their systems to just be built on search, such as the Guardian API which is served from Apache Solr. In this case though, the search engine is still not the system of record for the core data, which is still the Oracle based CMS which did not scale up enough to serve the API. There was some discussion at the talk about search engines that do support persistence (the D in ACID databases), something Lucene used to have a bad reputation for. My view here though is that, while actually making fsync work properly is a good thing, and you should not buy software that cannot recover from crashes, persistence involves a lot more than this now, such as replication, audit, versioning and access control. Building this directly into search products is a mistake. Another issue is that search engines are denormalized, and data stores of record should really be normalized to a large extent, to minimise the amount of data to be replicated.

There are two approaches that should work instead, however.

The first is more or less the current approach, to use the search engine as an index to a persistent store. I really like this approach if we follow it to its logical conclusion, which is that the persistent store in this type of application architecture should not be a relational database, but it should be a document store, that is a file, an HTTP resource, a document in a NoSQL document database, or an object in a replicated cloud storage system like S3. Modularize the database application, and split the persistence function from the index function. The persistence function provides the durability, versioning and audit and access control, with replication, backup. This can update the search index, and potentially any other types of index, such as a graph database for querying relationships, potentially even a relational database if that is the best way of querying some aspects of the data.

Obviously there is a potential consistency issue, if updates from the document store happen slowly, so potentially there is an eventual consistency model. Historically search was a bad offender here, as dynamic updates were not the norm and everything was batched into nightly updates, but that is going away and dynamic updates are more normal for search indexes. In principle you can have more consistency, especially in an architecture where there are fixed releases that can be consistently indexed, rather than distributed rolling updates, you choose your architecture and take your choice. Small consistency lags rarely matter in a lot of applications.

So you end up with an architecture with a well defined persistence layer that is not a relational database, and a set of indexes appropriate to the application, almost certainly including a full text search engine, but perhaps a graph engine too. Maybe you run consistency checks on your indexes for peace of mind.

The second approach is to see that search engines were some of the original NoSQL data stores, building custom storage and indexing engines, because they had such difficult problems. Indeed Google’s BigTable, and so the ancestry of a lot of NoSQL products came from search. However the search engines around now have not yet refactored themselves on top of the NoSQL engines that have emerged from this work, although this is starting with Lucandra which is Lucene persisted in Cassandra, which looks promising, offering seamless replication and distribution, and HBasene, an HBase Lucene backend. These make a huge amount of sense to me, as if you are developing sophisticated search algorithms, not having to build the whole index and persistence layer as well is a big advantage, as well as the scale out potential. Of course this approach does not conflict with the first one, in fact you could choose a NoSQL backend that is aimed more at read performance than persistence, and at storing small index values fast. The hard bits with this are that the search engines have specifically customised their data storage for the particular use cases, and reworking this onto a more general backend has few apparent advantages; as you can see from the examples above, most of these changes have come from people already using the backends in question and who want a single database to manage all their data requirements, particularly once they are working with high availability and replication. Software modularity really is not at the right level yet is it, I blame object oriented programming for this lack of reusability.

Anyway, back to the main point. For applications like content management, an architecture based on a content store that deals with persistence, versioning, access control, replication, with a set of indexes based on search engine techniques, graph databases, and anything else your applications needs. Ideally the indexes are all based on a common set of low level primitives so the backend can be swapped out or shared between the search store and other application specific indexing requirements, so there is a single low level indexing infrastructure that can be available as a common scalable service, with different implementations available. This type of architecture is quite buildable now, and is certainly used in quite a few applications, and I think it will become much more widespread, particularly in the cloud where it seems more natural, certainly for many types of application that fit into a document type model, such as content based applications.

Data segregation and access control are also important topics, but I am not going to cover them much in this post, as they are orthogonal. If your code is not secure, you can be pretty sure that there is a risk of your access controls being subverted, especially if the application is monolithic. There are interesting issues in how to manage data segregation, and how it is stored.

What are the threats we are trying to mitigate? If you are building a PAAS platform, something like Heroku, which quite a lot of people seem to be doing after Heroku got bought for $212m by Salesforce, then your entire business model is to run untrusted code from people you don’t really know. They may have intentional security holes, or accidental ones, which they may not even know about, and they may be looking to attack you. One recent example is the case of PHPFog who had their entire infrastructure taken over.

If you are developing SAAS applications you may be less worried, after all you just have to develop a secure application surely? It turns out to be a bit more complex than that. First many more complex applications do allow user code to run, or want to allow this, for customisation; at this point your application starts to become a platform. Aside from that, most applications process some form of external data that could have security issues. There have been widespread security holes in most media processing code, PDF, zlib, images and so on, causing buffer overflows and arbitrary code execution. In addition your applciation code could itself have bugs that allow remote code execution, or disclosure of data for the system tenants. As a service provider your reputation relies on optimal protection of the users data.

So the basic idea is of course that you build your system out of components based on their risk, and requirements, on the least privilege principle, where you try to put them in a sandboxed environment where they can do as little as possible. Obviously you do not have to sandbox at all, or you can choose a model with risks, but in an increasingly hostile world it is at least worth knowing what the better options are and how they could be built.

Virtualization

Virtualization is a key tool for user isolation, keeping them off real hardware and in a self contained environment which just looks like a computer. Of course you need an appropriate firewall as well, as there will be network access, and you probably don’t want it to be indiscriminate, just locked down to what is necessary to provide the service. The main issue is if you are running on a virtualized service such as Amazon EC2 that limits the smallest VM you can have, and hence sets a floor on your charges and profitability for small users of the service; this may or may not be a big issue for your application.

Pros: good isolation as cannot see anything else on the computer. Cons: heavyweight, as each instance needs a kernel and at least a skeleton bootable OS plus a fair memory overhead; not nestable with any performance (you have to use UML), so no use if you already run in a virtual environment; careful network firewalling necessary as you cant just pass sockets for communication for example.

Interpreters

Running code purely under an interpreter seems a very safe option, and indeed it is if you deal with a few risk factors. First, you need to make sure that the language has no libraries that can load or execute unsafe code. Some languages (like Lua make this easier than others. Always whitelist not blacklist features. For real security you want everything running in the interpreter, otherwise you are in the situation where you may be calling say a native image library with security issues. Obviously there is a big performance hit, so this works best for smaller pieces of code, places where none of the other isolation methods are appropriate, or as a temporary measure before introducing more sandboxing. Once you introduce say a JIT compiler you probably need to isolate the code, as a JIT compiler has to be able to make writeable memory executable, which makes attacks much easier.

Pros: very secure in the right situation. Cons: performance; non-interpreted code that is called may have security flaws that need sandboxing.

Managed code (Java and .Net)

These bytecode JIT compilers with their own sandboxes are a practical hybrid between Interpreters and native code validation. They are however complex systems, and Java in particular has had a number of vulnerabilities (for example CVE-2008-5353 to pick one at random), as has .Net (eg MS10-077) so cannot be considered to be a complete security solution.

Pros: vendor support, widely tested. Cons: complex environments increase risks; unclear how finegrain access controls are.

Native code validation

The Google Native Code browser plugin (NaCl) uses a code validator to check the binary code. There are some restrictions on the code to make it checkable (code generation, such as a JIT has a special interface, there are alignment constraints and other details, necessitating a modified toolchain). This is a similar approach of course to that used in Java bytecode, but with the more difficult problem of native x86 machine code. This sandbox provides an interface layer, somewhat like the operating systems system call layer, but more restricted. In addition this whole set of code is wrapped in an outer sandbox as well, using chroot and pid namespaces.

Pros: supports very general code with little slowdown. Cons: needs code to be targetted for it; aimed at computationally intensive code rather than IO based code; not easily portable as the security model depends on architectural features, although it is portable between operating systems; risk of incorrect validation.

Chroot plus

The Unix chroot call on its own, which isolates a process into a part of the filesystem is not very secure, it is easy to get out of it using the ptrace command on another process that is outside the chroot. Running each process as a different user helps here, as then the process will not have another process it can attach to with ptrace as it will not have permission. There are some potential race conditions in setting the new user that may cause an issue here. An example of this type of sandbox done well is Plash.

Pros: portable across unix versions. Cons: hard to do securely; cannot restrict network access.

LXC

Linux containers, unlike the equivalents in BSD and Solaris, are really a set of namespacing tools for different aspects of the system, set when a process calls clone. It can be viewed as a better set of tools to do chroot style isolation as well as a same-kernel virtualisation model; the tools support both running a whole system with startup scripts etc, or just a single process. Because it has been developed incrementally some of the support is new and if you want to run a whole system I recommend using something very new: Ubuntu 11.04 works nicely and has an lxcguest package, but I had odd issues with 10.10 not being fully isolated, although these might have been configuration. If you use it for single process isolation it is more straightforward as you can have a much more minimal environment. Currently the items that can be namespaced are process IDs, so your process cannot see or ptrace anything outside the container, file system, so it has its own mounts, network so it just sees a virtual network that can be firewalled as appropriate or can be passed a physical network adaptor exclusively, UTS so the process can see its own hostname, IPC for the SYSV IPC namespace. Coming soon is the addition of addition of the user namespace, so that new containers can be created by non root users.

Pros: process isolation done properly; allows controlled network isolation; still allows passing of file descriptors unlike virtualisation. Cons: only supported on newer Linux versions for some of the features.

Chroot/container hybrids

The current default Chrome Linux sandbox uses a mix of chroot and container calls, for maximum compatibility with common distributions; in fact it will work without the container calls but with reduced security. It uses chroot for filesystem isolation, PID namespace to isolate processes, disables ptrace with prctl (which is not a complete mitigation as this is reversible).

Pros: more compatibility, upgrades chroot model towards a container. Cons: network access unrestricted.

Seccomp

Seccomp is a very restricted security sandbox that has shipped with the Linux kernel for quite a while, that can be set using the prctl system call. It then only allows four system calls, read, write, _exit and sigreturn. This is very restrictive, as the process cannot even allocate memory, so it is rarely used. It also turned out to have a bug on 64 bit machines that allowed some other system calls. However Google did produce another sandbox for Chrome based on it, using a very restricted helper thread to perform memory allocations and other system calls. The thread is quite complex as it runs in the same process as the hostile code, so there are quite a few complexities, and it is not so clear that the particular solution as such works for general purposes, but similar approaches could be suitable for some problems.

Pros: small kernel whitelist with restricted additions. Cons: complex and architecture dependent code running in a difficult environment; may not be suited for all uses.

Ptrace

The ptrace system call, used for debugging, can also be used to sandbox a process, as it can intercept system calls. However it is beset with race conditions and other problems, and a hostile process can circumvent it. There do not seem to be any fixes at the moment.

Pros: portable. Cons: not reliable.

Selinux

The Selinux mandatory access controls, designed by the NSA, is a complex but very powerful set of access controls for processes. The big advantage from a sandboxing point of view is that the controls are enforced by the kernel, and are very fine grained, such as access to particular ports, files, sockets and system calls. Items such as files can be relabelled as they are processed, so for example you could not give users access to files before they had been validated or virus checked. Selinux adoption has been slow, with Redhat the first to really push it into their distribution, gradually being followed by others, but many users simply disabling it when it caused issues. Most distributions use it in the “targeted” policy, which only puts external facing daemons in a controlled state, and lets normal users do everything they could do before, but gradually more types of policy are being added, such as a user sandbox to run untrusted code. There is an extensive reference policy which the distributions base their code on which is a good reference for detailed customisation. It is also possible to push selinux controls into applications, such as Postgres, and to use it to store user validation through an application.

Pros: fine-grained controls, kernel mediated; encourages modular architectures; encourages a security as code model. Cons: not installed everywhere; complex; another system description language; best suited to very modular architectures; needs to be maintained with code, or people may just disable it to make applications work; some performance hit, estimated at 7% but obviously very application dependent.

Mitigation techniques

I have included these, although they are not a whole sandbox, because security is layered and they can be used to increase security in a sandbox that has some potential risks. There are a lot of potential techniques here, so I won’t cover them all. Address space layout randomisation (ASLR) is one technique, making it harder for an attacker to know where parts of the executable that they need to call to create an exploit are. This requires position independent code (PIC), and has some default support in Linux, but more is available for example in the PaX project. Another option, also supported by PaX, is to disable the ability to make writeable memory areas executable, which makes it impossible to inject new executable code into a process at runtime; this ability is however required by JIT compilers, something that caused issues with Javascript when it was introduced recently in iOS. Another area is stack buffer overflow prevention, for which there is now gcc support. These policies have rarely been used when compiling entire Linux distributions, with the notable exception of Hardened Gentoo, although they can also be used for individual applications.

Pros: adds more protection at little cost. Cons: only a mitigration, not a sandbox; some binaries need these capabilities for valid reasons.

Conclusions

As is probably clear from this brief summary, security is not just a simple compiler flag, it is a complex design process with a lot of work to do. It is an architectural issue to a large extent, as the more self contained your units are, the easier it is to use the least privilege principles, as for operating system controls the process is the unit of privilege (remember the Unix philosophy). Tooling and testing is fairly limited off the shelf, and debugging can be more difficult, so the overall cost of security is not negligible. On the other hand the cost of not implementing security is very high, particularly in the case of SAAS platforms where the industry is being held to a very high standard.

Which methods to choose? For a SAAS or PAAS multitenant platform, where the base OS is entirely under your control, some combination of lxc, selinux and other mitigation techniques seems to be a clear winner. This can be enhanced, starting with either lxc or selinux as a base and then adding more protections and more fine grain seperation of processes as things move on.

I am currently available for employment opportunities, so if you are looking for someone in architecture, operations, development who is interested in issues like this get in touch.

A lot of things are changing with the way we buy, sell, write and use software at the moment. We do live in very interesting times. The big umbrella movement called cloud is one of the most interesting things; for example there is a huge shift to on demand software and hardware, ideal for the agility it enables, as well as disintermediating expensive sales processes.

Hardware started commoditising a long time back, with the huge economies of scale in chipmaking and computers, but wheat has really hit is the 2005 shock when clock speeds hit the ceiling and Moore’s Law just went into more cores. Scale out was the only real option then for performance, although JIT compiler technology has been a hot topic, and if you program in some dynamic languages like Javascript you might be forgiven for thinking computers were still getting significantly faster. Simultaneously simpler lower power devices have been shifting up the performance curve, reaching a point where some people are starting to seriously build ARM servers, and mobile devices are reaching useful performance for wide ranging tasks.

Many thought that at this point we would be programming SMP architectures with shared memory models, but while you can now get systems with very large number of cores in the SMP/NUMA world, but these are very expensive systems, and little software outside specialist areas. The majority of multisocket systems now are probably virtualized to appear as multiple single socket ones. Scale out, message passing, commodity hardware are mostly where things are, with a sideshow of commoditized GPU technology giving us another parallelism to work with, or another competitor to the ageing x86 architecture. China is building its own MIPS64 based CPU too, designed to run Linux, more competition to lower prices. Computation, networking, storage are all getting very cheap.

I replaced my last dual socket 2GHz powerpc machine with a new dual core machine thats a fair bit faster but about 10-20% of the power consumption, and now almost all the ten or so computers around the house are small, low power, and dedicated to one function: netbook, 2 mobile phones, ipad, VOIP phone, router, storage, laptop, a couple of small computers. The highest power consumption is probably the old laptop, a last reminder of the old Intel that used to make chips up to 100W. It makes no sense for me to invest in large amounts of computer power in the home, because the bandwidth is not there: it takes too long to get larger datasets here, so I try to do computation where the data is and where there is bandwidth, so I outsource the computing.

When Douglas Parkhill wrote The Challenge of the Computer Utility in 1966, while working at the sinisterly capitalized MITRE, the reason for people to be interested in computers as utilities was the expense of actually owning a computer, which cost around $2m at the time, but could be rented at $450 an hour. But the economics of treating computers as a utility, improved utilisation, availability on demand are just as valid now, as well as the ability to shorten development feedback times and share common source code which he also cites. Utilities are largely centralised for reasons of economy of scale, ease of balancing changing demand, and ability to provide large quantities. Obviously some people have their own generators or solar cells and water supplies, and some people and companies sell power into the grid, but the majority comes from large sources, using varying technologies.

Other than PCs, which are increasingly mobile, most businesses don’t keep a lot of computing power on premise, most of it is in datacentres. And those datacentres are increasingly getting larger and further away, as the cost of power and real estate pushes them to cheaper locations. And they get bigger for cooling efficiency reasons, and start to turn into billion dollar projects, vastly more expensive in real terms than the computers of 1966. On demand computing suddenly starts to make sense for almost all businesses who do not wish to get into this type of expenditure. It also suddenly makes computation costs transparent, in terms of how much computation or accuracy is cost effective for a particular task, even if killing fixed cost budgeting is going to massively disrupt the financial planning of enterprises (another change that will need more computing power to manage).

What does a computer utility look like?

The early days of utilities are dominated by several issues, one being standardization, whether it be AC frequency and voltage, another being technological change, as from hollowed out trees, to clay pipes to plastic in water transmission, or financial boom and bust of the railway industry as the level of demand and supply were balanced out. Competition between new utilities and old ones was also significant, railways vs canals, roads vs railways, gas lighting vs electricity, hydraulic power vs electric power, and regulatory and legal issues from compulsory purchase, nationalisation and especially price regulation.

In the end though, the centralized delivery model means you end up with standardized services being delivered, you cannot choose the voltage and frequency you want your electricity delivered (other than wholesale bulk offerings, like three phase), the temperature or pressure of the tap water, the frequency of the trains, or the level of congestion on the roads. Utilities often have slightly odd externalities and natural distribution monopolies that vary from one to another.

Computer software architectures are very non standardized right now, as we are in the craft period of software. There is a lot more standardization than in 1966, when there was less mass production, and LSI was only just starting. Common abstractions have been slowly appearing, virtual memory being an early one in the B5000 in 1961, hiding the memory hierarchy in a way that simplifies a lot of code, even if other code needs to understand the abstraction for performance; that was around the same time that high level languages were being developed with mutable variables that matched the semantics of virtual memory. The next big abstraction, from the late 1960s, was the Unix (almost) everything is a file model, a universal stream processing model for many types of data and connections with a single software interface, which also allowed compositional software. In 1970 Codd created the relational data model, an abstraction for databases. It was a productive time, in which Paul Baran also invented the packet switched network made of unreliable components, another key building block of the cloud architecture. In the 1970s we had the language virtual machine, starting with the UCSD p-System which heavily influenced Java, the beginning of virtualisation of hardware and operating system characteristics that has had some, although still arguably limited, success and standardisation.

The characteristics of a utility architecture are those embodied in the internet: distributed, failure tolerant (up to a limit of course, we still get water and electricity outages, with very weak SLAs), standardised interfaces (plug sockets, HTML), usage agnostic, scalable. In particular the six constraints of the REST architectural style, client-server, stateless, cacheable, layered, optional code on demand, uniform interfaces currently give us our best architectural model of a scaleable, on-demand system, and so our best model of what the abstractions of a truly utility infrastructure will look like. Another characteristic is of course the use of open source software, which represents both the commoditisation of infrastructure software, and the right price for utility and on demand scaleable software.

Current environments

If we look at Amazon web services, the closest thing so far to a utility computing service (although with some transitional facilities bolted on), the storage abstraction, S3, looks pretty much like what we would expect from the REST style, resource based, supporting the HTTP concurrency model (etags, conditional updates); note it looks nothing like a POSIX file system, the previous abstraction. In the Amazon model, this is the only really persistent, reliable, scaleable, global storage. Other services like EBS are not fully reliable but can be snapshotted to S3, while SimpleDB is not really a database in the persistence sense, as it has severe limits on size and is not globally available; it is better seen as an abstract index, fulfilling that part of the database requirements without the persistence. Other storage such as local disks is even more ephemeral. I have another blog post in the works on how best to use these persistence models for real applications.

The network architectures in Amazon are similarly what you would expect for building REST architectures, with load balancers and an HTTP cache layer for example. In contrast, Amazon’s EC2 provides a Xen virtualised PC as the base abstraction which is pretty software architecture agnostic, compared to say Google App Engine, which provides an abstracted language runtime. The more general platform allows more experimentation, and crucially more legacy transition. We have a lot of historic usage of operating systems, security models, coding habits and productivity issues that make using “opinionated platforms” difficult.

In many ways though, whether the API provided on a utility compute resource is the Java servlet standard as in the GAE Java environment or virtual network devices as in EC2, thats really a middleware question, and a question of the amount of environment you have to wrap around your code. The Amazon approach encourages people to repackage services for different language communities on top of their offering. The important parts of the architecture are the parts that support failure, scale, reliability. The most important one of these, often neglected, is explicit management of state. The server itself must be stateless if it is to be failure proof. All state must be managed in the persistent storage abstraction, and software upgrades by starting up new machines, and upgradeable interface specifications, another REST idea. This is a big cultural leap for people, apart from not being used to actually accounting for every data change the program makes, sysadmins are used to patching systems at runtime, rather than starting up new, tested, instances when code needs to be updated. Unless state is explicit it is not testable or reliable, or known, or recoverable. These things were discovered of course in the Erlang community where backups are a sign of poorly designed systems that have not covered all the failure cases properly. More on this side of things in other posts soon too.

Once you have realised that there is only scale-out, all state must be explicitly managed in state abstractions, and failure must be designed for all over, the container shape makes much less difference, as your software architecture will be very similar. Sure there are points of difference, packets versus streams say, sync vs async, different security architectures, different programming languages, protocols, serializations and libraries. The network is really the only important IO point for the utility computation; the server environment can be much simpler in principle than the complex server environments we run predominantly now, and thus able to scale better on more and more commodity hardware, highly parallel and low power that we were talking about before.

Beyond Compute and Store

I said before that Amazon SimpleDB was best viewed as an index not a database, and lots of the software we use in a web environment is like that, key-value stores, memcache, and some of the NoSQL solutions are providing this type of function, not concentrating mainly on persistence; obvious product areas here are things like full text search which are not the ultimate data source. Others are trying to provide “small item persistence”, although most of these systems are not yet being aimed at utility computing provision, so they have not tended to cleanly separate store and index for example, and persistence is mainly aimed at hard drives. Given the historical popularity of databases, and the current flourishing of the NoSQL movement, I can see an opportunity for something like Cassandra with an S3-type backend as a utility computing key-value store. Why not SQL? The main problem is the lack of scalability of JOINs; making it difficult to scale as a true elastic utility service. In addition, I can see a place for a low latency reliable transaction log service built on replicated SSD that shifts over automatically to permanent long term storage (please Amazon), to cut the persistence time, especially for small data items.

Other facilities for utility computing include authorization infrastructure, message queues, payment infrastucture and so on, blending into more specialist services sold as SAAS on the infrastructure.

Beyond Now

Our current code model looks a bit like running code inside a web server; indeed that is one way to do organize the code, or you can reverse this and run a web library inside the language library, like Node.js does for example. Architectures could change though, perhaps (sometimes) moving the code to the storage for lower latency; the advantage of strongly decoupled code with explicit state management is that it is much easier to move it around; explicit state management with all lasting state over the network has the same properties as functional programming, indeed as we said before is very much like the Erlang functional process and message passing model. This flexibility in where code runs allows another level of infrastructure virtualisation and efficiency, moving code towards requesters or data based on latency and throughput requirements.

I don’t think this is going to change how programming works a lot, well at least it should be pretty familiar to web programmers, might be some adjustment for the enterprise lot of course. What is going to change is sysadmin. One of the issues now with cloud solutions is they require too much sysadmin to really allow us to scale up our usage. Jevon’s Paradox cannot come into effect and let us expand capacity as price falls if there are additional costs such as admin overhead. The infrastructure providers have managed to cut staff to less than 1 per 4000 machines, but the users of services like Amazon’s still have a lot of software environment management overhead. I think we can cut that too, substantially, using stateless servers and other changes, but that is another blog post.

This is perhaps a bit of a run through history, trying to make sense of the journey towards utility computing that has been going on for over 45 years but is really just starting to become mainstream, and where we still have the opportunity to build a new utility, which is something that does not happen very often. Some more concrete explorations coming soon.

Drupal (unofficially) competing in the BostonPHP Framework Bakeoff from New Leaf Digital on Vimeo.

The challenge

So first here comes the challenge to other CMS vendors and system integrators: spend three hours (or less) on repeating this using your system. You get an hour or so for planning, based on the wireframes. Note there are some incompletenesses in these (typical client!), like what the tag pages look like, so improvise, and some explicit open decision points (“require some sort of password”), so use whatever is easiest. You get up to an hour on the build (points given for less!) The aim is not a polished final shippable product, but a first alpha or beta. Then you are allowed another hour for annotating the screencast , putting the site online for us to look at etc, if you need it. You can do the annotation as you build if you are pushed for time and cut the planning and you should be able to get the whole thing done in an hour, so no excuses for anyone not to have a go.

Basically the site is a simple job board, in a circa 2005 Ruby on Rails style, so there is nothing that a decent web content management system should not be able to do out of the box. If there are bits your system technically finds hard its probably worth fixing them (the Drupal video reveals some small things that are apparently fixed in version 7 for example). But get it close enough, as really the aim is to give people an idea of how usable your system is for this sort of build task, the approach you have to take, and how quick it is! If your data model requires something a little different, by all means explain and make something close. There is no attempt for example to impose a REST architectural style and support an API that supports PUT and DELETE on jobs, but if that comes out of the box or trivially with the CMS you use, by all means show it off and put a bit more 2010 into the solution!

And no trying to get out of this, or I will write up your CMS as “pretty useless compared to Drupal” or your web CMS integrator as “rubbish, use New Leaf Digital, they have balls”. If you want to do it live as a webcast then feel free to invite me along (but record it too).

The bit about frameworks and CMSs

It is an interesting question though, whats the real difference between web development frameworks and CMS systems. Indeed, if you think that web frameworks are nicer in every way, feel free to enter the challenge above using your favourite framework.

My current thinking is that a CMS is best defined as a web development framework plus an IDE that persists configuration to a content repository or database. By all means disgree with me!

By web development framework, I mean something that gives you a full web server, front end and backend integration tools, authentication and access control, templating and so on. A CMS has an IDE in addition, a web frontend that lets you configure it, define datatypes, forms, layouts etc. But generally, unlike say visual code editors which persist to editable files, the results of these changes in the backend are persisted to the same content repository as the actual content, either as database entries or schemas, or as tree nodes (as in JCR based systems). Hence for example these changes do not generally end up in a source code control system, although they may be versioned in the way content is, or they may not as for example Drupal CCK schema changes change the underlying database, while with a standalone framework you might have to script schema changes for upgrades in the traditional way.

Also the issues in deployment with CMS products due to the use of a single repository for content and config are clear, even if many systems do try to make them separable by various means; the Drupal example shows this to some extent as all the frameworks have their code available on github but to set up your own instance of the Drupal example you would need some code, and a database dump which you would then have to filter the content in the example instance from.

This also makes it clear why CMS systems are usually very tied into one framework, as the IDE has to embody a lot of implicit knowledge of the data model and the way the framework wants to do things, as well as how to persist changes. Designing a more decoupled approach is much more difficult, although perhaps possible (thats another blog post to come, the decoupled CMS). But the streamlined coupling makes solving problems that fit the data model easy and quick, as the challenge above demonstrates. Of course if the type of problem does not match up to the the model that the CMS applies, then it comes down to trying to force a teapot into round hole.

In the beginning the web was editable in the browser. Tim Berners Lee made it so. But this did not last for long. Eventually Microsoft restored this, to some extent, with the contentEditable properties and related Javascript interfaces. Now these have a lot of foibles, and the current HTML5 standardization has not done a huge amount of work here yet, certainly not adding new features or changing the basic interface substantially, so a lot of behaviour is currently underdefined, which makes cross browser compatibility more difficult. While the main desktop browsers all implement the interface, mobile ones do not in general yet, even on form fators such as the iPad, which could very well be an effective content editing device.

Should you use one at all?

Almost every CMS supports wysywig HTML editing, but it is not entirely clear how much they are used. To be quite blunt, they have always been among the worst editing environments ever devised, overlaid with a poor HTML model and a tendency towards presentational markup. Most users do their editing elsewhere, at least for originating content. I think though that we are at the point where this can be improved.

The first thing is to kill the presentational markup, and instead support customised semantic use of classes. On the basic level this is easy to implement; there is probably more work to easily support more advanced HTML templates, for example to mark up recipes or other specialist content in consistent semantic ways through this type of interface. Pluggable schema guidance might be needed for this.

The second is to make the editors themselves sane. First step is of course local offline HTML5 storage, so that drafts are still there if you wander off, your computer runs out if battery or whatever. Then just make them nicer to use, keyboard friendly and so on.

Then there is the concurrency question, you may want to show concurrent editing, and the versioning model that applies. This does not have to be the Google Wave style simultaneous editng, although that is one option. People may be editing content for release in different versions.

Import from other editors is of course still important, and making it as painless as possible for the common tools is of course important.

Also, remember it is an editor, people like a choice of editors, so loosely couple it, make it standalone and easily changed.

What not to use

Old school implementations, with nothing now to recommend them that I can see; often they seem to need server side components too, which should not be required now when pure Javascript solutions should be sufficient.

1. Open Wysiwyg

   Not off to a good start here as the demo does not work in Chrome, works in Opera and Firefox. I guess it is not really in development then. Does not create paragraphs by default, just uses <br>. Still believes in web safe colours and fonts. A classic example of the old school, but not up to date. Open source.

2. CKEditor

   Much more promising demo, . Paragraphs are real (uses a &nbsp for blank ones, which is I suppose ok), there is a paragraph style dropdown that manages paragraphs, headings and divs. The style dropdown is a bit disappointing as it sets styles not classes, and you can still set fonts and so on. Works in Chrome, Firefox and Opera. The old school done right. Open source.

3. TinyMCE

   Ah yes, the classic one. Feature set similar to CKEditor, in taht it treats paragraphs the same, has a dropdown with predefined styles, that are style tags not classes. Seems a bit clunky and flaky in comparison, and has odd terminology, since when has a div been a layer? Both have a smiley insertion feature that suggests that people have been wasting time on pointless stuff rather than reconsidering how these programs should work. Open source.

4. XStandard

   Good blurb about supporting CSS properly and the importance of markup and presentation separation, but in a bout of massive fail there is no demo on the website. Your reviewer has no intention of installing it to test it, especially as it does not support Linux as a platform in a second fail. Commercial.

5. InnovaStudio

   No demo either, apart from animated screenshots. Why are these commercial companies so full of fail? Web 2.0 products with no web trial? Seems obsessively to be about dimensions and fonts from the features list, not about separation of concerns.

6. ActiveEdit

   ActiveX DHTML with Java fallback for other platforms. Need I say more? Commercial.

7. contentEditable

   Well at least here the demo is the homepage. All inline with a simple-ish mechanism to mark blocks as editable. Stupidly though, while it appears to be non modal, all the menus are modal popups, which completely destroys the usability.

8. widgEditor

   Very simple editor. Adds divs not paragraphs around everything. Open source.

Good implementations

Largely based on contentEditable, fixing up the inconsistencies, although Dijit is based on the less flexible designMode which means it has to run in an iFrame.

1. YUI editor

   Note there is also a YUI 2 version. In many ways the demo is not much use, as of course this is part of the YUI framework and needs some Javscript infrastructure to customise it effectively. There is also a YUI2 version demo, and linked demos of plugins. Much better than anything above. Open source. Iframe based for compatibility with the grade A browsers.

2. Dijit

   This is the editor that comes with the Dojo widget set. Still iframe based, not a full contentEditable implementation.

3. Aloha

   Some nice demos available, indeed probably the best demo site of any editor. Aims to fix up the issues and cross browser quirks contentEditable, and looks nice too. Open Source.

Other ways 1: Canvas

A survey of what is up is not complete without mentioning Skywriter, which is a (code) editor written entirely in canvas. As far as I can see this is an utterly pointless approach long run. It is especially a bad idea for HTML, where fitting in with the CSS of the page matters, so lets ignore this.

Other ways 2: pure Javascript

You can in principle implement the whole of a contentEditable-like editor in pure Javascript, using a little div as a cursor and everything, no help at all from the browser. That is a lot of work, of course. I know of two implementations, one being the 2010 release of Google Docs, which explains why the did that, in order to be able to provide the best cross browser support. The other implementation I know of is the one for Squiz CMS, no online demo, not open source and tied into one product. This is pretty functional although it had some browser compatibility issues last time I used it, and ther website still says it does not support Webkit.

Other ways 3: Markdown and Wiki markup

I have to say I use Markdown a lot, this blog is generally written in it. There are two issues I see. One is that it is limited, and while you can add native HTML, this often means everything has to be redone in HTML (trying to add anchors to a list for example). There are extensions with more features, but then you get into compatibility issues. ALso there is no sanely reversible transformation that will generate the sme output as there are notational choices, so it is generally best to keep the content in Markdown all the way through. There are much the same issues with Wiki markup, although it is extensible too, which causes more issues of trying to remember constructs, and compatibility is weakened. The difficulty is that HTML is not a very good markup for humans to write, but once you get past the very restricted domain that say Markdown handles, producing something that works well is hard, maybe impossible.

Other ways 4: Don’t use HTML

A number of people I have worked with, and some content management systems go with a very minimalist use of HTML, pretty much text is paragraphs, everything else is structured fields. A slightly enhanced version of this, effectively a very minimalist tiny markup, corresponding to even less than Markdown is the COPE system.

Personally I believe we need to expand the use of rich text, for many use cases “just the words” is not enough, and we need equations, images, notes, asides, sidebars, tables, captions, and all the richness of rich text, so I don’t think this is viable for much serious writing, and we need to expand from the dumbed down web that this enforces.

Concluding

While the wysiwyg editor as it has been in content management systems is pretty awful, I think we need to expand the principle and try to do better, go back to the really editable web, and support well structured semantic HTML in rich ways, not forms and simple text boxes. There is a lot to do to get this working though still.