As I said in the talk, D3 is really great to use, as it is about code not configuration, and lets you really do what you want. It is purely based on working with data and the DOM, without other abstractions layers. I hightly recommend it!

One thing I mentioned in the talk was using Google spreadsheets for storing arbitrary data that doesn’t already have somewhere to live. I have found this worked out quite well with clients who want an easy “CMS” for data that they can update and feed straight through to graphs and maps etc on the website. A few people asked about this, so I thought I would give a quick guide.

I have made a public Google spreadsheet for these examples. Will probably get messed up as it is public!

The easy way to use this is just to use CSV, as once you make the spreadsheet public, the interface will show you the CSV URL (of the first sheet only, regardless of the setting). In this case the link is https://docs.google.com/spreadsheet/pub?hl=en_US&hl=en_US&key=0Agr9u3lfdOGHdHZoc1dVYXhDdl9GSk5SbVRBSGZYRFE&single=true&gid=0&output=csv which gave me the following:

date,colour,value,genre,french
2011-01-01,red,2,rock,roches
2011-01-02,blue,1,grunge,grunge
2011-01-03,orange,66,pop,pop
2011-01-04,green,4,metal,métallique
2011-01-05,pink,6,jazz,le jazz
2011-01-06,brown,5,punk,Punk
2011-01-07,purple,33,rap,Rap
2011-01-08,black,12,reggae,reggae
2011-01-09,white,11,alternative,alternatives
2011-01-10,gray,5,dance,danse
2011-01-11,violet,14,country,pays
2011-01-12,yellow,9,blues,le blues
2011-01-13,indigo,10,funk,funk

This is clearly usable in D3, but it does have cross domain issues in most browsers, so you would normally have to proxy it back to your domain if you are calling via Ajax.

The options in the “publish to the web” menu for giving you URLs are HTML, CSV, text, PDF, XLS or ODS, none of which are what we really want, which is JSONP. However there is a JSONP option, it is just a very different URL, which is designed to be got through the API. The problem with the API is all the calls are authenticated so it is a real pain to just extract a (public!) URL. After hacking around with the Python client example code, I found a (messy) workaround however.

Every spreadsheet on Google has an ID, and each sheet within it also has an ID. These do not correspond to the keys in the public access links. You can get the spreadsheet ID from within the Javascript API available for spreadsheets, as SpreadsheetApp.getActiveSpreadsheet().getId(). You cannot get the sheet ID, but the first sheet you create always has the ID od6, the next od7, and then some other values that do not seem to change. So to write a little function that gives you the spreadsheet JSONP URL, do the following:

<

ol>

function getURL() {
  return "http://spreadsheets.google.com/feeds/list/" + SpreadsheetApp.getActiveSpreadsheet().getId() + "/" + "od6/public/values/?alt=json-in-script&callback=";
}

If we open that URL with a callback of test we get back:

// API callback
test(
{ "encoding" : "UTF-8",
  "feed" : { "author" : [ { "email" : { "$t" : "justin@specialbusservice.com" },
            "name" : { "$t" : "justin" }
          } ],
      "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
            "term" : "http://schemas.google.com/spreadsheets/2006#list"
          } ],
      "entry" : [ { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: red, value: 2, genre: rock, french: roches",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "red" },
            "gsx$date" : { "$t" : "1/1/2011" },
            "gsx$french" : { "$t" : "roches" },
            "gsx$genre" : { "$t" : "rock" },
            "gsx$value" : { "$t" : "2" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cokwr" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cokwr",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/1/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: blue, value: 1, genre: grunge, french: grunge",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "blue" },
            "gsx$date" : { "$t" : "1/2/2011" },
            "gsx$french" : { "$t" : "grunge" },
            "gsx$genre" : { "$t" : "grunge" },
            "gsx$value" : { "$t" : "1" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cpzh4" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cpzh4",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/2/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: orange, value: 66, genre: pop, french: pop",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "orange" },
            "gsx$date" : { "$t" : "1/3/2011" },
            "gsx$french" : { "$t" : "pop" },
            "gsx$genre" : { "$t" : "pop" },
            "gsx$value" : { "$t" : "66" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cre1l" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cre1l",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/3/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: green, value: 4, genre: metal, french: métallique",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "green" },
            "gsx$date" : { "$t" : "1/4/2011" },
            "gsx$french" : { "$t" : "métallique" },
            "gsx$genre" : { "$t" : "metal" },
            "gsx$value" : { "$t" : "4" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/chk2m" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/chk2m",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/4/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: pink, value: 6, genre: jazz, french: le jazz",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "pink" },
            "gsx$date" : { "$t" : "1/5/2011" },
            "gsx$french" : { "$t" : "le jazz" },
            "gsx$genre" : { "$t" : "jazz" },
            "gsx$value" : { "$t" : "6" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ciyn3" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ciyn3",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/5/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: brown, value: 5, genre: punk, french: Punk",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "brown" },
            "gsx$date" : { "$t" : "1/6/2011" },
            "gsx$french" : { "$t" : "Punk" },
            "gsx$genre" : { "$t" : "punk" },
            "gsx$value" : { "$t" : "5" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ckd7g" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/ckd7g",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/6/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: purple, value: 33, genre: rap, french: Rap",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "purple" },
            "gsx$date" : { "$t" : "1/7/2011" },
            "gsx$french" : { "$t" : "Rap" },
            "gsx$genre" : { "$t" : "rap" },
            "gsx$value" : { "$t" : "33" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/clrrx" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/clrrx",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/7/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: black, value: 12, genre: reggae, french: reggae",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "black" },
            "gsx$date" : { "$t" : "1/8/2011" },
            "gsx$french" : { "$t" : "reggae" },
            "gsx$genre" : { "$t" : "reggae" },
            "gsx$value" : { "$t" : "12" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cyevm" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cyevm",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/8/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: white, value: 11, genre: alternative, french: alternatives",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "white" },
            "gsx$date" : { "$t" : "1/9/2011" },
            "gsx$french" : { "$t" : "alternatives" },
            "gsx$genre" : { "$t" : "alternative" },
            "gsx$value" : { "$t" : "11" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cztg3" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cztg3",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/9/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: gray, value: 5, genre: dance, french: danse",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "gray" },
            "gsx$date" : { "$t" : "1/10/2011" },
            "gsx$french" : { "$t" : "danse" },
            "gsx$genre" : { "$t" : "dance" },
            "gsx$value" : { "$t" : "5" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d180g" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d180g",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/10/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: violet, value: 14, genre: country, french: pays",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "violet" },
            "gsx$date" : { "$t" : "1/11/2011" },
            "gsx$french" : { "$t" : "pays" },
            "gsx$genre" : { "$t" : "country" },
            "gsx$value" : { "$t" : "14" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d2mkx" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/d2mkx",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/11/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: yellow, value: 9, genre: blues, french: le blues",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "yellow" },
            "gsx$date" : { "$t" : "1/12/2011" },
            "gsx$french" : { "$t" : "le blues" },
            "gsx$genre" : { "$t" : "blues" },
            "gsx$value" : { "$t" : "9" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cssly" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cssly",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/12/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          },
          { "category" : [ { "scheme" : "http://schemas.google.com/spreadsheets/2006",
                  "term" : "http://schemas.google.com/spreadsheets/2006#list"
                } ],
            "content" : { "$t" : "colour: indigo, value: 10, genre: funk, french: funk",
                "type" : "text"
              },
            "gsx$colour" : { "$t" : "indigo" },
            "gsx$date" : { "$t" : "1/13/2011" },
            "gsx$french" : { "$t" : "funk" },
            "gsx$genre" : { "$t" : "funk" },
            "gsx$value" : { "$t" : "10" },
            "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cu76f" },
            "link" : [ { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values/cu76f",
                  "rel" : "self",
                  "type" : "application/atom+xml"
                } ],
            "title" : { "$t" : "1/13/2011",
                "type" : "text"
              },
            "updated" : { "$t" : "2012-01-29T15:22:57.368Z" }
          }
        ],
      "id" : { "$t" : "https://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values" },
      "link" : [ { "href" : "https://spreadsheets.google.com/pub?key=tvhsWUaxCv_FJNRmTAHfXDQ",
            "rel" : "alternate",
            "type" : "text/html"
          },
          { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values",
            "rel" : "http://schemas.google.com/g/2005#feed",
            "type" : "application/atom+xml"
          },
          { "href" : "http://spreadsheets.google.com/feeds/list/tvhsWUaxCv_FJNRmTAHfXDQ/od6/public/values?alt=json-in-script",
            "rel" : "self",
            "type" : "application/atom+xml"
          }
        ],
      "openSearch$startIndex" : { "$t" : "1" },
      "openSearch$totalResults" : { "$t" : "13" },
      "title" : { "$t" : "data",
          "type" : "text"
        },
      "updated" : { "$t" : "2012-01-29T15:22:57.368Z" },
      "xmlns" : "http://www.w3.org/2005/Atom",
      "xmlns$gsx" : "http://schemas.google.com/spreadsheets/2006/extended",
      "xmlns$openSearch" : "http://a9.com/-/spec/opensearchrss/1.0/"
    },
  "version" : "1.0"
}
);

Now you may think that is odd JSON, but it is a literal JSON conversion of ATOM, which is why it has a lot of apparent junk. The bit we want is the array of objects in feed.entry which has among other entries

[         { ...
            "gsx$colour" : { "$t" : "red" },
            "gsx$date" : { "$t" : "1/1/2011" },
            "gsx$french" : { "$t" : "roches" },
            "gsx$genre" : { "$t" : "rock" },
            "gsx$value" : { "$t" : "2" },...
          }, ...
]

Bonkers, you may think. But it does work, and what you get is a reliable easy to use http addressible tabular data source. Really, the whole of Google docs is like this, half genius half biscuit. It could be a great service, but almost seems to be wrapped up in legacy already. It could be so much better, but improvement seems slow; the work seems to have gone into the actual editing side not the backend, which is also saddled with Excel compatibility goals. A lot of scope for competition in this space.

Anyway, I hope that helps. I really do think for many applications it is a great backend system, if you hide the complexity from the end users!

I actually had native IPv6 ADSL many years ago, with the now defunct Black Cat Networks, although at some point ipv6 routing just stopped working, and actually apart from being able to ping6 my friend who had the same setup it was not terrible useful, although I did test all my code and make sure it was all ipv6 ready. But now there are actually compelling reasons for IPv6, like the fact it is getting harder to get IPv4 addresses. In particular I have been spending some time on Linux containers (lxc), and if you want to run network isolated containers they need addresses, and if you are doing this on many cloud services you cannot get extra IPv4 addresses, so either you use internal NAT, yuk, or use IPv6. Also the first steps towards Amazon web services supporting IPv6 came out the other day, with ELB supporting IPv6 termination, although there is not yet support for IPv6 addresses on EC2 instances directly, although you can add tunnels; see my answer on Quora for the current status.

So how do you go about getting yourself IPv6 enabled? First this blog, which is currently hosted on Slicehost, which does not yet have native IPv6 support, although they will have later in the year, as part of their full merger into Rackspace. I keep wondering about moving to Linode as it is cheaper, although the way they have set up IPv6 seems odd to me.

Stage one is signing up with someone who will give you an IPv6 tunnel to an IPv4 address. I would recommend using Hurricane Electric, as they have global tunnel endpoints and a good service. Sign up here, and you can have 5 free tunnels. It is very simple, just provide the IPv4 endpoint, and your details, choose an endpoint near your server, and you will get given an IPv6 /64 address and some helpful instructions as to how to configure it based on your OS. They had an endpoint in Dallas where my Slicehost is, which has a ping time of 1ms, which is nice. I used the modern style ip setup, which I just added to /etc/rc.local so it is recreated on reboot:

modprobe ipv6 
ip tunnel add he-ipv6 mode sit remote 216.218.224.42 local 67.23.6.148 ttl 255
ip link set he-ipv6 up
ip addr add 2001:470:1f0e:6e6::2/64 dev he-ipv6
ip route add ::/0 dev he-ipv6

Very easy, no authentication or anything as it is all fixed by IP address. My IP is 2001:470:1f0e:6e6::2. Whats the /64 you ask? Thats a block of 18,446,744,073,709,551,616 IP addresses, minus the first and last which are not used. That is generally the smallest globally routable allocation, and should be find for an organization to allocate to their users and devices, including virtual devices and so on; it also makes it easy to autoconfigure devices, as they can use their MAC addresses or random numbers to pick an address within the allocated block, so configuration like DHCP is not generally needed.

Next we need to configure DNS records for IPv6, that is AAAA records in addition to the A records that give IPv4 addresses. If your DNS provider does not support AAAA records it is probably time to get a new one! I use Zerigo for my Blog domain, and it was very easy to add records pointing at 2001:470:1f0e:6e6::2.

After that you need to make sure your services are listening on IPv6. netstat lists IPv4 only listening sockets as *:*, while IPv6 sockets, which can listen on IPv4 addresses too as there is a legacy mapping, are listed as [::]:* instead. In my case, I had to change Nginx to listen on IPv6 too, with the config being listen [::]:80; instead of listen :80;. Restart the web server, and then you can use the IPv6 test site validator to check for you that everything is up and ready, and get your badge:

All very well, but you really want to test it yourself don’t you. And of course you have other software you need to test on IPv6, so you want to set up your desktop machine with IPv6 too. Time to make another tunnel. Now this is one of those cases where maybe you do want to try this at home. Or at least tell your sysadmin before setting it up in the office. With IPv6 machines are directly addressable on the internet, and the tunnel probably bypasses any firewalling. WIth all the IPv6 addresses, portscanning is really difficult, unlike IPv4, but you should treat machines as connected and keep them secure, which of course you do at home, but judging from the amount of IE6 around does not happen at work so much.

Now my home connection is ADSL with a not very good modem, that I have been meaning to replace with a proper one. The main issue is I don’t really know how stable my IP is, so just going to have to experiment. If I had a UK server I could make my own tunnel I suppose, but will see how I manage with a Hurricane Electric tunnel which I will have to recreate if the IPv4 address changes, possibly with a new IPv6 address too. My guess is the IPv4 address will be pretty stable, it was the same over a box reset just now. Also Hurricane Electric have an API call to change your tunnel address! They also have an endpoint in London, 20ms away from me.

First issue was that Hurricane Electric complained it could not ping my IP address. I looked through the config options on the O2 ADSL box (really a Speedtouch WL780) and there was nothing, so the internet found the answer. telnet to the box, login as SuperUser with password O2Br0ad64nd and type service system ifadd name=PING_RESPONDER group=wan, then saveall to write the config to flash and all will be well. There is a manual for the router online.

Next realization though was that I need to get the ADSL NAT to forward the encapsulated IPv6 traffic to my Linux box, as the router has no ability to do IPv6 itself. At some point I will move the termination to my nice ultra low power dual core 1GHz ARM Trimslice box. So back to the telnet and some helpful guides, all written in Dutch, thanks Chrome for the translations. Make sure you don’t translate the router commands! Replace YOURNATIP with the NAT address of your IPv6 endpoint.

expr add name=IPv6to4_prot_41 type=serv proto=41
firewall rule add chain=forward_host_service index=10 name=map_41 serv=IPv6to4_prot_41 log=enabled state=enabled action=accept
nat tmpladd group=wan type=nat outside_addr=0.0.0.1 inside_addr=YOURNATIP protocol=6to4 weight=50
saveall

Then set up the Hurricane Electric tunnel on the endpoint, using the NAT address:

modprobe ipv6
ip tunnel add he-ipv6 mode sit remote 216.66.80.26 local 192.168.1.77 ttl 255
ip link set he-ipv6 up
ip addr add 2001:470:1f08:1962::2/64 dev he-ipv6
ip route add ::/0 dev he-ipv6

Now you should be able to ping6 inbound, and outbound to your IPv6 address, and reach ipv6.google.com for example. Going to an IPv6 test site such as http://test-ipv6.com/ finds the IPv4 and IPv6 addresses. It does point out that the name server I am using does not have any IPv6 support though, which could cause issues with IPv6 only sites with IPv6 only name servers. Not going to fix that for the moment though. So we can browse the web on IPv6, at least, although it is quite hard to tell if you actually are. You can check your web server logs, IPv4 addresses are listed in IPv6 notation as ::ffff:188.220.243.64, while the real IPv6 we are browsing from are given in full, so we can see that Firefox for example will use IPv6 addresses by default when available.

One more thing is the rest of the machines on the network at home (or in the office!). We have our /64 allocation, so lets use it. All you need to do is run a server that will reply to the stateless autoconfiguration requests, which is radvd on Linux:

apt-get install radvd
sysctl -w net.ipv6.conf.all.forwarding=1

Set the sysctl setting in /etc/sysctl.conf too so it is persistent. This stops you receiving IPv6 router advertisements and lets you forward IPv6. Then create an /etc/radvd.conf for the interface you want to broadcast on, yes mine is br0 as I have an bridge setup for my VMs.

 interface br0
{
  AdvSendAdvert on;
  MaxRtrAdvInterval 4;
  prefix 2001:470:1f09:1962::/64
  {
    AdvRouterAddr on;
  };
};   

Now you can start radvd and set up the service to be auto restarted. And then by magic any computers in the vicinity that do not have IPv6 disabled will automagically get an IPv6 address, based on their MAC address and your prefix. Magic! My Linux netbook got one, but I could not ping anything. First issue was that my bridge (or in your case your network interface) did not have an IPv6 address, so I added one with ip addr add 2001:470:1f09:1962::2/64 dev br0. Now I could ping6 that, but no external routes worked. It turns out that IPv6 forwarding is all very well, but you need explicit routing tables too. So add ip -6 route add 2000::/3 via 2001:470:1f08:1962::1, where 2000::/3 is all global routes, and the 2001:470:1f08:1962::1 is the other end of your IP6 tunnel. Sorted! Well kind of, both the machines, Linux and Mac, that I am testing with are constantly losing routing and ability to ping6 the gateway. Odd, the gateway machine is fine. OK fixed, see this forum page, actually the link address is not the allocated routing address. 2001:470:1f08:1962::/64 is the network of the point to point link, but my routed network is the one after, that is 2001:470:1f09:1962::/64. This is listed when you login to Hurricane Electric again. I have changed the configs above now. Everything just working as expected on all the computers.

Note that you should be able to do all this with a Mac OSX machine or Windows machine as the router or server, the setup will be a little different, but Hurricane Electric should give you the help, along with your OS vendor. As for your routers, you may have to read some Dutch too!

So there you are, a quick guide to getting started with IPv6. Happy protocol upgrades!

Note: I have now moved my blog to Hetzner which has native IPv6 availability, which makes life much simpler. If you are selecting a new provider, always choose one that supports IPv6! Rackspace/Slicehost my former provider seems rather behind the times.

I am also going to predict that I will write more blog posts next year; this year has been pretty sparse, and I should do better. I would have written more in the last month but I have been writing code instead, for reasons that will be the subject of a future blog post. I think it is pretty unlikely I am going to get another total NDA job again, far more likely back to open source and open communities. Its more fun when you can talk about what you are doing.

The other thing I an predict is that I will be doing some different and interesting things next year. I am no longer working on the big now-cancelled secret project, so I am looking for interesting opportunities, so get in touch and lets work together on something!

Bubbling Up

1. Mobile

   A bit obvious this one but mobile internet access which is about 3.5% of the total at the moment should reach over 10% by the end of the year and continue to rise. Arguably if you include netbooks as mobile this figure was probably reached this year and we may hit 20% by the end of 2011. What effect is this going to have? First development is going to be mobile first, web after, in terms of compatibility, testing and progressive enhancement. Second the obsession with apps will continue for a while, but the mobile web will continue to forge ahead in its stealth mode and will prove more successful long term. The big VC obsession with local (aka Groupon) will continue, although oddly people are using the web to globalize, and in countries like the UK much of what is local is disappearing businesses unless they have a much wider reach. Mobile is about people not places largely.

2. IPv6

   IPv4 addresses will run out shortly and no one is very prepared. I had IPv6 at home a few years back but it is still pretty hard to get. Low end routers do not support it, and there are issues with many operating systems of one sort or another. Mobile phones are generally not ready. Amazon web services do not support IPv6 either. It is a mess. And by the end of the year it will be causing real growing pain for the internet, so expect to have to pay for the transition for no visible short term benefit. The long term benefit is the death of NAT and the rebirth of P2P in a way that will make the first Napster P2P boom look tiny.

3. Touch

   The touchscreen is the interface of the future, but for next year Apple still has the upper hand to themselves. Some iPad competitors will come out but they will not get much traction as the software is so behind in terms of the touch interface for anything larger than a phone, and even there they are behind. Don’t expect real competition until 2012.

4. Javascript

   Javascript is currently the programming language to watch. After a few missteps, node.js has finally made server side Javascript programming popular, while the new browser wars have made it faster than most dynamic languages in most benchmarks (yes, Chrome’s V8 Javascript is 25 times faster than PHP, and catching up with Java), only LuaJIT is faster (and has coroutines, sorely lacking from Javascript). Node.js (subject of a blog post soon) also adds an interesting server side asynchronous programming interface that makes writing high performance non threaded code pretty easy. This year we will see server side Javascript really take off, as the code reuse, and programmer reuse arguments become compelling, and also as APIs start to get real and offer server side scripting. Also people are going to notice all the software engineering efforts going into making Javascript usable in the large, with testing and so on and start using them.

5. SSD

   In the last few years solid state drives appeared, and reduced transaction time to non valitile storae by about a factor of 1000 compared to hard drives, and then started to get cheap. It is not yet quite clear what the best way to abstract this transactional (but relatively small) store to programs is, or whether it will end up being bundled with hard drives or other such uses, or how much the costs will fall so that they replace hard drives for many purposes. Database performance should be dramatically improved on many workloads. I think there will be interesting new work in file systems to make use of this; also expect SSD to become available in cloud environments in 2011.

Sliding Down

1. Centralization

   The internet is going through one of its centralization phases. Something like 25% of US page views are on Facebook now, as it has become the new AOL. Similar but more niche points of concentration exist around services such as Youtube. Facebook pages are replacing web pages for many, or substituting for them, and people are giving up control for this. Obviously it is self fulfilling as that is where the audience is now. However this sort of behaviour is very cyclical, and now the geeks are reacting against this, and a decantralizing infrastructure will start to be built again. Hey and once we get IPv6 we can do P2P without everything being dependent on NAT traversal hacks. Don’t expect too much in 2011, but the infrastructure will be being built, we have seen the risks of Delicious being closed.

2. PHP

   So PHP6, the attempt to make PHP understand Unicode was scrapped in 2010, and server side Javascript is eating the easiness factor. It is not very popular on Github where much of the open source coding is going on now. While there are some existing code bases that will continue to be developed in PHP, it is pretty hard to imagine anyone starting a serious development project in it now.

3. Hybrid tech-media companies

   Myspace is in continued decline, losing about half its traffic every year, and Yahoo! has not found out how to be either a successful tech or media company. Microsoft has lost $2bn in a year on its Online division that includes MSN. The tech-media hybrid just does not seem to do one or the other well. So big media has decided to pay the tax and let Apple be their tech department it seems, going iPad mad.

4. Prices

   Downward price pressure is going to keep up in all areas of technology, from mobile phones to cloud services to storage. Service prices can only fall by commoditizing whole areas, particularly operations, and by building for integration, through better APIs. Any code without an API is legacy now. The recession is going to continue to emphasise cheap, and we will not be going back to “enterprise” pricing. Commoditize now!

5. IE6

   Yay, it doesn’t matter any more, and it is hindering mobile, which matters much more. Both Google and Facebook started dropping support in 2010. The excellent YUI graded browser support predicts it will drop A grade support on the next update in Q1 2011, which will mean it will recommend not serving any CSS or Javascript to IE6. One to celebrate!

Have a happy new year!

Out of the box

The out of the box experience is pretty terrible you sit in the airline lounge and open it up and turn it on and then it just insistently asks for the mothers nipple iTunes parent for imprinting. No playing until that’s done.

It gets better after that though. As you expect it is nice to look at and to use like most Apple products. Pictures in particular look lovely and it is fast smooth and responsive. Web pages vary in how quick they are, but somehow slow pages seem more annoying.

I haven’t installed many apps for various reasons: the UK app store does not have all the apps in, for example none of the Apple ones. The pad won’t even download apps over the air saying “the app store is not supported in your country” but it will sync them. However my work Mac that it was mothered to died leaving it orphaned at a young age.

Comparison to a netbook

I mostly write my blog posts on my Eee PC running Ubuntu netbook remix. The iPad seems noticeably faster. Part of this is clearly the screen rendering which on the Eee is a terrible integrated Intel graphics chipset. Mind you it was much cheaper too. The bigger screen size and higher resolution of the iPad make it nicer to use too. But there is a limit to what you can do as there are very few applications. Personally I like to have a command line and a programming language locally rather than on the web, although I guess one could rig something up in browser with local storage; there seem to be a few editors around now. Will think about that model although it needs a lot of infrastructure to work.

Web

The web works well. In portrait view which I have mostly been using you can see a long way down a page and generally read the text too. That’s quite a nice page view. The big issue though is browser detection which is a terrible thing. People are detecting browsers not capabilities. For example the BBC iplayer thinks the browser wants flash even though they have a QuickTime iPhone version (the videos may be iPhone resolution only I suppose but that would be better than trying to show flash. Other sites show the mobile version which should mainly be about screen resolution detection not the browser identifier. Some sites just don’t work because they expect hover states. Other than that you just want to manipulate things rather than press buttons, as a mouse interface just feels unnatural. It is going to take a while before many web sites have gestural interfaces.

Writing

Google docs turned out to be a bit confused and I was unable to create a new doc on the website for this review I think it was only showing the mobile version even when I selected desktop. I couldn’t bear to use Notes with it’s hideous use of the Marker Felt font. Fortunately I had installed the Evernote app earlier so that’s what I am using. Typing is not great, pressing on glass and you have to hold it up with the other hand as typing on the lap doesn’t feel right. Doable but not ideal. Also I can’t work out how to turn the clickiness off other than just turning the volume down.

Gestures

After a short time gestural interfaces become very natural. There are a few issues with standards and ways of doing things that are not yet well defined but the basic movements are simple and other operations are easily learned. The big screen makes things much easier than the iPhone and multi finger operations make sense like bunching and spreading out photo albums in the picture viewer. Our hands are good at learning these sorts of operations and being precise about them. Touch is far more natural than say speaking to a computer. It is interesting that Apple and others have gradually been introducing elements of touch such as two finger scrolling on touch pads that I find it hard to manage without. Who wants to move a mouse to a picture of an elevator when you can just stroke the screen?

Walking around

The iPad feels the write sort of thing to use in meetings for looking at reference material (the Basecamp overview page works well for example), looking at diagrams, the web, taking notes and so on. It is almost as easy to walk around with as a pad of paper and generally as useful although I don’t find diagram drawing very intuitive yet. A camera would be useful for capturing whiteboard pictures and so on; having a device without a way to get rough pictures easily is a bit annoying. Oddly though I don’t feel the same way about the netbook which has a front facing camera for video calls that I don’t use. If the phone could talk to the iPad easily that would help but it doesn’t. It is one of the Apple annoyances that they want to sell iPad 3G contracts rather than make the iPhone and iPad work together as a unit.

Future of portable devices?

I think the gestural touch interface is going to win over the mouse mediated interface. The keyboard will last, but maybe as an accessory like with the iPad rather than joined. However vertical screens don’t work with touch as your arms get tired. Pad is perhaps the right model reflecting how we use paper most of the time. There are issues about how to hold and use it that will need to be ironed out, and there are issues with Apple’s idea that it should be a simplified computer as they have perhaps gone too far. Indeed I would be very happy if it had a gestural version of Ubuntu on it like my netbook I think that might be perfect.

I have been looking at the issue of writing filters for formats that basically change them as little as possible. This is a slightly difficult field in many ways. You have to store a fair amount of extra information in order to do this, but if you are say changing some metadata items for a user they may not want their CDATA removed and replaced with a semantically identical but syntactically different form. So I am looking at the formats partly from this point of view. Sometimes this brings up the conflicts between the human readable and computer manipulation aspects of these formats, the logical and physical structures. I have also been looking at making simple tools to allow modification of document formats, which has also raised some issues.

Pro JSON

JSON is simple and now well defined. It used not be clear in a few places, like how to encode items outside the 2 byte UTF-16 encoding using the \u notation. Many people do manage to generate invalid JSON (no quotes around identifiers, use of single quotes, use of a BOM at the start) it seems, which is a problem. I mean if people cannot even get possibly the simplest format ever invented right, what hope is there for civilization? This should get better as standard libraries that do the right thing come out one would hope! Introducing lax JSON parsers (in the style of HTML 5) seem to be unnecessary for a simple format that is normally generated by a computer. A strict JSON parser is not a lot of code.

JSON has a simple way of showing which of the allowable encodings it is in, based on the zero bytes at the start, as the first two characters must be ASCII. Allowing a BOM and then allowing unicode whitespace might be more standard, but the whitespace has no function except for use in text editors.

Despite teh attempts by, for example, E4X to add a simple XML native format and processing model to Javascript, JSON remains much easier in most languages to process, as it is built around structures that most languages just have natively, while XML is not. Some languages have issues with a mismatch to JSON, but most are fine. E4X on the other hand has security issues client side, and is not seeing adoption except in some server side applications.

Against JSON

JSON does not have a native hyperlink type. This is unacceptable in a web format in my opinion. For example a REST interface (a real one, not the clean URLs many people who use JSON think are REST, one with HATEOS) requires native links and link types. A native link format such as {“next”: http://example.com/next} would solve a lot of issues, and would be compatible. There is JSON-Schema trying to add schemas that can extend the type system, but having to have a schema to understand links just seems overkill to me.

JSON ought to specify that unicode strings are normalized really. I guess most of them are, but it does mean you should normalize before doing comparisons on keys for example.

There are some syntactically different representations: arbitrary white space, although this does not including the full unicode white space definition, and backslash escaped characters, which can mostly be represented directly in the unicode encoding of the document. Whitespace clearly needs to be preserved for readability and use of line oriented editors and tools. It is unclear how inconvenient it would be if \u codes were normalized to unicode which is the sane default. I think there were some tools that did not support unicode, although it is mandated by the standard; it is odd perhaps that an ASCII encoding is not an option, but it seems unlikely to be important. In fact though, preserving the exact used syntax is not difficult in many applications as, unlike with some cases in XML, this does not involve much additional state.

Although JSON was designed to serialize data structures, its small set of types is limiting. We are not going to get around this though for now, as computer languages still have such different ideas of types. Most uses of JSON have an implicit schema, which is both a strength and weakness. Most implementations were tightly coupled, for example in AJAX. Now we are seeing more APIs exposed to the world using JSON; these have more need for a schema. I tend to prefer the idea of HATEOS, the REST idea of hypertext as the interface design constraint, rather than published schemas in the SOAP WSDL style, and JSON seems to be more inclined to move to the latter. Especially if people use JSON-RPC, I thought people had given up on the RPC style on the web but it appears not.

Data model

The JSON data model is simpler than XML. This is less clear as a differentiator. XML nodes have attributes and children, JSON ones attributes or children if you consider the object to model an attributes set and the array or list type to model an ordered list of children. This difference is not hard to work around, and it is very domain specific what the requirements are.

Bill de hOra pointed out “you should add field cardinality to the distinctions – json needs to change structure [], xml needs just another element” which is a very good point.

Schemas

Schemas for validating are great. Validation is an important activity. It is complicated though in general, rules such as this must be filled in if that is not and so on. Essentially a validation schema might need to be very complicated, but many are very simple. Having a choice of languages to express these constraints in seems to me to be a good thing. The XML DTD is too weak, and should not have been included in the language, as discussed below. Some constraints are computationally complex and need very expressive languages.

The second function of a schema is interpretation; this may relate to validation in that a field must be readable as a number say, and we are also going to read it as a number. This is a different requirement, as in many cases it is about object modelling and code generation, when a validated structure is then mapped to a native language object. These are conceptually separate processes, as a number may be constrained to be between 3 and 5 for domain reasons, but the representation in say Java may be an integer, but it need not be. Of course the validation stage here is essential for security reasons, to stop overflows and type errors; however these are conceptually different activities and may have different schemas.

Against both

Binary data is a big problem. We will need a lot of other formats for anything that has binary data, they are just so much more efficient, even after compression. So ideas of a universal format are not going to happen.

Against XML

XML is weak on unordered items. Most of the structure in an XML document is the child relations and these are ordered. This is used as a criticism, but I am not sure it is that reasonable, as attributes are unordered and as said elsewhere there is an equivalence with the two structures provided by JSON, named and unordered items, and unnamed ordered ones which seems natural.

Pro XML

It was pointed out by Erik Wilde that I had missed out the pro XML section. This was an accident. I am actually very pro XML in many ways. First it has enough structure that we can build rich data structures; and to add to that it has some standard forms (such as XHTML) with rich sets of attributes and elements which can be reused in a variety of domains, and standard link relations. The other big thing is the set of extraction and transformation tools, which are generally quite well designed and fairly complete. There are stream and DOM parsers widely available.

Against XML DTDs

The DTD, which XML inherited from SGML, is an anomaly in many ways. First it has a non XML syntax, so we need another set of parsers and tools to work with it. It has several functions that really need to be separated. The first function is as a schema for validating documents against. Unfortunately it is not a very good schema language, as the constraints it can apply against documents are limited. Now we have for example XML Schema and RELAX-NG, which are better schema languages, but the DTD has a special position in the specification that is difficult to drop.

In addition to being a schema, the DTD can also define default values for attributes that the application should see just as if they were in the document. This is the kind of thing that makes preserving the textual form difficult, as there is a syntactic but not semantic difference between certain attributes. I also do not think that this is used much, as real defaults would be implied by the processing model not the document. Clearly it is easy to remove this feature from documents simply by adding in all the implied defaults explicitly.

There are security issues due to the parsing issues with entities, which means that some parsers disable DTD parsing for security reasons. SOAP for example does not support DTDs. This is of course non conforming, but clearly a good idea in many situations.

DTDs are not namespace aware, which makes them unusable in many cases with documents with namespaces. Another reason to deprecate them.

Against XML entities

Then there are entities. My reading of the initial spec is that entities were designed to save typing for people, but I do not think that they are used for anything except for memorable encodings of characters outside the ASCII set. The thing about this use case is it is perfectly alright to substitute the values for them, as they never change, whereas if I create my own arbitrary entity inn a DTD for the name of something it may be because I wish to use this like a search and replace function to substitute whatever I want in. This is in my opinion not really appropriate at the document format level, this is an application level tool, and the application should use regular XML tags for this type of user level structure.

XML entities can also be used as an inclusion mechanism; again the DTD is not the place to define this. XInclude seems much better if this facility is needed.

Entities can contain other entities, markup and so on. Recursion, and unbalanced markup are not allowed. This whole thing adds enormously to parsing complexity, when the use case is entirely as character data.

Against XML namespaces

I am not against XML namespaces per se, but there are pathological cases which make them very hard to process sanely. In particular, you can redefine the same namespace name to refer to multiple URIs in the same document, and you can refer to the same URI with different names. This effectively means that all processing needs to refer to both the short name and the full name. As this is exactly what the spec was trying to avoid it is pretty bad. The amount of state you need to keep to keep a namespaced document textually the same after processing is very large; the nasty mess one tends to get from parsers to let you cope with namespaces is one measure; another is the complexities of xpath on namespaced documents, especially ones with any of the pathological cases in.

The simple solutions seem to involve not allowing redefinition of namespaces to a different URI in the same document, or the converse; declaring all the namespaces that will be used in the root element is also an option. This means processing can be more or less namespace unaware, as xsd:type will mean the same thing regardless of the context. This falls in with the standard usage, where a fairly small set of namespaces are used and they have abbreviations by convention that remain constant across large sets of documents. This means that very little namespace awareness complexity is needed.

Other issues

Mixed content, the role of CDATA, the significance of whitespace, these are all extremely complex issues that could be simplified.

Minimal XML proposals

XML, quite hard but worth it? For the applications I am interested in, I think simplification is needed. The first issue is that security and simplicity are related. Anything web facing will get hostile documents thrown at it, and having more constraint helps, in a way that the document processing industry does not see so much as an issue.

There was a time ten years or so ago, when minimal XML proposals were fashionable. XML itself was of course an attempt at a minimal SGML proposal, but not enough was cut or changed, and much compatibility was kept. Common XML seems the most reasonable to me, and addresses many of the issues. XML tools do not work in the way that was perhaps envisaged, and making things simpler and easier, evolving them, will make them more robust. JSON shows that the demands for simplicity are there, and XML will suffer if it does not answer these.

The first thing is to drop the DTD. It serves no real function now we have alternative schema languages for XML. Radically, I think we can drop entities too, other than the necessary ones for escaping (amp, quot etc), and numeric ones which are again syntactic. The only possibility for requiring named entities is XHTML, but it barely exists now, and those entities could be special cased there without difficulty, as their values will never change and they do not contain markup or other things that cause parsing issues. Arguably these named entities could be added to the XML spec anyway for all documents, changed to a purely syntactic thing. I am not aware of any other XML usage of entities; there may be a few I suppose.

For namespaces, there needs to be a solution that maps syntax to semantics, so that an attribute or element syntactic name has the same semantics throughout the document. Renaming in different scopes makes global transformations, comparisons, and simple processing too hard. It breaks simple search and replace, even that needs to be namespace aware.

Data versus applications

Part of the conflict is due to whether XML is an application protocol, or a data format. Some of the bits that have issues, like entities, are really part of an application data format, for a class of applications that work according to the model in the mind of the XML designers, which in turn was based on real SGML applications. But data formats are winning really. We want to attach additional semantics to data now through standard mechanisms, such as relations, RDF and so on, not be expanding the storage format. Simplicity is winning here: complexity in a data format does not add to the richness that can be expressed; simple uniform mechanisms can do this. And simplicity is going to win; linked data over Microsoft Word style application data formats.

What will happen?

I actually think these changes are, informally, happening. DTDs and entities are not used in many cases now. They may be in some publishing applications, especially those based on SGML, but the web document architecture does not use them significantly. Namespaces are used in a particular way, usually. HTML5 has shown what the logic of human readability and writeability implies, which is a non XML language. The great advantage of XML is the variety of ways in which it can be processed, but issues such as security to hostile documents, parsing complexity, performance, and ease of processing really matter a lot, and despite many weaknesses JSON is showing the way of radical simplicity. But a simplified XML would be no more complex than JSON I think, and have the advantages of richer tool support, and widespread use. Most of the XML in the wild an APIs is very simple; the sorts of XML that are embedded in other documents as metadata are simple too. Security is limiting processing, and the traditional publishing applications that historically used more of the functionality could change too, although more slowly. Will simplicity win, and wil JSON replace XML? I think not, because so much XML is in use, but I think a specification of an XML subset is needed to stabilise the situation.

10. The Pirate Bay saga

In yet another mess in the ongoing spectacle of the entertainment industry preferring legal to creative solutions was the Pirate Bay trial. All this really showed us was that the laws are just not well framed, so anyone could win, and it may all change on appeal. This time it was not suing your customers directly, but legal action is not going to make anyone change their mind. Obviously the next step is going to be to influence the passing of bad laws, not the creation of business value. It seems uncoincidental that Spotify is Swedish. In times of change, business model engineering and service engineering are as important as product engineering. Legal action in the way the entertainment business is conducting it creates nothing long term.

9. Australian internet censorship

Get your act together Australians and stop this. Many other governments are looking at ways to start doing this, so it is an important example.

8. The EU MySQL Oracle Sun delay

You cannot make industrial policy on this sort of timeline. If the EU were to turn down the deal now Sun would be destroyed. Oddly MySQL was at a transition point anyway. I am very much in favour of the Drizzle idea of the future of MySQL; who knows where it will end up but it may well be outside Oracle anyway.

7. SPARQL is a query language without a resource model

Looks like this has a chance of being fixed in 2010 at last, although I have temporarily mislaid the references, check for the newer references to named graphs. The idea that you could launch a query language for the web without a resource model was yet another of the dumb W3C ideas. The model appeared to be to build XML in Prolog. That sucks. Unfortunately the fixes are quite substantial (quads not triples for example).

6. WebDAV

Although not exactly something from this year, remarkably it has kind of held on and since people still specifically mention it as an alternative to CMIS. Indeed it is kind of useful sometimes, in strange situations, and it does work in a limited way, but it is not a modern HTTP interface. You have to remember how early it is, as work started in 1996, when it was not clear how the web would develop, or indeed how HTTP would develop (HTTP 1.1 was out but not much used and it was shipped in a mostly 1.0 environment). Even at the time some of the mistakes were clear, but the great thing is they are all documented in Yaron Goland’s The WebDAV Book of Why. Like the issues of hierarchy that make mixing WebDAV with normal HTTP impossible, and the depth header disaster. There are also some comments about it in the Roy Fielding podcast in which Roy tries to avoid talking about JSR. The best thing about WebDAV is how well documented the mistakes are; this should be compulsory for all standards.

5. Geocities

The embarrassing kiddie years of the internet dead and buried. Mostly not the worst of 2009, but the idea you could still nurture your city. Obviously an anti archival moment for future historians to curse about. Still, a hubris reminder too, this was once the third most popular site on the intranet, and sold for $3.57 billion; look on their works ye mighty and really despair.

4. Microformats

Never going to work. We really need generic metadata representations that have sane serializations or embeddings into all formats. Metadata now lives within documents; it used to get lost before that. So the RDF model has won, and microformats have lost. Oh, and the standards process sucked.

3. The XHTML2 débâcle

Had to happen, but why did it take so long for the W3C to fall behind HTML5 rather than XHTML2? This was a huge diversion of resource. The W3C churns out stuff and some of it gets adopted, some is implementable, some of it is not implementable realistically. The organization needs to change or it will be irrelevant.

2. The Go programming language

I am an aficianado of programming languages. I have programmed in many of them, C, Haskell, you name it. Lua and Erlang my new ones for the year though its getting a bit late and I have barely started. I know my combinators from my closures. What is the point of Go? It does not really offer anything for the currently interesting problems, I do not think it is going to make it anywhere. I would be surprised if it ever gets onto the allowed Google programming language list, which is C++, Python, Java, Javascript since you ask. Google is doing some cool performance work on python though under the name unladen swallow.http://code.google.com/p/unladen-swallow/wiki/ProjectPlan).

1. I4I’s patent win over Microsoft

A last minute entry here. I4I has an injunction against Microsoft selling Word without the generic XML editing functionality removed. Obviously it will be removed, and it is not a feature that a lot of people used. However analysis of the patent indicates that it clearly has prior art, is unclearly applicable, and could affect many other XML applications. The affected part of Word is designed to be a fairly general XML processor, with similar capabilities to XForms. We need to support Microsoft in getting the judgement reversed.

I happen to like pictures of people unposed, and I sometimes take them when I am in the right mood. The decisive moment of course has an important role to play in the history of photography; when I was on holiday I happened to take a picture of a man sitting on the street outside a Starbucks, and when I posted it to Flickr I thought I would find a Starbucks group to post it to. And thats where I found this hilarious thread, which is a warning to people about what happens if you jump into social media at the deep end.

At the end of September, when the official Starbucks group was started, in the social media frenzy of 2009, this thread was started, pointing out that many people had been asked not to photograph in Starbucks stores, or had been thrown out for taking photographs.

The official responses from the official moderator analisamarie started off fairly optimistically

Our formal policy is that all press-related photo inquiries need to contact press@starbucks.com prior to taking pictures in a Starbucks store. However, we have no formal policy around customers taking non-press related pictures in-store so if you hear otherwise, it might just be because your barista is camera-shy

Hmmm- good discussion! Sounds like there is a bit of confusion out there – let me take this back to my team and see what we can do to help. Thanks for bringing this up…more to come!

Then got bogged down in legal

I am making great headway here and hope to have some detailed information for you all shortly. To give you an idea of what I’m up to, I am researching if some of our international markets have policies around photography in stores. Since international laws and regulations vary country by country, this is quite the task I’m also working to see where the confusion is stemming from in some US stores. Again, stay tuned. I’m working on it!

I have been meeting with various teams in the building and learning a lot about the world of policies I hope to have something more concrete to share with you soon – thanks for your patience while I work through the details.

I am getting closer to a final ruling each day. I have a big meeting on Wednesday and after that, I will post here with an update.

I did have a very productive meeting on Wednesday of last week. We read through each of your comments and now the legal team is reviewing some of your feedback around public and private property. More meetings this week…more to come!

The social networker starts networking internally

Still here and haven’t forgotten about you. I’m writing a blog this weekend/next week about this discussion and hope to post by the end of the week. I’ll keep you in the know. Have a good weekend!

Just wrote a blog response that my legal team is currently reviewing…once I have final approval I’ll post it and let you know. I know it’s taken a while and I know I’ve said it before but I appreciate your patience. This has been quite an interesting project to work on and has involved many meetings with all sorts of teams throughout the building. SO glad you guys brought this to our attention so that we could sort it out for you!

Hints on something more negative

We want to do this in the best way possible. There are many perspectives to take into consideration as part of this discussion. That means considering our baristas’ daily work and their privacy, our customers’ experience in our stores as well as your photographic expression of that experience. We have a lot of things to consider when making decisions that affect what happens in our stores. It has to be the right thing for our partners (employees) and customers, and it has to work well for stores around the world. Please continue to be patient while we work on a solution. In the meantime, I do ask that you continue to be respectful of customers and partners in our stores. If a barista asks you not to take pictures, please respect their request. More to come – Anali

And more strongly hinting that the answer is going to be no. As several of the people in the thread point out they might as well close the sponsorship agreement with Flickr if they are going to say no to photography.

I have to add that this group isn’t explicitly here for the purpose of taking pictures inside Starbucks stores. That is one part of the Starbucks Experience but pictures of your experience out-of-store are welcome in this group as well.

This is currently her last post, a few days ago, so the story may still unfold. Possibly not as dramatically as when Amtrak police arrested someone for taking pictures for their photo competition but it has slightly broader issues than that.

First there is the timetable thing. Do not bother with social media if you can’t make decisions quickly as an organization. Period. When issues are raised by social media, you have to respond fast, because things have a habit of going viral. Two months is a joke. Fix the response times before you do anything, or when something blows up you will bodge the response.

Second, be a bit lateral. I mean, surely someone would have thought about this issue, maye the “camera-shy baristas” if internally the social media plans were discussed, but obviously this social media plan comes from head office.

Third, social media is not about head office marketing, it is about running an open, transparent business. Flickr is not Facebook, and does not have quite the same vibe (and far fewer photos actually). It is mainly a subscription platform, and many of the people there are generally quite articulate. The issue is not that they are going to complain about your coffee, which has carefully planned responses, they are complaining about the way you treat them as a commmunity. Walk right into it, unprepared.

Engagement has to be seen as a two way street; if you are not prepared to change in the social engagement and you treat it like an advertising campaign you may come unstuck.

The stupid thing is of course that Starbucks is a social space. The coffee shop, its Central Perk in Friends, its the pub in the British community, its the office when travelling. Casual photography is entwined with the social space since the Kodak Brownie; it has been reckoned that most of the data the human race has ever produced is in the form of photos (lacking a citation for that right now; would welcome one). Every mobile phone has a camera now. It is not actually hard to work out what the answer to the question should be; it even seems that that is already the policy, though no one can actually tell, as Starbucks policies appear to be secrets.

If your organization doesn’t grok social media, don’t copycat and try it anyway. maybe try a dose of Enterprise 2.0 first, and I don’t mean writing blogs for lawyers to read. This online stuff, it is going to change the way things work, until you understand that you will get it wrong.

Will be interesting to watch and see if Starbucks manage to sort this out.

Update We finally have a complete copout policy “Here’s the answer that you’ve been waiting for …Photos are allowed in our stores for the purpose of sharing them in our Flickr group.”

(Note picture taken outside Starbucks in a public space, without purchase of Starbucks beverage; however I cannot post it to the Starbucks pool as I don’t have a model release).

I like books. I read a lot of them. The office is full of books, almost all of them mine. Apparently technical people don’t read books much. I seem to remember a figure (from Joel Spolsky? I am on a plane so I can’t check right now) saying that the average programmer had never read any books on programming, and never would. What’s that about? If you don’t think about what you do in that sort of introspective way that makes you want to read books about other people thinking about it that way, then you are not really very engaged in what you are doing.

This blog was meant to have more book reviews than it does; in fact this is the first one. Expect more. Its not that I am not reading the books, but there are other things I need too write about. Expect some more. Why this book? Well I had a plane trip, and it was on the shelf at Foyles and it sounded interesting. I follow @halvorson on twitter, and she sounds interesting, though I have not heard her speak. Its a short book, so I wont make the review too long, after all you can just read the book next time you are on a plane.

I am not a content strategist, but I do like writing. And reading, as I mentioned. Now it is therefore possible that there is a big self selection issue here. It could well be that everyone who reads this book will agree with its central premise about the importance of being strategic and serious about writing on the web. It could be that actually only people who read books ever read anything on the web either. Maybe everyone else just looks at the pictures and laughs at the funny meerkat. Igor the meerkat is the most successful advertising campaign this year, and has turned a failing web property into a great success with no words at all, and not much money either.

Fortunately I don’t have any figures on the importance of written content, and no internet access on the plane to find any. So I am going to say, I read content on the internet. Almost all of what I read is good content, I don’t read the other. If it is boring I skip it. If it is too short I skip it. I don’t watch videos, due to lack of time and headphones. Reading is faster. If you want me in your target audience, you need to write, and write well. Or catch me in the pub over a pint of bitter. (Actually I like good visuals too, underused on the web).

So what are the key things I took away from the book? Think like a publisher. Remember there used to be a whole business managing content; it was called publishing. They planned it and commissioned it and had whole branded collections of it called “magazines”, “books” and “newspapers”. People paid for these they were so good. There is a whole industry to steal ideas and methods from (and people for that matter).

If you don’t have a content strategy, you are just hoping it will all work out. You might get lucky, especially if you have a good writer, who will unconsciously perhaps create you a strategy, or at least stuff people want to read. But help yourself, think strategic in content like you do in other areas of the business. Plan, execute, measure, regroup.

“Page tables” a content wireframe. Don’t like the term, but they are needed for a web project and I cant think of a better name.

Content is to support the aims of the organization. It is not just marketing any more, it is branding, it is product design, it is sales, and it is the conversations you are having through social media that position your company in the marketplace. “Recognize content as a valuable business asset” is it in your investment programme? Is it on your balance sheet?

Web content is long lasting; unlike print which cannot be changed and often has a short shelf life. Kristina has a chilling example of a corporate Youtube channel that no one has logged into for a year. Many corporate blogs just die from lack of care and feeding. A blog is for life, not just for Christmas.

“Push ‘user experience design’ off the pedestal”. UX without content is like a fish without a bicycle, looks pretty and maybe its tasty, but it is not getting you out to the next village. Or something. Actually some agencies do cover content quite well, but many do not, sticking with the pretty pictures and the ooh shiny widgets.

So there you are. Content, words, unsexy, black like coal, and almost as much effort to get out of the ground. An anthropomorphic pun might work for some people, but not you and me, those of us who reached the end of these words.

My argument was mainly that storing flat, unstructured data was not enough for most content projects, and the difficult questions of structure needed to be addressed. Daniel’s third part addresses how they do this. Actually when I first looked through the NPR content after reading the first two articles I could not find any content that had inline HTML, clearly was an accident should have read more, as it is used.

The actual NPR process is interesting. In particular it shows the amount of care and attention in curating content that is needed to keep it reusable, repurposable and valuable. Semantic content requires you understand the range of meanings that it encodes and how to work with them, and transform them, and quality control them. And that means you need to know every tag and attribute that is going into the system and what that means for every output you are or may using.

One of the outputs that many people forget is plain text, and NPR is very clear on the writing style that is necessary for writing for HTML and plain text. Everything must make sense in the text form; links should be additional information that adds to the text not necessary to understand it. And of course no “click here”. Text output for other devices may vary between the expressiveness of HTML and that of plain text. Text that reads sanely also helps screen readers and other assistive technologies make the content understandable.

The key points here are that content markup must be

1. Valid. Processing is likely to be inaccurate without valid content, and tools will be more limited in how the process it, or will fail unexpectedly. Best fix this at the beginning of the pipeline.
2. Meaningful. You need the markup to mean what the author intended, so look at interface usability and training.
3. Accepted. You do not have to accept all valid XHTML, say. For a start, XML is an extensible language! You can choose a whole range of markup for a story, from the very minimal, to marking up each person and place involved, or more.
4. Stored. The marked up text must be stored; the NPR decomposition is plain text plus plus normalized markup which may work for some systems; storing marked up HTML without output transforms may work out better for others.
5. Processed. You need to handle each kind of markup for all output mechanisms, so they need to be introduced in a controlled way, although this should not be difficult. Changing markup is something that may need to happen.

I think the third part is the interesting one. Information architecture with websites often stops at the content level, missing out on this information microarchitecture of the textual content itself, leaving this to authors without enough guidance to build a consistent structure to maximise the long term content value.

Now Daniel has cllearly done an excellent job of analyzing and modelling the NPR content, a job which is vital to do well, and has chosen the simplest solution that works for it. The simplicity has a lot of benefits, because simplicity cascades in a lovely way. My argument today though is that for most people this type of model is too simple, and the general problem domain means that people need to embrace a more complicated solution for their content modelling, and that means learning to love hTML.

One way to start looking at the problem is to take your content and consider the case of inline hypertext links. No hypertext is the simple case, you can not solve the problem by forbidding inline linking in body text, which NPR chose, and which I will discuss below. With hypertext the challenges are to either

1. Rewrite links to what will be the output format’s reference
2. Rewrite links to a canonical output medium
3. Strip links

The choice may depend on the medium – your mobile site wants internal links to itself, but output to email might have links to the main site. This violates the clean layering diagram, as the output is not a pure content object. The first option is easier in some ways, as it is not much different from getting the non inline links that a model usually has to do, although NPR actually store all the link URLs in the content store. The difficulties come in this case when you want to output a subset of a site to mobile say, and have to deal with potential dangling links. The second option involves asking another output processor what its URL would be for a content item, which again violates the clean layering diagram. Some media require the third option, print for example. Making this work is a question of writing style, making sure that the hyperlinks just flow in with the text and are not required for sense – yes that means not allowing “to see this story click here”, but thats how writing for the web should be written, for reading with and without the hyperlinks.

This messiness is important though, and needs to be done well to not impinge on the API layer more than necessary, and to keep the caching working well. Adding this to the cleaner model allows the use of free hypertext, the foundation of the web.

Daniel goes on to say

But to truly separate content from display, the content repository needs to also avoid storing “dirty” content. Dirty content is content that contains any presentation layer information embedded in it, including HTML, XML, character encodings, microformats, and any other markup or rich formatting information.

Now I half agree with this. You cannot avoid character encodings, although mixing them is not usually advisable. But, like the issue of hyperlinks, he seems to be recommending content without any inline structure, just text. Anything else has to be normalized into a separate structure.

Now you can normalize and flatten everything into structures where everything is stored in low level structures, not say XML or HTML documents, which can be reformed into XML for output. But one of the points of the web project is that well written semantic (X)HTML or XML is a neutral semantic data format that it is transformable into other forms if necessary, and does make a good base format. Otherwise you are just normalizing into a database structure instead.

The NPR data model has the following structure

We then attach “resources” to the story, each of which is its own object in the database (examples of resources include full text with each paragraph stored as distinct records, audio, video, images, related links, and a range of other object types)

So no structure within a paragraph. No microformats, as we cannot mark up a name or place. No hyperlinks in running text, no equations or formulas. No rich text. Just boxouts, diagrams between paragraphs. It works, it can be very good, it is easier to implement, but its not as rich or excellent as truly structured, hierarchical, denormalized content can be.

Fundamentally the structural of the story in the NPR model is flat. There is hierarchy above the story, but it is flat within it. For some types of application, perhaps like the NPRs journalistic style this can work. I have worked with clients who do not want hypertext, just boxed lists of links, often as a stylistic issue. It has a certain formality, and the unbroken text has a traditional look, more Britannica than Wikipedia. The NPR is extreme as the model does not even as far as I have seen allow a word to be emphasised; emphasis is not however presentation layer information, it is semantic information that can be conveyed in pretty much every form of human communication.

Maybe it is because even for print I have worked often with richly substructured texts, or maybe just because I appreciate the browsability of well structured hypertext, flat text is not my preference.

Once you give in to stucture within the paragraph level, the normalization question becomes more interesting. You can still normalize items below a paragraph level but it becomes less satisfactory, as the objects start to lose context and stop being reusable on their own, divided into increasingly small units. There are still important normalization issues, as just because the document format is structured does not mean that everything should be embedded into it; items that are reusable within a document should be nested by reference so they can be reused elsewhere, so you end up with a tree of components for a document rather than the flat NPR structure of a list of components; just a few extra levels of structure, as normal articles are not very heavily tree structured, and just a bit more processing to turn these into items for the API and content output layers.

Daniel raises more issues

First, as an example, the image references within the block of text will contain HTML and possibly other markup, making the text block dirty. Any distribution to other platforms could then require special treatment to prepare the content for that destination. More importantly, however, is the fact that these same images are very difficult to repurpose because they are embedded in text. So, it would be quite a challenge to make a feed of images, to identify only those posts that contain images, to resize some or all images in the system, or to consistently restrict distribution of images that do not have the rights cleared.

Now some of these points are important. Journalists were traditionally encouraged to write as if there were no pictures, which always struck me as odd as a child, describing in 1000 words what is in a picture right next to the story, but content has always been repurposed, or reprinted, or read out. But the sizing issues are easily dealt with by linking to a class of images not a particular size, and letting the output process choose the size. And the system can understand the content types being used, such as HTML or XML and say strip out an image feed, or answer questions about the images. HTML (and SGML and XML) have been designed as one of the first presentation independent structured, semantic, content description languages to solve many problems that simpler, flatter representations did not solve, initially being developed for struuctured document applications.

Of course not any old HTML, say, is a good choice. It needs to be valid, and there must be agreed restrictions. These could, but need not, be a minimal, NPR style, only paragraphs, block quotes, lists, UTF8, no HTML entities, or they could allow more structure but still not allow presentational elements. This contract of what is acceptable because it is not presentation layer and it has transform rules to the output layers, must be strongly enforced. Authoring tools still need work; especially tools for structured, denormalized content need more support.

Overall then, web content management, and structured content mnagement needs to embrace the document types that the web has adopted. Native HTML and XML storage is not the wrong approach. Yes there are issues and complexities that have to be addressed, and addressed really well because if done wrong they can hugely hinder flexible reuse and repurposing, but if done right can enable a rich, expressive, hypertext, denormalized content world.