We Built this Firehose on Open Specs

photo CC joevans

Last week, we launched the Buzz Firehose feed.  You can see it in action at the sample Buzz Mood app (code).  One thing which we didn't highlight too much (though DeWitt did) is that this is completely built on non-proprietary protocols and formats: Atom, PubSubHubbub, and Activity Streams are able to deliver real time push updates easily and quickly.

We did do some special things on the supply side for efficiency.  If you GET the firehose feed you'll see that it never contains any updates, just feed metadata and the hub link.  Conceptually, it contains the last 0 seconds of updates.  This is primarily because it doesn't make sense to poll this feed, so it exists only to provide a subscription mechanism.  Internally, for efficiency we're pushing new updates to the hub with full data rather than pinging the hub and waiting for it to poll the feed.  But, we're using the standard Atom format to push the data. The hub is doing coalescing of updates so when a bunch come in over a small period of time, as with the firehose, it will send them in batch every few hundred msecs to each subscriber.  So the POSTs you get for the firehose will almost always contain a feed with multiple entries in it -- whatever has come in since the last time you've received an update.  There are also some tweaks to the hub retry configuration; if your server doesn't keep up with the firehose the hub will back off fairly quickly and won't attempt many retries.