But here is a kicker – it’s not the end. I saw this tweet, read this post and checked out a book by Charles Perrow titled “Normal Accidents” from the library. Published in 1984, the book is not about IT, but its material fits our field nicely. And boy, was I enlightened!

The book’s main point: no matter how much thought is put into the system design, or how many safeguards are implemented, a sufficiently complex system sooner or later will experience a significant breakdown that was impossible to foresee beforehand, principally due to unexpected interaction between components, tight coupling or bizarre coincidence. For us in IT, it translates to “no matter how much planning you do or how many safeguards you implement, failures will still happen.”

There are at least 3 common themes that are present in multiple illustrations in the book:

1. A big failure was usually a result of multiple smaller failures; these smaller failures were often not even related
2. Operators (people or systems) were frequently misled by inaccurate monitoring data
3. In a lot of cases, human operators were used to a given set of circumstances, and their thinking and analysis were misled by their habits and expectations (”when X happens, we always do Y and it comes back” – except for this one time, when it didn’t)

I have had my share of outages and downtimes, and I can attest that I have seen these 3 factors play a big role in tech ops. Some were bugs in management and monitoring code, some where human error, some where bizarre set of dependencies but all were a combination of multiple factors. For example, who would have thought that with a failure of primary DNS resolution server, the VIP would not fail over to the secondary; and even though hosts had more than one “nameserver” line in /etc/resolv.conf, application timed out waiting for DNS to respond before getting to ask the second nameserver; without name resolution, multiple load balancers independently thought that there was no capacity behind them (because management code calculated capacity in near real-time relying on worker hosts’ names) and disabled themselves, thus taking down the entire farm – now I know of course…

It turns out we can’t eliminate normal accidents altogether, but here are several techniques that I have been using to speed up detection and response in order to reduce the downtime.

Complexity budget. Described by Benjamin Black, this is a technique to allocate complexity among components beforehand and strictly follow the allocation during implementation phase. It helps avoid unnecessary fanciness and leads to simpler code, which tends to be easier to troubleshoot and recover after a failure.

Control knobs/switches for individual components. As John Allspaw shows on this slide, you need to be able to turn off any component in an emergency, or throttle it up or down. Planning this feature and building it in from the very beginning is very important.

Accuracy of monitoring data. Ensure your alarms are as accurate as possible. No matter how much chaos is going on inside the system during a severe failure, last thing you can afford is misleading the operators with wrong information. If you tried to ping a host A and didn’t get a response, your alarm should not say “host A is down” because it’s not the knowledge you obtained – it’s an assumption that you made. It should say “failed to ping host A from host B” – maybe it was network on host B that was an issue when a ping attempt was made, how do you know?

Availability of monitoring data. There is a reason first thing the military try to do when attacking, is disrupting enemy’s means of communication – it’s that important, which applies to our case as well. You either design your systems to be able to get monitoring data even during the worst outage imaginable (ideally from more than one source), or you at least should be getting an alarm about lack of such monitoring data (it’s a very weak substitute though).

All in all, to everybody in IT, I highly recommend the Normal Accidents book as well as this whitepaper (linked from John Allspaw’s blog).

Distributed applications are known for complexity of assuring all components are on the same page as to what’s going on around them. Hardware failures, network failures, operator errors can all cause chaos; distributed applications foresee these exception situations and attempt to know how to deal with them.

Up until now, the network piece of the puzzle has been usually under application owner’s control – it could be a LAN, or it could be a leased line to remote datacenter. Occasionally, a VPN would be used to provide a dedicated communication channel between locations over public Internet but its use was rarely focused on important stuff – a mission critical application would usually get a leased line.

With advance of public clouds such as Amazon EC2 and Google AppEngine however, these notions are changing. One day you may decide to leverage each cloud’s strengths and distinct features to build your app, or may want to avoid cloud lock-in or provide redundancy. In short, you may want to multi-source your infrastructure.

Your multi-sourced infrastructure will of course be a distributed application. But there is a significant difference between this and old-style distributed apps – this time you no longer have network connectvity under your control. And as a result, you will face 3 significant phenomena that substantially complicate using today’s distributed algorithms – uneven bandwidth, uneven latency and increased probability of connectivity loss (I blogged about the latter here).

And this is what I call a hyper distributed application. In other words, hyper distribution application is a distributed app which runs on a network with uneven bandwidth, uneven latencies and increased probability of connectivity loss (as measured against that on a regular LAN), usually outside of application owner’s control (for example, Internet).

One example of a hyper distributed application is VPN-Cubed that we at CohesiveFT created to address emerging needs to multisource infrastructure. By the very nature of functionality it provides, its components (we call them VPN-Cubed Managers – they act as virtual routers and switches) are sometimes distributed over LAN, sometimes distributed over WAN, sometimes both. Communications between manager 1 and manager 2 can be fast and reliable, but between manager 1 and 3 slow and less reliable, with more frequent resets. Or manager 3 may simply disappear (as seen by its peers) – no, it doesn’t have to be down due to crash; it can simply mean that its network connection to the outside world was down, possibly temporarily.

Hyper distributed applications are relatively rare, because most architects tend to avoid this if they can. For example, Amazon EC2 has 2 regions – US and EU. Each region is a distinct EC2 system, with its own API endpoint, its own AMI IDs, kernel IDs, security groups, keypairs. There is no replication or conflict resolution between the regions – they are totally independent of each other. Why? Because it would be quite difficult to interconnect them into a single entity over public Internet. (I won’t be surprised if it gets implemented in the future though.)

Another example showing that hyper distributed applications are a distinct breed comes from Facebook Engineering blog post titled Scaling Out:

This setup works really well with only one set of databases because we only delete the value from memcache after the database has confirmed the write of the new value. That way we are guaranteed the next read will get the updated value from the database and put it in to memcache. With a slave database on the east coast, however, the situation got a little tricky.

When we update a west coast master database with some new data there is a replication lag before the new value is properly reflected in the east coast slave database. Normally this replication lag is under a second but in periods of high load it can spike up to 20 seconds.

It nicely illustrates how hyper distributed nature of the application adds complexity on top of what a plain distributed app already has.

In conclusion, I would like to propose to separate a category of distributed applications that run on top of networks with uneven bandwidth and uneven latencies into their own (I don’t care much if they end up being called hyper distributed or something else), and start building up research and practical approaches focusing specifically on this area.

P.S. Also consider the future: when we reach inter-planet or inter-galactic communications, you know that latencies and bandwidth in space would not be (initially) the same as on our planet Earth. Better start working on this research now in order to be prepared…

One huge difference between a pair and N-way (N>2) is how difficult it is to detect partitioning (of CAP Theorem fame – you can simultaneously achieve only two properties from the following three: data Consistency, high Availability and Partition tolerance). Assuming symmetrical communications (A can talk to B if and only if B can talk to A), partitioning detection in a pair is trivial, because there can be only one option – system A can’t talk to system B. With N>2 however, there are way more scenarios to deal with: A can’t talk to B while both A and B can talk to C, A can’t talk to B and C , etc. Additionally, communications may be restored in some random order – A may first be able to talk to B, and only some time later get its visibility to C back.

Interestingly, also from personal experience, if you manage to build a 3-way redundancy, building 4-way or even 5-way is relatively not that difficult.

There are also a couple of purely practical aspects that make a 2-way redundancy an attractive option, even if it’s going to be intermediate step before N-way is achieved. 2-way can serve as a working prototype – you can observe it, learn and analyze its failure scenarios and make sure your response to each is optimal. This can validate your approach before you sink all this time in partitioning detection for N-way.

And secondly, after you build an easier 2-way, you might as well discover that you don’t need an N-way redundancy. If a pair meets your goal (say a given percentage of service availability), you can save a lot of time and effort.

My advice – don’t skip two on your way from one to many.

Interesting. Looks to me it all depends on how you look at different clouds – as infrastructure providers or as software platforms.

The former case is roughly similar to buying Internet connectivity for your office from 2 different ISPs for redundancy.

The latter case, however, is roughly similar to a process of selecting platform for a project – say between Weblogic and JBoss. For a new project, a single platform is usually selected – I don’t think there are many cases when an app is built on top of both for better resiliency or to increase capacity (even though I admit that it’s not impossible).

In both cases, products are very similar or nearly identical to a certain extent, but the way you look at them makes you select 2 in one case and only 1 in another.

Right now, I think choosing a cloud is akin to selecting a software platform. So one will choose only one. However, the future may very well change this trend like you said, especially as interop gets better and each cloud gets its strengths and weaknesses better defined.

One interesting scenario popped up in my head. What if, after initial discovery of each peer’s internal IP address, one of the instances goes down (let’s say it was at IP1) and at least one other instance fails to notice this fact and continues to communicate with IP1. EC2 assigns IP addresses dynamically, and as far as I can say, IP1 can get assigned to someone else’s instance within same minute. So my instance will be unknowingly communicating with someone else’s instance – not something that I want to allow.

A solution can be something what the military call Identification Friend or Foe (IFF). You can read about it in Wikipedia or here. Note that you may want to consider an IFF anytime you are running applications in any IaaS cloud that assigns IP addresses dynamically and/or has no way of predicting which IP address your next host is going to get.

My Basic IFF Solution

First of all, my instances do not have access to AWS credentials (here is why). Secondly, I set up a requirement that all instances that needed to communicate with each other were to be launched with the same user-data (from a running instance, you can obtain user-data from http://169.254.169.254/latest/user-data).

I then created 2 checksums (SHA1 or MD5) – 4633e65fce4cf3b40648f574f4b60070 was a checksum of user-data plus some file in the AMI (say /usr/share/doc/coreutils/NEWS.gz) and 7a66a9361b14e95c14d98522502b9487 was a checksum of user-data plus another file in the AMI (say /bin/rmdir). Note that if user-data on each instance are the same, these checksums will be the same, because the files I selected are the same.

Then in Apache, I have the following configuration:

<Location /4633e65fce4cf3b40648f574f4b60070>
AuthType basic
AuthName 7a66a9361b14e95c14d98522502b9487
AuthUserFile /etc/apache2/users
Require valid-user
</Location>

Before establishing communications to a peer instance (and regularly afterwards), I set up my instances to get HTTP headers from above location (without actually submitting HTTP auth username and password), check WWW-Authenticate header and look for the second checksum there. Easy and efficient. If both checksums match, the other instance is a friend. If not, a foe. In this case I also assume that if I didn’t get a response, it’s not a friend – an instance might have gone down or it might not have apache listening on 80 or its web server might not know what to do with my URI.

You can further enhance this solution by creating new checksums every N minutes – this should work reliably for as long as EC2 infrastructure has no trouble keeping the clock accurate. You can also embed the timestamp in data used to generate checksums. Furthermore, if you monitor your access logs for bad interrogations (for example, old checksum or wrong checksum), you might be able to easily detect attacks against your IFF system.

A More Scalable IFF

Peer-to-peer IFF implementation that I described above may not work well for large deployments or for enterprise. If you are within either of these categories, I can recommend that you take a look at VPN-Cubed, a product offered by my employer CohesiveFT. Its features essentially serve as a scalable encrypted IFF.

But it’s not what I wanted to write about. As you might imagine, I read a lot on the subject – learning from other people’s mistakes and experiences in distributed systems world has never been easier, thanks to blogging and general tendency towards openness and disclosure. In all this stream of data that I get, the most frequent failure scenarios can by typically categorized as a “hardware crash” or “software crash.” Something was running fine, and then – BAM! – it crashed. It no longer exists. Nothing can talk to it anymore. Nothing can ask it how it’s doing, or what was the last thing it did. It crashed. Died. Disappeared.

But is crash the worst that could happen? Unfortunately not. Connectivity loss is way more tricky to deal with. Your Nagios thinks your web server crashed because it’s not responding? Can’t tell – not enough information. Everything you know is that nagios could not connect to the web server. It doesn’t mean that the latter crashed. Or you can’t connect to your messaging backend – did it crash? Not necessarily, everything you know at the moment is that connectivity between you and remote end is broken.

So why do I say the connectivity loss is way worse than crash?

1. Crash is the same crash to all clients. All clients will fail to connect. Connectivity loss however can impact only a fraction of your client base. So half of your clients are failing over to the secondary, while the other half are still attached to primary. And you neglected to implement an alarm for that – and now your customers see only half of your inventory on the site? Oops.
2. Crash is usually a terminal state, as in your application can’t easily leave a crash state on its own. And what about connectivity? Oh, not at all – connectivity can be restored without your direct intervention. It can range from route convergence after a backup link gets up, to easing network congestion after a spike in traffic. Are you going to be prepared?

And here is yet another twist. No matter what your position is on cloud computing, it is here to stay. And it is only a matter of time before many more services on which you rely for your operations will be scattered all over the world (or space, but that’s later). Connectivity loss will be occurring way more often than crashes, and unless you start approaching it as a different problem, you might be in for a big surprise.