It’s got many lessons for software engineers as well. For example, in chapter 9 “Phases of the game” Kasparov talks about inexperienced players blindly following openings by famous grandmasters and how this can carry one only so far and ultimately is a trap.

Players, even club amateurs, dedicate hours to studying and memorizing the lines of the preferred opening. This knowledge is invaluable, but it can also be a trap. Many make the mistake of believing that if they know what a famous Grandmaster played in this exact position back in 1962, they don’t have to think for themselves. [...] Without knowing why all the moves are made, he’ll have little idea of how to continue when play inevitably advances beyond the moves he was able to store in his memory.

In software engineering, we have many conferences and online tutorials and blogs where our own Grandmasters talk about how they tackled a particular problem or resolved a particular outage. Sharing experiences is invaluable, but like Kasparov says, it can only carry you so far. Many people will blindly follow solutions described during conference talks, without understanding why it was done this way and not the other. Some people base their selection of a certain technology on opinion of a guru. Again – without fully understanding the context and reasons behind the decision.

What I am trying to say is Learn from other people’s experiences, but don’t forget to understand their context and their reasons. Your ability as a software engineer is based on your ability to adapt the solution to your needs, not simply copy it. Or if you copy, you need to know exactly why it will work for you.

Earlier this month, Seth Godin, one of the leading thinkers of our digital age, wrote a post titled Do ads work?

If the local bank were offering a sale on dollar bills, ninety cents each, how many would you buy?

Most rational people would say, “I’ll take them all please.” Especially if you had thirty days to pay for them.

So, why, precisely, do you have an ad budget?

If your ads work, if you can measure them and they return more profit than they cost, why not keep buying them until they stop working?

And if they don’t work, why are you running them

Some of you may recall my love towards parallels between IT and other disciplines. If we twist these words a bit, we get this:

So, why, precisely do you have specific maintenance windows which are the only times when changes in production are allowed – don’t they just unnecessarily delay the changes that inevitably are going to be made?

If the changes you are about to make work, if you can measure them and they improve your product more than they cost, why not keep making these changes until they stop working?

And if they don’t work, why are you making these changes?

I know it’s a stretch but worth thinking about nevertheless… Release often FTW.

In economics, this is called a non-zero-sum situation.

But it’s not what I wanted to write about. As you might imagine, I read a lot on the subject – learning from other people’s mistakes and experiences in distributed systems world has never been easier, thanks to blogging and general tendency towards openness and disclosure. In all this stream of data that I get, the most frequent failure scenarios can by typically categorized as a “hardware crash” or “software crash.” Something was running fine, and then – BAM! – it crashed. It no longer exists. Nothing can talk to it anymore. Nothing can ask it how it’s doing, or what was the last thing it did. It crashed. Died. Disappeared.

But is crash the worst that could happen? Unfortunately not. Connectivity loss is way more tricky to deal with. Your Nagios thinks your web server crashed because it’s not responding? Can’t tell – not enough information. Everything you know is that nagios could not connect to the web server. It doesn’t mean that the latter crashed. Or you can’t connect to your messaging backend – did it crash? Not necessarily, everything you know at the moment is that connectivity between you and remote end is broken.

So why do I say the connectivity loss is way worse than crash?

1. Crash is the same crash to all clients. All clients will fail to connect. Connectivity loss however can impact only a fraction of your client base. So half of your clients are failing over to the secondary, while the other half are still attached to primary. And you neglected to implement an alarm for that – and now your customers see only half of your inventory on the site? Oops.
2. Crash is usually a terminal state, as in your application can’t easily leave a crash state on its own. And what about connectivity? Oh, not at all – connectivity can be restored without your direct intervention. It can range from route convergence after a backup link gets up, to easing network congestion after a spike in traffic. Are you going to be prepared?

And here is yet another twist. No matter what your position is on cloud computing, it is here to stay. And it is only a matter of time before many more services on which you rely for your operations will be scattered all over the world (or space, but that’s later). Connectivity loss will be occurring way more often than crashes, and unless you start approaching it as a different problem, you might be in for a big surprise.

On your phone, head over to http://www.google.com/gwt/n and enter URL you are trying to get. GWT (which stands for Google Web Toolkit) will fetch the content and optimize it for your mobile browser. Additionally, it will adjust all links to also go through GWT, which makes Internet surfing with Blazer not painful at all.

For example, I like checking Techmeme on train on my way to work. They offer mini version, which renders well on my Palm Centro from Sprint. But if I want to follow a story and click on a link, I usually get the page not optimized for mobile (there are several exceptions that detect user agent and adjust content formatting). Instead, in my Blazer bookmarks, I have this – http://www.google.com/gwt/n?u=http%3A%2F%2Ftechmeme.com. From this page, I can jump to any story and get the content nicely formatted for my Centro.

UPDATE: It looks like I might have confused Google Wireless Transcoder (GWT) with Google Web Toolkit (GWT).

I did it a couple of weeks ago when I was moving this site to its current address, but noticed today that my old address still shows up in Google at the top of search results. I got curious, and checked both Yahoo and MSN and both of them properly do not display links that have been redirected.

Am I missing anything, or is it a bug in GoogleBot?

Preamble

I have spent a big part of my career in technology operations of small, medium and huge companies, so a concept of “operations alert” is very dear to my heart. For those who are not familiar with this concept, operations alert is an automated message about something in your IT environment or infrastructure that went wrong. For example, a server crashed or application stopped responding. Some people call these things “alarms” instead of alerts.

These messages can take many forms. When a company is small, it almost always starts with alerts sent out as email messages or SMS. Later on, as the number of alerts sent and analyzed each day grows, companies usually deploy a dedicated system that centralizes, aggregates and presents the alerts in a more manageable way. It’s usually a client-server architecture, where clients are monitoring agents deployed on all or most machines, that send the information to a central server for processing. Or sometimes there are no agents, and server regularly performs checks (sends probes, also sometimes called active monitoring) of network services and generates alarms off of responses (or lack thereof). Examples of open source solutions in this area are Big Brother (and clones/descendants), Hyperic, OpenNMS, Nagios, Zabbix, Zenoss and many others.

When organization gets a ton of alerts each day, it needs to prioritize them. And a concept of “alert severity” is born. It’s usually one of “critical”, “major”, “minor”, “warning”, “info” and “debug”. The higher the severity, the more important an alert is and the sooner it needs to be analyzed. Usually, alerts are created by specialized engineers who are responsible for a particular server or application (called SME – subject matter expert), while people who receive them and react to them are generalists (engineers not focusing on a particular technology but with very broad expertise in system and network administration).

Who Sets Severity?

I looked at many tools and observed how several organizations implemented operations monitoring, and I noticed a pattern – alerts severity is set by SMEs (I was such an SME up until recently). An SME analyzes the pool of alerts that his systems can ever generate, rates them by how important they are, and assigns priorities accordingly. Generalists monitor the dashboard and supposedly react to alarms in the order of decreasing severity.

All good, right? Wrong! Enter the Tragedy of the Commons. Generalists’ attention and time are finite resource. In order for SME to get attention to alerts sent from his systems, he tends to inflate severity of his alerts to draw more attention of generalists. As a result, quite soon, all your alerts are marked “critical”. All SMEs combined would be better off if *all* their peers fairly assigned severity, but each individual SME is better off if they inflate the severity for alerts sent by their systems. Niiiiice!!!

Solution

I think there might be a solution to the tragedy of commons problem in IT operations monitoring after all. It’s easy to explain but difficult to implement. Your alerts should not have severity at all. In other words, when an alert message reaches central server, it should have no severity. Once an alert is received, its severity should be a function of real-time status of entire environment. One minute a fan failure on your secondary DNS server is top priority (and hence a “crit”), but next minute a network interface failure on your primary DNS becomes a much higher priority. And of course web front door outage half an hour later easliy trumps both of these problems (provided they are not related of course).

I have some ideas how this can be implemented, but not ready to write them up yet. For now, when you evaluate monitoring solutions and vendors, consider that red severity field in their nice screenshots and ask yourself if it’s going to help you achieve better operations efficiency, or lead you down the path of the tragedy of the commons.