I think that the biggest challenge for Infrastructure as Code today is its current lack of anything resembling a Makefile – a relatively simple description of how input could be transformed into output ready for use end to end, given a set of basic tools or a preset build environment (for example, for a project written in C it would be apt-get install build-essential on Debian and its derivatives). If you want an example, please take a look at deployment instructions for openstack/nova (”on the cloud controller, do this… on volume node, do that…”). While it is indeed infrastructure code, its end-to-end build and deployment instructions are provided in textual form, not as code.

Why is it a problem you may ask. First and foremost, build/deploy instructions provided in textual form can’t be easily consumed by a machine – it feels like we are back in the dark ages, without APIs where all work must be performed manually.

Secondly, because they are not fully formalized, they can’t be as easily shared – there could be many uncaptured context requirements that could lead to different people transforming identical inputs to outputs that would not function identically. And if they are not shared, same functionality is being worked on by many separate teams at the same time, which leads to incompatible, sometimes competing implementations and creates wasted effort by not allowing code reuse.

Thirdly, since they are not code, they are not as easy to test and verify test coverage for, or to fork and merge, or to port to other platforms.

My point is that while individual parts or steps of an infrastructure deployment could be automated, a whole thing rarely is, especially when a system is to be deployed to multiple hosts connected over the network. This would be similar to a software project with various directories, each with its own Makefile but without a top-level Makefile – such that you’d have to follow a HOWTO telling you which arguments to pass to make in each directory and in which order to run the commands.

What to do? I call on all infrastructure projects to make every attempt to ship deployment instructions not as textual step-by-step howto documents, but as code – be it Chef cookbooks, Puppet recipes, shell scripts, Fabric/Capistrano scripts and so on, or a combination of any of the above. Please consider providing cloud images (in at least one region of at least one public cloud) with your canonical build environment (your equivalent of build-essential). Please consider including canonical network topologies for your deployment – since you can’t predict IP addresses each user is going to allocate, all configuration files will need to be autogenerated or built from templates.

I am well aware it’s easier said than done, but if we do this, I hope a tentative consensus on best practices for infrastructure as code deployments could emerge over time which could then facilitate creation of a common “infrastructure make” tool.

Availability of your system, just as any other part of your service, is a feature. Just as with anything else, one needs to invest time, effort and resources in building it out. And whatever you dedicate to availability (such as development time) can’t be used for other features – this is what’s known as opportunity cost. If you could put same resources to a better use somewhere else, investing them in availability may not be the optimal decision. Additionally, availability draws from your complexity budget which is going to impact other areas – HA systems tend to be more complex and hence require more effort to develop, maintain and improve them over time. Availability, just as any other feature, has a price tag that you will have to pay to get it. Because you own your site’s availability, it’s up to you to decide how much availability you want AND can afford to build.

The last point is very important. Our daily lives are filled with points of failure – home appliances (can break), a usual route you take to work (could be impacted by road construction), your regular coffee place (your favorite barista could transfer to a different location). Do you maintain 2 different non-overlapping routes to work? Or do you frequent 2 coffee shops in order to have an alternative if one shop drops from your list? In other words, in our lives we regularly forgo availability when it doesn’t make sense – why shouldn’t we follow the same rule in our professional lives?

Availability is not a binary option. You could have all-active N-tuple, you could have active-active pair, you could have an active-passive pair with automatic failover, or same active-passive pair with manual failover. And finally, in today’s cloudy world, you could also have just a single resource with ability to replace this resource quickly if it goes down. Options include geographic redundancy, vendor/provider diversity, and so on. Availability could be as simple as host your systems at a very reliable provider. Or at the very least – be able to detect when there is a problem and be able to restore the system within a preset amount of time. Different levels of availability obviously don’t cost the same – pick one that you want and can afford.

Secondly, if your overall service consists of multiple smaller parts, you are free to choose different levels of availability for individual parts. Anything which responds to synchronous calls (a call that expects a reply immediately) – like web front door – may have one level of availability (higher), background jobs may have lower level. Designing each subsystem with appropriate level of availability will reduce your costs and most likely will let you save some of complexity budget for other things.

Thirdly, while availability is a single metric, problems that impact it are not. Some problems could be frequent and easy to deal with, other problems could be rare and catastrophic. Do you want to build your service to withstand a failure of a host, all hosts, all of your ISP, entire Internet? It’s all about the tradeoffs between costs, severity of each type of problem and probability of these problems to occur.

Fourthly, remember that availability measures that you build are your defenses against problems. A particular type of problem that you want to protect against, requires an availability measure targeted at this very problem – matching it by functionality, size and cost (a single defense measure may work against multiple threats). Imbalance in any of these three categories between your defenses and the problems they are meant to prevent will lead to suboptimal results. After all, you don’t use a shield to defend against a cannon and you don’t duplicate your entire operation into the second datacenter just to protect against a router failure.

And finally, beware of peer pressure. If your web front door’s availability costs $1m per month and it’s bringing in $10m per month worth of revenues,  it can be a no-brainer. But if you are investing 50% of your complexity budget in availability just because everybody else is doing it, I think it could be a problem.

Going back to AWS and putting my amateur behavioral economist’s hat on, I am curious how many people decided to take advantage of lower price for lower availability of RRS. And even more interestingly, if S3 initially were at RRS availability and AWS announced better availability for higher price, would we end up with the same distribution of people using higher and lower availability?

DevOps is on the rise as a newly re-defined standalone discipline, as evidenced by increased number of good articles about it around the blogosphere. In this post, I am going to take a stab at outlining what DevOps means to me.

I’ve got some devops cred. Before joining my current employer where my role morphed over time away from devops, for over 2 years I had been at Orbitz.com where I was in a group in charge of monitoring and automating a hugely distributed multi-datacenter custom airfare search application, running on many hundreds of machines, with several times as many separate entities and processes that needed to be coordinated, restarted, tweaked and so on (our group was in charge of everything above hardware, OS and basic network services such as connectivity, DNS and DHCP). Before that I had had various sysadmin roles, which all involved a large degree of coding beyond the level of simple shell or perl scripts.

To me, devops is a distinct discipline at the border between software engineering and ops, which focuses on developing software for the infrastructure on top of which end-user-facing software is running. It’s sometimes referred to as development of infrastructure software and includes release deployment. Devops has the following distinguishing characteristics.

1. Ability to write code beyond simple scripts

Obvious necessary condition.

2. Focus on stability and uptime

Stability and uptime in devops almost always trump features.

3, Extra focus on moving between states

In dev land, I have often observed situations when the end result of a particular feature was analyzed on its own merit, without taking into consideration how a system can be moved from its current state to its desired future state. Devops pays extra attention to this problematic area.

4. Different angle on business revenue

While developers usually work on things that are meant to increase or sustain business revenues, devops often work on things that are meant to prevent or reduce loss of business revenues. This is somewhat similar to defense vs offense in team sports. The key word is “balance.”

5. In devops, we are users of our own software

This is one of the most important distinctions. Unlike developers who create software to be used by someone else (internal customers, end users, site visitors, etc), devops is about developing software for internal needs. For example, you can certainly get sloppy in logging that error, but it’s you, not someone else, who’s going to suffer the consequences of having to waste extra time to find necessary information.

6. Architect, developer, tester, product manager, project manager – all in one.

My personal experience in devops is I/team get(s) an area of responsibility and it’s up to me/us to make it happen. Assigning priorities, figuring out dependencies, reacting to unexpected changes, managing resources – all of these functions are performed in devops by the same group of individuals.

7. Awareness of normal accidents

I have an entire blog post dedicated to this – check it out.

8. QA in production

Some tasks in devops can’t be adequately tested in smaller synthetic environments. Lack of scale, lack of unique hardware, lack of sufficient capacity in vendor’s test environment, lack of sufficient connectivity from test site to vendor’s systems – all could be factors. Phased deployment and other techniques designed to reduce the risk of a complete meltdown are (or should be) used extensively in such scenarios, but the truth is – from time to time in devops I had no other way but to actually run a test system in a live production environment.

9. Manual first, then automate

In my experience, a devops task is more likely to start out as something done manually at first, and automated later. In dev land, tasks rarely go through manual phase before being coded up and shipped in a release.

10. Almost always distributed or hyper-distributed

Conclusion

Devops is on the rise primarily due to realization that there is a big gap between developing end-user systems and bare-bones systems administration, in large part due to fast growth of IaaS cloud computing. Devops originated at places where a relatively few sysadmins were in charge of many hundreds or even thousands of hosts – where doing their job without automation was impossible. As time goes on, I expect devops will further solidify its role as a first-class citizen and make inroads into non-cloudy companies as well.

But here is a kicker – it’s not the end. I saw this tweet, read this post and checked out a book by Charles Perrow titled “Normal Accidents” from the library. Published in 1984, the book is not about IT, but its material fits our field nicely. And boy, was I enlightened!

The book’s main point: no matter how much thought is put into the system design, or how many safeguards are implemented, a sufficiently complex system sooner or later will experience a significant breakdown that was impossible to foresee beforehand, principally due to unexpected interaction between components, tight coupling or bizarre coincidence. For us in IT, it translates to “no matter how much planning you do or how many safeguards you implement, failures will still happen.”

There are at least 3 common themes that are present in multiple illustrations in the book:

1. A big failure was usually a result of multiple smaller failures; these smaller failures were often not even related
2. Operators (people or systems) were frequently misled by inaccurate monitoring data
3. In a lot of cases, human operators were used to a given set of circumstances, and their thinking and analysis were misled by their habits and expectations (”when X happens, we always do Y and it comes back” – except for this one time, when it didn’t)

I have had my share of outages and downtimes, and I can attest that I have seen these 3 factors play a big role in tech ops. Some were bugs in management and monitoring code, some where human error, some where bizarre set of dependencies but all were a combination of multiple factors. For example, who would have thought that with a failure of primary DNS resolution server, the VIP would not fail over to the secondary; and even though hosts had more than one “nameserver” line in /etc/resolv.conf, application timed out waiting for DNS to respond before getting to ask the second nameserver; without name resolution, multiple load balancers independently thought that there was no capacity behind them (because management code calculated capacity in near real-time relying on worker hosts’ names) and disabled themselves, thus taking down the entire farm – now I know of course…

It turns out we can’t eliminate normal accidents altogether, but here are several techniques that I have been using to speed up detection and response in order to reduce the downtime.

Complexity budget. Described by Benjamin Black, this is a technique to allocate complexity among components beforehand and strictly follow the allocation during implementation phase. It helps avoid unnecessary fanciness and leads to simpler code, which tends to be easier to troubleshoot and recover after a failure.

Control knobs/switches for individual components. As John Allspaw shows on this slide, you need to be able to turn off any component in an emergency, or throttle it up or down. Planning this feature and building it in from the very beginning is very important.

Accuracy of monitoring data. Ensure your alarms are as accurate as possible. No matter how much chaos is going on inside the system during a severe failure, last thing you can afford is misleading the operators with wrong information. If you tried to ping a host A and didn’t get a response, your alarm should not say “host A is down” because it’s not the knowledge you obtained – it’s an assumption that you made. It should say “failed to ping host A from host B” – maybe it was network on host B that was an issue when a ping attempt was made, how do you know?

Availability of monitoring data. There is a reason first thing the military try to do when attacking, is disrupting enemy’s means of communication – it’s that important, which applies to our case as well. You either design your systems to be able to get monitoring data even during the worst outage imaginable (ideally from more than one source), or you at least should be getting an alarm about lack of such monitoring data (it’s a very weak substitute though).

All in all, to everybody in IT, I highly recommend the Normal Accidents book as well as this whitepaper (linked from John Allspaw’s blog).

However, not all people know what an overlay network is or what its benefits and strengths are. This holiday season, as we were putting up our outdoor decorations and holiday lighting, I realized that what my wife and I were doing was essentially building an overlay network. Let’s follow the similarities.

Imagine a regular house with a front yard where for the holidays you want to set up a bunch of lighted Christmas trees, deer and other holiday figures. All of them require electricity – but there is no power installed in the ground (parallel with VPN-Cubed overlay network: you are deploying servers to third-party cloud and want to continue using your IP addressing schemes, want to ensure that all communications are encrypted – but provider doesn’t offer any of these services out of the box).

You don’t need power out on your front yard all year around – so there is usually no point in investing money in installing one. Cloud computing is all about elasticity. As a complement to clouds, VPN-Cubed is easy to set up and take down if necessary for an experiment, or it can be running for long periods of time.

There are several outdoor outlets on the front wall so you are deciding to power your decorations from these outlets (you have VPN devices installed on the edge of your network – you will use them to offer connectivity to your servers from your network using VPN). The first obvious solution is to run a power cord from each piece towards an outlet. While it’s possible in theory, it will turn out ugly in practice. Firstly, a lot of long outdoor power cords are expensive. Secondly, it will create a cabling mess near the outlet. Thirdly, if a cord goes bad, you need to trace where exactly it’s plugged in and replace it. Fourthly, the more stuff you have to power up, the more difficult this octopus made of power cords is going to be. Absolutely the same problems apply in our parallel use case.

So you come up with optimization #1 – you go out and buy several outdoor power strips with several outlets each. By placing these power strips where your lighted trees and deer are, you are reducing cabling issues, gain ability to use shorter power cords and most likely save money on power cords. That’s your VPN-Cubed manager server instance. When you place it next to your cloud-based servers, you reduce latency for your endpoints and cut down on VPN connections from the edge of your network that you need to build and maintain.

If you are well prepared (i.e., have enough of everything), your composition will drive how many power cords and strips you will need and how long your cords need to be, not the other way around. Same with VPN-Cubed – you mold it to fit your use case, your desired topology or application – you don’t adjust your application to be able to work within VPN-Cubed overlay network.

Outdoor power strips have additional protection to let them function outdoors in low temperatures. And so are VPN-Cubed manager instances – they are running a hardened OS, with minimal set of enabled services, behind firewall protection. You can grab a regular switch and make it work outdoors – but why waste your time when these things don’t cost that much? Same with VPN-Cubed.

But power strips may fail – and if they do, entire section of your composition will be turned off. So you get a cold standby sitting in your garage in case a primary goes out. Or better – you install 2 power strips next to each other, connect them and evenly plug in your endpoints. If one goes out, you switch all connections to the other strip and it’s back. VPN-Cubed allows you to deploy a hot spare with automatic failover capability, which can help balance the load as well. Your outdoor lighted Christmas tree is connected to one power strip at any given time, but if one fails it can be reconnected to another within a power cord distance. Same with VPN-Cubed – your servers are connected to a single manager at any given time, but if a manager becomes unavailable, your servers can automatically re-connect to another manager.

And what happens if one of your outlets goes bad? Moving a handful of cables to another outlet is much easier than moving a whole lot. Same with VPN-Cubed – if your network loses one entry point, you just re-connect VPN-Cubed to another.

There are many more parallels between the two. Most of us have been building overlay networks of decorations for quite some time. Building overlay networks for the cloud may be new, but CohesiveFT VPN-Cubed product makes it easy and fun. Don’t be stuck with long power cords – get yourself some nice outdoor power strips. And enjoy the holidays!

Disclaimer 2: Conclusions reached at the end of this post are my educated guesses, and may turn out to be not true. They are based on my observations and not on actual knowledge how EC2 internals are designed or implemented.

I was once working on a setup in Amazon EC2 and came across an oddity, which when coupled with my interest in EC2 security groups mechanism, turned into this post.

UDP hole punching, in a nutshell, is a technique which allows two cooperating hosts, potentially located behind NAT and/or firewalls, to establish a peer-to-peer UDP communication channel directly to each other. It’s a technique used by Skype, for example, – you can read more about it in a Wikipedia article. If two hosts start sending UDP packets to each other on pre-agreed ports, bi-directional flow of packets leads NAT devices and firewalls to think that all these packets are a part of an established communication channel.

EC2 allows a lighter form of this technique because EC2 NAT never rewrites source port of outgoing packet (recall that in EC2, NAT is always 1-to-1 such that port rewriting isn’t necessary). We know with 100% certainty that a packet we are sending with a given source port X will be seen by remote instance with the same source port.

I wrote a small Python tool (available at http://gist.github.com/224795) to test UDP hole punching and set out to discover if it could work in EC2. My expectation was that it should work. Unless explicitly noted, I used ports above 45,000 and none of security groups explicitly allowed UDP traffic on these ports.

I was able to easily punch UDP holes between any two instances using each instance’s public IP address – in line with my expectation. But I hit a major snag when using private IP addresses of 2 instances in the same region (I used EC2-US) – I couldn’t get it to work no matter what I tried: same availability zone, different availability zones, same security groups, different security groups, same AWS account, different AWS accounts. I even tried punching a hole over port 53 (all EC2 instances support DNS name resolution which happens over this port without an explicit corresponding rule in security groups) – no luck (EC2 DNS servers are not located on 10.0.0.0/8 where all instances reside).

The only way I could get it to work using private IPs, is to allow my UDP port in security groups of at least one of the instances. When I did this, both hosts reported success.

This observation leads to several thoughts that might help uncover some aspects of EC2 firewall’s internal design (these are all more or less educated guesses):

• You can punch a UDP hole between any 2 instances using their public IPs, even if your security groups do not allow such communication.
• Private IP traffic is treated totally differently than traffic over public IPs.
• You can punch a UDP hole on port X using private IP addresses of 2 instances in the same region only if at least one of the instances allows port X in its security groups (can be used as a test if you don’t have access to query EC2 API endpoint)
• EC2 firewall somehow implements more logic than “all outgoing packets are allowed” when dealing with traffic over private IPs (if it were not the case, hole punching should have worked – see below).
• If we assume that security group rules are applied at an instance’s dom0 (as makes at least some sense and as this research implies), I now suspect that all dom0 hosts have entire view of all security groups in the region and are getting real time updates when a rule is added or deleted (modification of rules is currently not supported). This in fact was contrary to my expectation – initially I thought each dom0 “subscribes” to updates for only those security groups which correspond to instances running on this dom0 and I thought this was the reason why dynamic group membership changes were not possible (say I want to move an instance from “db” security group to “webapp” security group).

To clarify: under the above assumption, in order for hole punching to NOT work, an outgoing packet from instance A must not reach dom0 of instance B – and the only way it’s possible under “all outgoing packets are allowed” policy is if dom0 of instance A knows that dom0 of instance B will block this packet and somehow takes this into consideration – which in general case can only happen if all dom0 hosts have entire view of all security groups and permissions in the region.

I would love to hear your thoughts on what could possibly explain this behavior, please let me know in the comments below.

This outage shed more light on some internal designs of EC2 itself, as described here. It might have also showcased our over-confidence in EC2’s ability to detect and defeat certain types of network attacks. But this post is about something else.

BitBucket was running their web front door and their backend application on the same instance. Front door is a part of the system which is facing the Internet and its task is to accept connections from clients. For obvious reasons, front door is running on the service’s discoverable IP address – whether they used Elastic IP or not, bitbucket.org resolved to that IP. Note that front door (usually) doesn’t need EBS.

Backend, however, is what needs EBS for disk persistence. At the same time, backend does not need to be publicly discoverable – as long as front door knows where its backend worker(s) is/are running, the app should be functioning just fine.

With front door and backend running on different instances, UDP flood would have saturated only the former’s network interface and would have had no impact on the backend and its EBS.

I know that AWS reportedly fixed the flood issue, but looks to me like separating front door and application backend may still be a good preventive measure – after all, it’s considered a good practice for a reason.

Please note that I am not trying to accuse BitBucket of running a bad architecture and causing their own outage. All I am doing is trying to learn a lesson.

One obstacle was that because we were emulating various failures, at times our local capistrano process, which was driving the tests, had to lose SSH connectivity to its target servers. We quickly discovered that this resulted in exception and cap process would die.

To work around this, I added yet another level on top of cap which uses GNU make (one of my all time favorites). In a nutshell, user controls the testing process via make, and make starts cap. In this case, it’s ok for cap process to occasionally exit.

But then – and we are finally getting to the point of this post – another issue came up: I didn’t want to keep typing password into cap each time it was started by make. Here is how I ended up implementing it to avoid re-typing password.


# in Makefile
USER_PASS := $(shell read -s -p "[make] user's password: " P; echo $$P )
export USER_PASS

all: set_password
# do something here

set_password:
    @test "$(USER_PASS)"


# in Capfile
set :password, lambda { ENV['USER_PASS'] ||
CLI.password_prompt("[cap] #{user}'s password: ") }