I recently came across something that was not available via EC2 API – lists of IP address ranges for each EC2 region. AWS team maintain such lists in their forums – currently at http://developer.amazonwebservices.com/connect/ann.jspa?annID=735 (I say “currently” because annID has already changed at least once). I asked for API, but in the meantime since I needed this now, I wrote a simple python script to scrape and parse the data – http://gist.github.com/559397 or embedded below. Enjoy! By the way, who would have thought that one would need to resort to scraping AWS, the very pioneers of infrastructure APIs, to get this simple bit of information?

If I were to query each endpoint one after another, I quickly discovered it would take too long. Therefore, I created a small helper class called BotoWorkerPool (in lib/boto_worker_pool.py), which wraps Python’s standard threading module around calls to boto – this helps achieve some amount of parallelism without introducing significant complexity of dealing with and sharing data among multiple processes. This also allows to potentially migrate to processing or multiprocessing libraries in the future, which offer threading-like interfaces for a multi-process model.

There are 2 tools at the moment.

onesnapshot.py creates new snapshots for all volumes that already have one snapshot marked with “__onesnapshot__” token. The rationale for this tool came in part from the following statement on AWS main page for EBS about durability:

The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot.

imageequiv.py takes AMI ID, kernel ID or ramdisk ID and finds equivalent IDs in all regions, based on matching name or manifest file location. This tool is a response to the following tweet of mine:

wanted – equivalence lists for kernel and ramdisk images (aki-, ari-) across all ec2 regions

Hope these are useful to someone.

Suppose there is a town with just one public IaaS cloud provider, and that every business in the town runs their own IT: some by hosting it on-premises, some by running it in the cloud. It seems reasonable to imagine that the cloud provider obeys the following rule: it runs IT for all and only those businesses that do not run their IT on-premises.

Under this scenario, we can ask the following question: is cloud provider’s own IT in the cloud or on-premises? (remember that the cloud provider itself is also a business)

Of course this “faux” paradox is not a paradox at all. As I suggested on Twitter, “if you build a cloud, to your customers it will indeed look like a cloud; but to you it will look like a regular datacenter.”

It has been about 6 months since I last blogged about work, so I figured an update may be in order, especially since today CohesiveFT announced availability of VPN-Cubed on Flexiant’s cloud offerings.

We’ve been very busy on VPN-Cubed engineering side. Along with features already on the roadmap, we delivered several enhancements that were directly driven or requested by VPN-Cubed users. On the product support side, we continued to expand a range of devices with which VPN-Cubed can do IPsec interop, which now include even ones I personally have never heard about before. We grew our experience and expertise in the area of troubleshooting intra-cloud and cloud-to-datacenter connectivity issues (there are many!). We’ve also worked on a few projects that required non-trivial topologies or interconnects, successfully mapping customer requirements to VPN-Cubed capabilities.

One theme that I have had in my head for some time now, is VPN-Cubed as the networking fabric of the Intercloud. Let me explain.

VPS was a predecessor of modern IaaS clouds. In VPS land, boxes are usually provisioned individually, one by one. Typical setups in VPS consisted of 1, 2 or 3 boxes. Networking 3 independent boxes together is relatively straightforward.

At the beginning of IaaS era, I imagine most setups were also 1 or 2 boxes. But as IaaS is gaining ground, topologies headed to the cloud are getting bigger, more complex and more dependent on access to external resources. Setting up networking consistently is becoming a bigger deal. But it’s not the end.

One of the roles of Intercloud is providing customers with an alternative (competition, in other words) – if one doesn’t like cloud A, she may take entire topology to cloud B. I’d say 99 of 100 public cloud justification documents being submitted to CIOs worldwide today include a statement saying something like this: “If this cloud provider fails to deliver what we need at a level we need it, we will switch to another provider.” This is actually not as easy in practice as it may sound.

Each cloud’s networking has unique aspects, no two are alike. Public IPs, private IPs, dynamic or not, customer assignable or not, eth0 private or public, cloud-provided firewall exists or not, peculiarities of firewall – these are some of the differences (as of today, I have set up boxes in 6 IaaS clouds with admin endpoints facing public Internet – I have seen many network setups). Taking images of N boxes from one cloud and dropping them in another cloud is well understood, recreating one cloud’s networking in another cloud is where the challenge is.

It is here where I think VPN-Cubed shines as a customer-controlled network abstraction – it’s an overlay built on top of service provider’s network, which allows it to be identical no matter what the underlying infrastructure looks like.

Same story plays out when an application is hyper-distributed and runs in multiple clouds or multiple regions of one cloud (where regions are connected via public Internet). And here as well VPN-Cubed provides an abstraction that allows one to treat all compute resources as being on the same network, regardless where they are actually located at the moment.

At the same time, VPN-Cubed can be appealing to topologies that don’t care about Intercloud. Networking and network security are areas that don’t get enough attention from cloud developers today, because developers are used to working within a perimeter. Excessively wide-open security group setups, using public IPs instead of private for communications, disabled local firewalls – these are all time bombs. They don’t affect the app right now (”look, it works!”) but they can be catastrophic over time when they could become an attack vector. For such topologies, VPN-Cubed provides a virtual perimeter that confines authorized communications to a mutually-authenticated tunnel encrypted end-to-end (are you sure you want to continue forcing your incoming web traffic to HTTPS but not encrypting writes and reads from app servers to database? or do you think application-level encryption could be better, faster or easier to maintain than transport-level?)

To get started with VPN-Cubed, visit http://cohesiveft.com/vpncubed. If you have a question about how VPN-Cubed can help in your particular use case, you can ask here.

UPDATE 08/21/2010: Randy Bias in his excellent new post touches on the very similar theme:

Where is the lock-in then? If it’s not the hypervisor, what makes moving from one cloud to another so difficult? Simply put, it’s architectural differences. Every cloud chooses to do storage and networking differently.

On several occasions, when troubleshooting a connectivity issue or verifying a deployment’s security, I needed to check local and/or remote security group rules from within my instances. I never have AWS credentials on the instances, so I couldn’t use the API. And EC2 metadata server (169.254.169.254) returns only the names of security groups, not the rules themselves. In other words, I needed to answer the following question – “do remote instance’s security groups allow or not allow communications from this instance on TCP/UDP port X.” Note that trying to open a connection and not getting a response doesn’t answer this question – packets could be blocked by security groups or by remote instance’s local firewall (iptables). It turns out there is a way…

Let me first start with a warning. You probably are OK using this technique against your own instances, or instances of your friends, partners, vendors and customers with their permission and when you know specific ports you need to probe, for verification or troubleshooting purposes. Probing someone else’s instances or random ports may get you in trouble – such activities could fall into a category prohibited by AWS Acceptable Use Policy. You have been warned.

Summary

This technique works using private IP addresses, so both instances must be running within the same EC2 region.

To check if remote instance’s security groups allow communications from your instance on UDP port $X, do this:

traceroute -p $X -w 1 -q 1 -m 1 $REMOTE_INSTANCE_PRIVATE_IP

To check if remote instance’s security groups allow communications from your instance on TCP port $X, do this:

tcptracerute -w 1 -q 1 -m 1 $REMOTE_INSTANCE_PRIVATE_IP $X

(tcptraceroute is not a standard tool installed by default; you can get it from here or apt-get install tcptraceroute on Debian or Ubuntu)

If you see “1 {IP address of first hop} XX.XXX ms“, remote instance’s security groups allow such communications from your private IP address. If you see “1 *“, they don’t.

Background

This technique is based on one of the conclusions of my UDP hole punching post.

In it, it looked to me like each dom0 in the region knows about all security group rules in this region. Additionally, it looked like when dom0 knows that remote dom0 will not accept traffic, it doesn’t bother even sending it. If we assume they do it on dom0 with iptables, they probably simply DROP such traffic (at least I would if I were them). And if they do, they won’t send ICMP Time Exceeded packet back to us – and hence, traceroute won’t be able to report the first hop.

If we do get that ICMP packet from dom0 back, it means it didn’t drop our packet. Which means it knows the other dom0 currently has a rule in security groups allowing it.

I set max_ttl to 1, since first hop in traceroute is believed to represent dom0.

Obviously, I verified this theoretical hypothesis on a pair of EC2 instances in us-east-1, and didn’t see any indication that it could be wrong.

Protecting Against Such Probes

If you have instances in EC2, you may be wondering if you could somehow protect them from such probes. After all, AWS network monitoring is top-notch, but they have a lot of hosts to watch over, so some small amount of probing may still occur and you may not even know it.

I’ve got good and bad news on this front. The good news is that you can do it, the bad news is it’s going to be ugly. Essentially, to avoid disclosing the ports open in your security groups to such probes, you must configure your security groups not to allow connections from arbitrary instances in your EC2 region. In other words, you must exclude 10.0.0.0/8 from all IP-address-based rules (rules granting access by name of security group are not affected). It means that you have to replace each rule that references 0.0.0.0/0 with multiple rules individually referencing 128.0.0.0/1, 64.0.0.0/2, 32.0.0.0/3, 16.0.0.0/4, 1.0.0.0/5 (I would avoid 0.0.0.0 just in case – hence 1 as first octet), 12.0.0.0/6, 8.0.0.0/7 and 11.0.0.0/8. Yep, 8 rules instead of 1. You have to also repeat the same exercise for all IP subnets referenced in your rules that include 10.0.0.0/8.

Conclusion

In the future, of course, I hope security groups API could be extended to support functionality “allow connections from Internet excluding private IPs in this region”. I am going to add it to my previous wishlist.

Speaking of public IaaS vs datacenter virtualization runoff, however, I would like to point out that Theo’s points make a lot of sense for someone who already has an established well-functioning server hardware and network bandwidth operation. If you do, indeed in a lot of cases, as Theo shows in his post, IaaS cloud may be a weak alternative.

But reality however is that not all organizations fall into that category. I needed to borrow a pen (!) the other day in the office, let alone have a screwdriver to rack up all those servers and insert more RAM into them. And I am not even talking about having a network cable punchdown tool at work. If I get tasked with building out a hardware platform to support business, I’d be quite stuck.

Dealing with hardware is a complicated process. Having an established relationship with vendors to get better service, building up volume to earn better prices, etc are not things that get accomplished overnight (btw, I have never dealt a lot with hardware, you can probably tell). Having space in a rack, having enough power cords, and so on and so forth – the list goes on and on.

Bandwidth is the same. It’s not like you can go to Best Buy to pick up whatever you need to get more bandwidth for the marketing campaign you are about to do.

It’s all about marginal cost – cost of adding one more server or one more Mbps of bandwidth. If you have nothing (or it’s dysfunctional due to required approvals, or ridiculous wait times, etc), IaaS cloud very well may look like a better alternative.

I hope you all follow Theo’s advice – “Use the cloud where it makes sense.” And please – stop the hype!

When you hear about abstraction in the context of virtualization-based IaaS cloud computing, the most well known abstraction is computing resources themselves (encapsulation is at play here as well). You don’t need to know exact hardware on which your instance is running, or exact network setup – you only need to be able to treat your compute instances as nearly identical units that respond to certain set of signals in a predictable way.

With emergence of multiple IaaS clouds however, there is a second abstraction that is going to play a big role – workload.

Workload is an abstraction of the actual work that your instance or a set of instances are going to perform. Running a web server or a web server farm, or being a Hadoop data node – these are all valid workloads. I am treating a workload as an abstraction, because I intentionally leave out a huge component – exact way how work in a given workload gets mapped to resources offered by a given cloud. When speaking in terms of workloads, I want to focus on what needs to be done, as opposed to how it’s going to be done in the context of a particular cloud (remember that from technical architecture perspective, clouds are far from identical).

For example, “run this blog for 1 year, for up to 100 visitors a day” is a what (workload), while “run this blog on m1.small EC2 instance in us-east-1 for 1 year” or “run this blog on Terremark instance with 1 VPU and 1 GB of RAM for 1 year” are a how (for lack of a better word, I am going to call them deployments).

I think such an abstraction is very helpful. Running this blog may take 1 small instance in one cloud, half of a small instance in another, a third of an instance plus a dedicated load balancer in third. As you can see, once you map a workload to a set of compute, storage and network resources offered by one cloud, you can no longer move it to another cloud – your deployments are non-transferable from one cloud to another. Workloads here serve as transferable equivalents of your cloud deployments.

Secondly, workloads by themselves may have properties or attributes that could dictate where workload can or can’t run. This justifies existence of a workload as a separate entity – it is in theory possible to construct a workload for which no deployment can exist in any of the clouds available today.

There are many examples what kind of attributes a workload may possess. A workload may have a compliance attribute, which says that this workload must run in an environment with a certain certification. Another attribute may be a geo location requirement, whereas it must run within a certain geographic region for a legal reason.

A workload may be time-bound (”runs for 5 hours”) or time-unbound. A workload may have a specific start time or flexible start time, in which case it may have a hard stop time (for example, must finish by a certain time in the future). It can be interruptable or must run without interruptions.

A workload may have a certain lower limit of resources that it needs, expressed in work-independent form. For example, serving Wordpress blog for 1 visitor a day as opposed to 100 visitors an hour are two very distinct workloads (note that workloads are different, while the application inside the workload is the same). The latter will certainly end up consuming more resources than the former.

Workload may have a budget associated with it, it may have redundancy requirements. It may require a certain OS or distribution. It may require a certain feature (for example, persistent disk or non-private IP address directly attached to eth0). It may require a certain minimal access speed to some data source (for example, if my data are in S3 on the East coast, I may want my workload to run somewhere near). Each requirement is a restriction – the more requirements a workload has, the fewer clouds can potentially run it.

Conclusion

The answer to the question “Where is the best place to run this task?” used to be treated as a binary decision (”on premises” vs “in the cloud”) but not any more – because there are many different and incompatible implementations of the latter. Looking at your tasks via workloads/deployments prism may open new horizons for computing mobility. There is a saying “select the right tool for the job.” It can be now extended to “select the right tool and the right location for the job.”

If you like the idea of cloud computing workloads, you may find this post by James Urquhart interesting as well.

P.S. Believe it or not, this is my 100th post on this blog. Not bad. Hope at least some of you enjoy reading my posts as much as I enjoy writing them.

Several notes first.

1. All references to times and dates below are GMT for all regions.
2. Spot instances went live on December 14, therefore I ignore all data points before that (for simplicity, my cutoff was set at UNIX timestamp 1260777600 – it’s 8am on December 14 GMT, which translates to midnight in Seattle where AWS is headquartered).
3. Spot price history was obtained on January 25, 2010 at 10:54pm via API and cached locally for analysis.
4. In order to be able to deal with integers instead of floats, all prices below are represented in points where 1,000 points = $1 per compute hour.
5. Each product is specified as [region, instance_type, product_description] tuple.
6. I am only going to outline facts below, all interpretation is up to you.
7. These results have not been exhaustively verified, my analysis code may have bugs. Use at your own risk.

#1 Price averages

Here is a chart of average spot price for each product relative to regular price for the same product (averages take into consideration for how long each price was valid). Percentage next to a product identification represents the ratio between average spot price and regular price.

#2 Price increases in a row

Maximum number of price increases in a row was 6. It occurred on January 23-24 for [us-west-1, m1.large, Windows] and the price went up from 256 to 273.

5 price increases in a row happened also once, 4 in a row – 16 times, 3 in a row – 95 times, 2 in a row – 643 times, and a single increase immediately followed by price reduction happened 2,433 times. Of the latter, 684 times (28%) were a single price increase followed by price returning to where it used to be right before the increase (X -> X+Y -> X).

#3 Individual price increases

Maximum single price increase in absolute terms was 928 – it occurred for [us-east-1, m2.2xlarge, Windows] when the price went up from 572 to 1,500. Second biggest was 890 for [us-east-1, m1.large, Linux/UNIX] and third biggest – 551 for [us-east-1, m1.small, Windows]. Note that all of these occurred in us-east-1.

The biggest price increase as percentage of the regular price was 460% when a price for [us-east-1, m1.small, Windows] jumped from 49 to 600 on January 24. The second and third biggest in the category were 262% increase for [us-east-1, m1.large, Linux/UNIX] (110 -> 1000) and 64% increase for [us-east-1, m2.2xlarge, Windows] (572 -> 1500).

The same two biggest increases were also the biggest price increases as percentage of current spot price – 1,124% and 809%, respectively. Third place in this category was a 186% increase for [eu-west-1, m1.small, Linux/UNIX] when the price went up from 28 to 80.

Here is a chart showing price increases and reductions day by day.

#4 Number of datapoints per product and/or product family

There were a total of 4,469 spot price revisions for Windows and 3,885 for Linux/UNIX. By region, us-east-1 had the least price revisions in total – 2,491, of which 1,254 were for Windows and 1,237 for Linux/UNIX (50.3% vs 49.7%). A total of 2,809 price revisions in eu-west-1 were distributed 1,518 for Windows vs 1,291 for Linux/UNIX (54% vs 46%). A total of 3,054 price revisions in us-west-1 were distributed 1,697 for Windows vs 1,357 for Linux/UNIX (56% vs 44%).

[eu-west-1, m1.small, Windows] had the most price revisions – 287. [us-east-1, m2.4xlarge, Windows] had the least – 40.

Across all regions combined, the most price revisions per day happened on January 22, 2010 – 351 price revisions.

#5 Percentiles

Here is a Google Fusion table with percentile estimates for each product. I tried to calculate percentiles from 50th through 95th (step 5) and 99th, but since a price function consists of discrete values, not all percentiles could be estimated. For each percentile, a nominal price is provided along with its percentage of the regular instance price for a given product. Percentiles take into consideration for how long a given price was valid.

#6 Spot price over regular price

Situations when a spot price is equal or exceeds regular price are especially interesting. Most such situations occurred in us-east-1, and none of them occurred in eu-west-1.

Spot price has reached but not exceeded the regular price for [c1.xlarge, Linux/UNIX] twice, [c1.medium, Windows] twice, [m1.small, Windows] 6 times, [m1.large, Windows] once – all in us-east-1.

In us-west-1, spot price for [m1.large, Linux/UNIX] exceeded the regular price by 20 for under 2 hours on December 29.

Spot price for [us-east-1, m2.2xlarge, Windows] exceeded the regular price by 60 for over 20 hours on January 11-12.

Spot price for [us-east-1, m1.large, Linux/UNIX] exceeded the regular price by 64 on December 17 and by 660 twice on December 18.

And finally, spot price for [us-east-1, m1.small, Windows] exceeded the regular price by 480 once and by 430 once – both on January 24.

Conclusion

There are hardly any surprises in the spot price history so far, but it’s only been less than 2 months since the feature was launched. As the usage ramps up, I expect it will become more interesting. Kudos to AWS team for coming up with this innovative pricing mechanism and being the first to introduce it at such a large scale in a real environment. Only time will tell if it will stick in its current form or if it will morph into something else (I have a couple of ideas), but the first small step towards dynamic pricing of computing resources has been made.

However, not all people know what an overlay network is or what its benefits and strengths are. This holiday season, as we were putting up our outdoor decorations and holiday lighting, I realized that what my wife and I were doing was essentially building an overlay network. Let’s follow the similarities.

Imagine a regular house with a front yard where for the holidays you want to set up a bunch of lighted Christmas trees, deer and other holiday figures. All of them require electricity – but there is no power installed in the ground (parallel with VPN-Cubed overlay network: you are deploying servers to third-party cloud and want to continue using your IP addressing schemes, want to ensure that all communications are encrypted – but provider doesn’t offer any of these services out of the box).

You don’t need power out on your front yard all year around – so there is usually no point in investing money in installing one. Cloud computing is all about elasticity. As a complement to clouds, VPN-Cubed is easy to set up and take down if necessary for an experiment, or it can be running for long periods of time.

There are several outdoor outlets on the front wall so you are deciding to power your decorations from these outlets (you have VPN devices installed on the edge of your network – you will use them to offer connectivity to your servers from your network using VPN). The first obvious solution is to run a power cord from each piece towards an outlet. While it’s possible in theory, it will turn out ugly in practice. Firstly, a lot of long outdoor power cords are expensive. Secondly, it will create a cabling mess near the outlet. Thirdly, if a cord goes bad, you need to trace where exactly it’s plugged in and replace it. Fourthly, the more stuff you have to power up, the more difficult this octopus made of power cords is going to be. Absolutely the same problems apply in our parallel use case.

So you come up with optimization #1 – you go out and buy several outdoor power strips with several outlets each. By placing these power strips where your lighted trees and deer are, you are reducing cabling issues, gain ability to use shorter power cords and most likely save money on power cords. That’s your VPN-Cubed manager server instance. When you place it next to your cloud-based servers, you reduce latency for your endpoints and cut down on VPN connections from the edge of your network that you need to build and maintain.

If you are well prepared (i.e., have enough of everything), your composition will drive how many power cords and strips you will need and how long your cords need to be, not the other way around. Same with VPN-Cubed – you mold it to fit your use case, your desired topology or application – you don’t adjust your application to be able to work within VPN-Cubed overlay network.

Outdoor power strips have additional protection to let them function outdoors in low temperatures. And so are VPN-Cubed manager instances – they are running a hardened OS, with minimal set of enabled services, behind firewall protection. You can grab a regular switch and make it work outdoors – but why waste your time when these things don’t cost that much? Same with VPN-Cubed.

But power strips may fail – and if they do, entire section of your composition will be turned off. So you get a cold standby sitting in your garage in case a primary goes out. Or better – you install 2 power strips next to each other, connect them and evenly plug in your endpoints. If one goes out, you switch all connections to the other strip and it’s back. VPN-Cubed allows you to deploy a hot spare with automatic failover capability, which can help balance the load as well. Your outdoor lighted Christmas tree is connected to one power strip at any given time, but if one fails it can be reconnected to another within a power cord distance. Same with VPN-Cubed – your servers are connected to a single manager at any given time, but if a manager becomes unavailable, your servers can automatically re-connect to another manager.

And what happens if one of your outlets goes bad? Moving a handful of cables to another outlet is much easier than moving a whole lot. Same with VPN-Cubed – if your network loses one entry point, you just re-connect VPN-Cubed to another.

There are many more parallels between the two. Most of us have been building overlay networks of decorations for quite some time. Building overlay networks for the cloud may be new, but CohesiveFT VPN-Cubed product makes it easy and fun. Don’t be stuck with long power cords – get yourself some nice outdoor power strips. And enjoy the holidays!

The lightning talks section started with Microsoft representative giving an overview of Azure. Then, John Willis gave a talk titled What color is your cloud in which he talked about various types of IaaS clouds. George Reese gave a talk where he compared a successful cloud deployment to reaching the Emerald City and pointed out that the yellow brick road to this goal is not always an easy one (great analogy!). Iron Mountain representative gave a talk about how one needs to be aware where their data are in the cloud, and emphasized security measures at their datacenter. Intuit representative talked about their PaaS, which allows developers to easily reach millions of small businesses already running Intuit products. PaaS is not my thing, but the idea makes sense and if I understand correctly, is very similar to idea behind Salesforce platform – develop against something which many organizations already use. And finally, Cory Von Wallenstein of Dyn, operators of a well-known DynDNS.org, gave a talk about their enterprise features like anycast DNS, CDN etc. Interestingly, now that I think about it, DynDNS offered a way to update DNS programmatically way back when, which definitely qualifies them as one of the earliest cloud APIs out there.

After a break, I attended a cloud security talk titled Cloudifornication by Chris Hoff. I’ve seen the slides and video of this talk before (for example, see here), but seeing it live was more than worth it. This is a very good and important talk for all cloud practitioners and especially architects and developers, and I highly recommend it. I personally had 3 main takeaways. Firstly, information security is based on C-I-A (confidentiality, integrity, availability). Therefore, any outage or service disruption is classified by a customer as a security issue, not only as an SLA issue. I didn’t know about this until a couple of weeks ago when Chris explained it to me on Twitter, and the talk also emphasized this fact. Secondly, I loved a series of slides about increasing complexity of interconnects as more and more vendors, intermediaries and brokers are added to one’s cloud mix. We at CohesiveFT are very aware of this as an emerging issue, and our VPN-Cubed product is targeted at such cases, among other things.

And thirdly, Chris very skillfully highlighted the brittleness of the foundation on top of which we collectively as an industry are currently building out our cloud offerings. When Internet was designed as it was for world wide web and static pages, it was all good. When we started doing e-commerce and social media on top of the same infrastructure, the risk increased many-fold but was still somewhat manageable (after all, it’s only buying stuff online). But now with cloud computing we are putting absolutely everything (!) on top of the same brittle foundation, and the risks are truly enormous.

Then, I attended a session on private cloud led by John Willis, where we discussed various private cloud technologies and ideas. My main takeaway was that there is or there will be a huge demand for private externally-operated clouds for mid-sized organizations, and that’s where I think future of colo and hosting is going to be.

All in all, this was a great event and thanks to all organizers and sponsors for putting it together, and to all participants for interesting discussion.