I recently came across something that was not available via EC2 API – lists of IP address ranges for each EC2 region. AWS team maintain such lists in their forums – currently at http://developer.amazonwebservices.com/connect/ann.jspa?annID=735 (I say “currently” because annID has already changed at least once). I asked for API, but in the meantime since I needed this now, I wrote a simple python script to scrape and parse the data – http://gist.github.com/559397 or embedded below. Enjoy! By the way, who would have thought that one would need to resort to scraping AWS, the very pioneers of infrastructure APIs, to get this simple bit of information?

If I were to query each endpoint one after another, I quickly discovered it would take too long. Therefore, I created a small helper class called BotoWorkerPool (in lib/boto_worker_pool.py), which wraps Python’s standard threading module around calls to boto – this helps achieve some amount of parallelism without introducing significant complexity of dealing with and sharing data among multiple processes. This also allows to potentially migrate to processing or multiprocessing libraries in the future, which offer threading-like interfaces for a multi-process model.

There are 2 tools at the moment.

onesnapshot.py creates new snapshots for all volumes that already have one snapshot marked with “__onesnapshot__” token. The rationale for this tool came in part from the following statement on AWS main page for EBS about durability:

The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot.

imageequiv.py takes AMI ID, kernel ID or ramdisk ID and finds equivalent IDs in all regions, based on matching name or manifest file location. This tool is a response to the following tweet of mine:

wanted – equivalence lists for kernel and ramdisk images (aki-, ari-) across all ec2 regions

Hope these are useful to someone.

And here are my reasons.

Memory Management

Shell scripts are excellent in managing their memory and one has to try real hard to cause a shell script to leak memory. This makes shell a very convenient tool for long running processes, supervisors in multiple-workers models, daemons and so on. There is an easy explanation for this. In shell, there are only a handful of built-in primitives – everything else is an external command, which gets started and then finishes before giving control back to your script. If there is a memory leak in that command, it won’t damage your calling script and will usually be insignificant because it will return quickly.

No Exceptions

This is a double edged sword, and you need to be careful how you exploit this “weakness.” This feature allows me to write compact code which is easy to understand without enclosing every single command in “try… except”. For naysayers, I would like to point out that a strict mode exists, where every error is treated as fatal and causes the script to exit (set -e).

In general, not all unforeseen error conditions warrant a crash, like you get in Python or Ruby when an unhandled exception gets propagated all the way to the top. If a problem is transient, it may be better to ignore it temporarily.

To assure a Ruby or Python script doesn’t crash on some unforeseen transient problem, many people often end up enclosing their entire program in a wildcard try… except block to catch any exception – but to me this approach is dangerous, even though I sometimes end up using it myself.

If you are writing a daemon process to perform some action in a loop, shell is often by far the most stable alternative.

When Not To Use Shell

My personal rule of thumb is don’t use shell when you expect to need high-level data structures like hashes or arrays beyond what for loop can give you, or when you can see potential for code reuse following OOP patterns like inheritance, or when your program needs to participate in some orchestration schemes that go beyond creating and removing files on the filesystem.

Conclusion

I wouldn’t overlook shell if I were you.

Graphite CLI Screenshot

Graphite is a flexible and powerful tool to build charts. It’s also a data series analytics framework. It was developed inside Orbitz by my former colleague, originally for use within a single group (of which I was a part). However, its power did not remain a secret for too long – it quickly spread to entire organization and became an irreplaceable tool for both development and engineering/operations. Graphite was then open-sourced under Apache license. It currently lives at http://graphite.wikidot.com/

The key to Graphite’s power, in addition to dynamic web UI, an improved RRD implementation called “whisper” (read this FAQ – highly recommend!) web-based command line with auto-completion which allows you to overlay any metrics on a single chart, IMHO is the fact that you are in control what kind of data to send to it, how often, and how to set up hierarchies of your metrics – by environment, by machine type, by datacenter, etc. Graphite doesn’t do its own polling that won’t scale to hundreds or thousands of metrics. Nor does it enforce anything but the fact that your metrics are dot-separated hierarchies (as in routing keys of AMQP topic exchanges – my.metric.name) and that their values are numeric (int or float).

If you are still reading this but still are not convinced that it’s the way to go, I’ve got one last argument. If you already use RabbitMQ to publish and consume data, wouldn’t it be nice to get a powerful charts without touching your application AND without installing agents on your publishers or consumers? Recall the duplication pattern of RabbitMQ – you can fork the incoming stream of messages into another queue (without impacting your original consumers and the queues to which they attach) and set up Graphite+RabbitMQ off of this new queue.

If you are planning to run multiple carbon instances, remember that heavy lifting (writing to disk) is actually performed by another process called carbon-persister.py (it’s started by carbon-agent, with communications over a pipe) – try to avoid multiple persisters writing data within the same hierarchy to avoid slow down and possible data corruption. RabbitMQ can help you sort out what messages go where, thus minimizing this risk.

I am very excited about future opportunities that a Graphite-RabbitMQ combination can deliver, and I hope someone finds my scripts useful. Both tools bring a lot of awesomeness to the table, and nicely complement each other forming a great charts and data series analytics solution you have been searching for. Check it out!

• Mac. I got a Macbook Pro as my work laptop. Feels great every time I sit down to work, even though I am not a very demanding desktop user – browser, email, IM, rss reader for non-public feeds, terminal, occasional Office document work, occasional PDF to read. And most importantly – VMware Fusion to run my Linux development box.
• Debian. Our paths surprisingly have not crossed until this year. Probably because I lost interest in low level OS maintenance a long time ago and was not actively looking for a new distro to try (my home server is running Red Hat Linux 7.0 with everything important upgraded to later versions – I don’t use RPMs and prefer GNU Stow for package management). Overall, I can’t say I dislike it. apt-get is nice. And deb packaging format imho is superior to rpm (writing standalone rules files using GNU make is easier than writing shell within spec files). Like rpm however, apt-get does not handle edge cases, when I don’t want computer to think it’s smarter than me.
• Ruby. The second attempt to learn ruby was successful. Ruby now joined Python on the list of my favorite languages. They are both equally great for doing all kinds of development, provided your project doesn’t use both at the same time – each has its own patterns and I personally find it somewhat hard to adjust when I try to write in both in parallel. I am also happy that Ruby joined Python on my list, not displaced it (several years back, Python did displace Perl on my list) – so I am kind of bi-lingual now, which mimics my real life situation with real languages.
• Rails. Yes, Ruby on Rails and Django on Python are my favorites now.
• RabbitMQ. This is a big one. For quite some time I have been searching for a highly available messaging bus as a holy grail of building and operating a scalable system. I think I found what I was looking for in RabbitMQ and its implementation of AMQP standard.

Elastic Server On Demand is built with each of these somehow tied to it behind the scenes. Shaping up very good so far…

Google released their App Engine yesterday. Very cool python goodness, supposedly with access to their BigTable and GFS and supposedly free for use up to a certain limit.

In the meantime, Google engineers (or was it designers?) came up with a new type of aircraft. You probably have heard about flying wings, but it looks to me like the concept of “a flying engine” has been introduced for the first time. And it’s a very simple design too – you take a jet engine, and attach wings and a tail to it. I am not a specialist, but I don’t see a reason why it won’t fly.

Seriously though, Google definitely has my respect for doing things like this.

A note to self: come back to to this link when it’s time to migrate to py3k.

There are at least 2 scenarios where non-blocking sockets help, and both are applicable to consumers:
1) when you want to be able to interrupt consumer’s event loop without waiting for a next incoming message;
2) when you want to consume messages from multiple brokers (or over multiple connections) in a single event loop.

Special thanks to Barry Pederson who created py-amqplib library and made it available to the public.

Please see my announcement on rabbitmq-discuss mailing list. If your architecture still relies on custom or proprietary messaging protocols, I strongly suggest that you take a look at AMQP and especially RabbitMQ.

UPDATED: You can download  the add-on directly from http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080302/455b87df/attachment.txt

UPDATED x2: If you came to this post today (2009 and beyond), please note that this add-on has been out of date for some time. Non-blocking AMQP with python can be done today with txamqp if you are using twisted, or with pika.