Query API vs REST API

To start, one should read this blog post by Jan-Philip Gehrcke about various types of AWS APIs and differences between RESTful and query API, and this blog post by William Vambenepe where he analyzes various IaaS API implementations (it’s a series of 3 posts). Then read description of Richardson Maturity Model by Martin Fowler.

In a nutshell, I think from practical standpoint, if one’s domain maps easily to a set of entities (nouns) and API operations on these entities are primarily CRUD, in this case one’s best bet is to go with at least Level 2 REST. If either doesn’t work, I’d go with Level 0 REST, which is essentially what query API is.

My main reason for not going with Level 0 when entities and operations do map, is that I hate to see this meta data go to waste because it doesn’t cost almost anything to include.

Between Level 2 REST and Level 3 REST, I think Level 2 is more practical. According to Fowler, “Level 3 introduces discoverability, providing a way of making a protocol more self-documenting.” It’s certainly a nice feature but I am not sure this added benefit justifies extra development effort and slightly increased complexity (some might argue it may actually reduce complexity though).

API frontend vs API methods implementation

Keep implementation of your API methods separate from whatever frontend you are deploying (REST, SOAP, etc). API methods are probably going to be the same no matter how they are called, so they should be frontend-independent. This will make it easier for you to introduce new frontends (AMQP, for example) and should facilitate code maintenance.

HTTP verbs

Read and delete operations are easy – they map to GET and DELETE.

Create and update are trickier. Canonical description of HTTP verbs can be found in Section 9 of RFC 2616 and I use the table here as an addendum. In short, for both create and update, if an operation is idempotent and URI of entity on which this operation is being performed is known, use PUT. Otherwise, use POST (it is often used on entities representing “factories” – say a factory of new postings; you don’t know URI of a posting before you create it, so you POST to a factory which will create a new entity at a new URI; note that POST is not idempotent).

Note the RFC definition of idempotent methods (9.1.2) – it’s not defined as “multiple invocations must lead to the same result as a single invocation.” It’s “(aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request.”

HTTP return codes

Section 10 of RFC 2616 is a canonical description of HTTP status codes.

Successful completion should be signaled as HTTP 200 OK and, if it’s important for client to know that an entity was created as a part of operation, HTTP 201 Created. The latter may be redundant – code that handles 200 and 201 most likely will be identical or very similar.

Speaking of errors, I don’t think it’s practical to map each type of error to its own HTTP error code. Unexpected server side errors (frontend exceptions or uncaught exceptions raised by your API methods) could be HTTP 500 Internal Server Error. If a resource is not found, it should be HTTP 404 Not Found. If your API server uses an external service to perform certain operations and upstream service did not respond or returned an unknown error, I would signal this fact with HTTP 502 Bad Gateway.

The rest of the errors are all client-side, and I like to classify them into 2 categories. When something is wrong with submitted request (missing header, missing argument, argument of wrong type), I think server should return HTTP 400 Bad Request. This way server is telling the client that no matter how many times this request will be submitted, it won’t work and will produce identical response.

I then group all other client-side errors together and think they should lead to HTTP 403 Forbidden. It means request by itself is fine, but something is preventing server from executing it – such as a missing prerequisite. Re-submitting the request may work in this case, because by the time the request is re-submitted, something might have happened and prerequisite is already in place.

Error response could include application-level exception and its description – this way you are letting the client know exactly what was wrong. Whether processing these ends up automated or not – it’s up to the client.

Message formats

I can’t easily justify this one, but I feel that bodies of request and response should be in the same format (there could be exceptions – for example, when client must upload a binary artifact). vCloud does it this way – request body is XML, and response is XML. EC2 API sends request arguments in query string (because all requests are GET, since it’s query API) and response is XML. OCCI API defines request body as form-urlencoded (application/x-www-form-urlencoded) and response is XML as well (all of the above might support JSON as well).

I have 2 weak justifications for this.

Firstly, it somewhat mimics our regular human behavior. If 2 people are communicating in real time, they usually use same medium and same format. It’s rare when one person is on IM speaking English, while the other is on the phone speaking French – not saying it’s impossible but relatively rare.

Secondly, in the future I foresee a greater use of messaging in API operations (read this post by George Reese). Notions of request/response come from HTTP, in messaging it doesn’t matter – the same message could be response to one message and request to another. For example, a message requesting server start may lead to a message saying “server started” to the client. At the same time, the same “server started” message may go to an internal billing system, where it would be a request to start billing.

Having these message in the same format might be beneficial.

Command line tool

AWS set the bar with EC2 here. For every API call, they ship a command line tool to perform said call. No matter what you think whether it’s right or wrong, I think every provider should match this behavior. It’s a good practice after all – when someone is about to try API, it’s much easier to get going using command line tools instead of embedding API calls straight into the application.

Instead of EC2 practice of one command line tool per API call however (even though inside they still call ec2-cmd), I favor Sun Cloud’s approach – they were planning a single unified tool where an API call would be identified by an option or a subcommand.

Conclusion

As the Zen of Python goes, “[...] practicality beats purity.” This should be your main guiding principle when designing API server side.

As a user who stands to benefit from increased number of services allowing third-party applications and mashups, I certainly tend to agree. But as a developer, I realize that prematurely making API public may be a disaster.

Publication of API represents a long-term commitment. You as a developer are committing to supporting this API for some non-trivial amount of time (at least 12 months I would imagine) and are essentially inviting other developers to create new functionality against this API. No one likes to spend their time developing against a given API just to discover shortly that API changed, or some functionality that used to be offered is no longer available.

By making your API public you are signaling that this part of your system is very stable, its functionality well established, understood and developed, usage patterns well thought out. Or at least that’s how I as a third-party developer interpret your action.

If you know your audience well enough and are pretty confident that they won’t mind your tweaking things after initial publication, you may take a risk. Twitter famously launched their API very very early, and in the end it proved a huge success for them. (So if they listened to my advice in this post, they would be worse off).

But not all developer audiences may be as agile and forgiving as Twitter’s. I can imagine a very conservative big user of your API that will very strongly object to your changing the API. What do you do next? Maintain 2 versions? But what if underlying database schema changes make old API incompatible with what you are trying to do in the future? Fork and host 2 different systems, old and new? I can’t honestly imagine a worse scenario.

My advice – before publishing your new API, make sure you are not going to force yourself into a corner down the road. Only publish API for those parts of your systems that are very stable (both operationally and from perspective of internal mechanics and functionality) and where usage patterns are well researched and predictable to a certain extent. Don’t rush it.

But here is a kicker – it’s not the end. I saw this tweet, read this post and checked out a book by Charles Perrow titled “Normal Accidents” from the library. Published in 1984, the book is not about IT, but its material fits our field nicely. And boy, was I enlightened!

The book’s main point: no matter how much thought is put into the system design, or how many safeguards are implemented, a sufficiently complex system sooner or later will experience a significant breakdown that was impossible to foresee beforehand, principally due to unexpected interaction between components, tight coupling or bizarre coincidence. For us in IT, it translates to “no matter how much planning you do or how many safeguards you implement, failures will still happen.”

There are at least 3 common themes that are present in multiple illustrations in the book:

1. A big failure was usually a result of multiple smaller failures; these smaller failures were often not even related
2. Operators (people or systems) were frequently misled by inaccurate monitoring data
3. In a lot of cases, human operators were used to a given set of circumstances, and their thinking and analysis were misled by their habits and expectations (”when X happens, we always do Y and it comes back” – except for this one time, when it didn’t)

I have had my share of outages and downtimes, and I can attest that I have seen these 3 factors play a big role in tech ops. Some were bugs in management and monitoring code, some where human error, some where bizarre set of dependencies but all were a combination of multiple factors. For example, who would have thought that with a failure of primary DNS resolution server, the VIP would not fail over to the secondary; and even though hosts had more than one “nameserver” line in /etc/resolv.conf, application timed out waiting for DNS to respond before getting to ask the second nameserver; without name resolution, multiple load balancers independently thought that there was no capacity behind them (because management code calculated capacity in near real-time relying on worker hosts’ names) and disabled themselves, thus taking down the entire farm – now I know of course…

It turns out we can’t eliminate normal accidents altogether, but here are several techniques that I have been using to speed up detection and response in order to reduce the downtime.

Complexity budget. Described by Benjamin Black, this is a technique to allocate complexity among components beforehand and strictly follow the allocation during implementation phase. It helps avoid unnecessary fanciness and leads to simpler code, which tends to be easier to troubleshoot and recover after a failure.

Control knobs/switches for individual components. As John Allspaw shows on this slide, you need to be able to turn off any component in an emergency, or throttle it up or down. Planning this feature and building it in from the very beginning is very important.

Accuracy of monitoring data. Ensure your alarms are as accurate as possible. No matter how much chaos is going on inside the system during a severe failure, last thing you can afford is misleading the operators with wrong information. If you tried to ping a host A and didn’t get a response, your alarm should not say “host A is down” because it’s not the knowledge you obtained – it’s an assumption that you made. It should say “failed to ping host A from host B” – maybe it was network on host B that was an issue when a ping attempt was made, how do you know?

Availability of monitoring data. There is a reason first thing the military try to do when attacking, is disrupting enemy’s means of communication – it’s that important, which applies to our case as well. You either design your systems to be able to get monitoring data even during the worst outage imaginable (ideally from more than one source), or you at least should be getting an alarm about lack of such monitoring data (it’s a very weak substitute though).

All in all, to everybody in IT, I highly recommend the Normal Accidents book as well as this whitepaper (linked from John Allspaw’s blog).

This outage shed more light on some internal designs of EC2 itself, as described here. It might have also showcased our over-confidence in EC2’s ability to detect and defeat certain types of network attacks. But this post is about something else.

BitBucket was running their web front door and their backend application on the same instance. Front door is a part of the system which is facing the Internet and its task is to accept connections from clients. For obvious reasons, front door is running on the service’s discoverable IP address – whether they used Elastic IP or not, bitbucket.org resolved to that IP. Note that front door (usually) doesn’t need EBS.

Backend, however, is what needs EBS for disk persistence. At the same time, backend does not need to be publicly discoverable – as long as front door knows where its backend worker(s) is/are running, the app should be functioning just fine.

With front door and backend running on different instances, UDP flood would have saturated only the former’s network interface and would have had no impact on the backend and its EBS.

I know that AWS reportedly fixed the flood issue, but looks to me like separating front door and application backend may still be a good preventive measure – after all, it’s considered a good practice for a reason.

Please note that I am not trying to accuse BitBucket of running a bad architecture and causing their own outage. All I am doing is trying to learn a lesson.

Distributed applications are known for complexity of assuring all components are on the same page as to what’s going on around them. Hardware failures, network failures, operator errors can all cause chaos; distributed applications foresee these exception situations and attempt to know how to deal with them.

Up until now, the network piece of the puzzle has been usually under application owner’s control – it could be a LAN, or it could be a leased line to remote datacenter. Occasionally, a VPN would be used to provide a dedicated communication channel between locations over public Internet but its use was rarely focused on important stuff – a mission critical application would usually get a leased line.

With advance of public clouds such as Amazon EC2 and Google AppEngine however, these notions are changing. One day you may decide to leverage each cloud’s strengths and distinct features to build your app, or may want to avoid cloud lock-in or provide redundancy. In short, you may want to multi-source your infrastructure.

Your multi-sourced infrastructure will of course be a distributed application. But there is a significant difference between this and old-style distributed apps – this time you no longer have network connectvity under your control. And as a result, you will face 3 significant phenomena that substantially complicate using today’s distributed algorithms – uneven bandwidth, uneven latency and increased probability of connectivity loss (I blogged about the latter here).

And this is what I call a hyper distributed application. In other words, hyper distribution application is a distributed app which runs on a network with uneven bandwidth, uneven latencies and increased probability of connectivity loss (as measured against that on a regular LAN), usually outside of application owner’s control (for example, Internet).

One example of a hyper distributed application is VPN-Cubed that we at CohesiveFT created to address emerging needs to multisource infrastructure. By the very nature of functionality it provides, its components (we call them VPN-Cubed Managers – they act as virtual routers and switches) are sometimes distributed over LAN, sometimes distributed over WAN, sometimes both. Communications between manager 1 and manager 2 can be fast and reliable, but between manager 1 and 3 slow and less reliable, with more frequent resets. Or manager 3 may simply disappear (as seen by its peers) – no, it doesn’t have to be down due to crash; it can simply mean that its network connection to the outside world was down, possibly temporarily.

Hyper distributed applications are relatively rare, because most architects tend to avoid this if they can. For example, Amazon EC2 has 2 regions – US and EU. Each region is a distinct EC2 system, with its own API endpoint, its own AMI IDs, kernel IDs, security groups, keypairs. There is no replication or conflict resolution between the regions – they are totally independent of each other. Why? Because it would be quite difficult to interconnect them into a single entity over public Internet. (I won’t be surprised if it gets implemented in the future though.)

Another example showing that hyper distributed applications are a distinct breed comes from Facebook Engineering blog post titled Scaling Out:

This setup works really well with only one set of databases because we only delete the value from memcache after the database has confirmed the write of the new value. That way we are guaranteed the next read will get the updated value from the database and put it in to memcache. With a slave database on the east coast, however, the situation got a little tricky.

When we update a west coast master database with some new data there is a replication lag before the new value is properly reflected in the east coast slave database. Normally this replication lag is under a second but in periods of high load it can spike up to 20 seconds.

It nicely illustrates how hyper distributed nature of the application adds complexity on top of what a plain distributed app already has.

In conclusion, I would like to propose to separate a category of distributed applications that run on top of networks with uneven bandwidth and uneven latencies into their own (I don’t care much if they end up being called hyper distributed or something else), and start building up research and practical approaches focusing specifically on this area.

P.S. Also consider the future: when we reach inter-planet or inter-galactic communications, you know that latencies and bandwidth in space would not be (initially) the same as on our planet Earth. Better start working on this research now in order to be prepared…

But there are also differences. Every child knows that electric shock can cause injury even as a result of a short exposure – hence most perceive electricity as a powerful force. This force however has a binary switch attached to it, in the form of switches, circuit breakers and distribution board. Turn it on – electricity is flowing, turn it off – it’s not. When off, electricity can’t leak by design.

Water, on the other hand, is not perceived as such a great force because damage from short exposure is unlikely to be too severe. Additionally, indoor plumbing has no binary on-off switches – it’s measured by a degree of “open” or “close”, “hot” or “cold.” As a result, leaks can and do occur from time to time. And it’s these leaks that have a potential to do costly damage over time but still are not perceived dangerous enough to warrant immediate attention.

There are many things in software applications that are binary in nature – web server daemon is up or down, for example. We all take these all-or-nothing components seriously, because when it’s nothing, the app is down.

But we have our fair share of potentially leaky stuff as well – memory leaks, file descriptor leaks, network connection leaks, and so on. In other words, things that don’t happen instantaneously but build up over time, often hidden behind other bigger component. Some of us don’t take these issues seriously enough because they lack the perceived power of being able to cause significant damage quickly enough. And it’s a mistake.

When monitoring a component of “electricity” type, most common test is to send a probe – if it returns OK, the component is up (”active monitoring”, “active polling” or simply “polling”). But this doesn’t work when monitoring a component of “plumbing” type – if water is flowing, it doesn’t mean there is no leak. In this case, a set of alarms instrumented into the component itself would be a better fit.

The sooner we realize different nature of various components of our applications and the need to monitor them differently, the higher uptime for our applications we are going to achieve.

The term “cloud computing” consists of 2 words – “cloud” and “computing.”

Cloud

Traditionally, an image of cloud is used on network diagrams to denote an opaque network entity (for example, Internet or MPLS cloud). Opaque in this case means that to an enduser it’s a black box – you hook up inputs and outputs as directed, and you get functionality. In addition to opaqueness, there are other less obvious properties that clouds on network diagrams usually possess:

• cloud is multi-tenant (many endusers use same one)
• cloud resources (links, bandwidth) are not dedicated (each enduser gets to use some up to their quota; if user A no longer uses a resource, cloud can assign it to user B)
• cloud is outside of enduser’s full control

Computing

Firstly, allow me to note that I strongly disagree with pure linguistic approach here – to linguists, “computing” and “computer” are derived from the same root, such that “computing” is an action which involves a “computer.” I disagree with it because it’s too general and useless for our case.

I define “computing” as running user-provided software. It doesn’t have to be developed by user – one can download it from the web and run it. But it’s still the user who provides this software in this particular case. In contrast, if you use a web site to perform a certain operation, you also use software – but in this case, it’s the software developed and operated by the web site, hence it’s a service, not computing.

My Definition of Cloud Computing

Cloud computing is a form of using opaque multi-tenant networks of computers outside of enduser’s full control with primary goal to run software provided by the enduser, in which computational resources are allocated dynamically (as opposed to being permanently assigned).

Examples and Caveats

• If we take a well-known SPI model (Software as a Service, Platform as a Service, Infrastructure as a Service), contrary to current mainstream thinking, only IaaS can be cloud computing when enduser provides the software to run.

• I added a clause about “primary goal” to eliminate things like Google Spreadsheet from cloud computing – even though a spreadsheet program may run macros (which are software code) and such macros could be provided by enduser, it’s still not cloud computing, because the primary goal of a spreadsheet program is number crunching, not running macros.

• Programming frameworks (such as Hadoop for example) can be both: Hadoop can be cloud computing when enduser provides their map and reduce functions; but if enduser ends up running defaults or functions that ship with Hadoop distribution, there is no software supplied by enduser so it’s not cloud computing.

• Things like storage as a service, backup as a service are all “cloudy,” but they are not computing. There is already a term for this – Internet. Therefore, I consider “cloudy” by itself to be a redundant term.

• Google AppEngine (GAE) is a cloud computing platform. Many don’t put it into IaaS category because it doesn’t provide customers with access to low-level hypervisor-based VMs. But this alone doesn’t make it non-IaaS from developer’s standpoint – after all, a VM in hypervisor model is one thing, and a VM in language interpreter model is another (JVM, Erlang VM, Python VM, etc) but it’s still a VM in a sense that it encapsulates running code inside and proxies all system-level requests through its abstraction layer. GAE provides access to its BigTable infrastructure, its memcache infrastructure so to me it’s very much an IaaS  system and satisfies my definition of “cloud computing.”

• In my opinion, multi-tenancy is a necessary condition of a cloud computing platform. Multiple tenants must not be different companies – they can be different business units, different departments. The key is that there must be dynamic allocation of resources and scarcity. If all resources are dedicated to one organization and simply switched between applications, it’s not cloud computing – it would be simply an infrastructure controlled via API.

• Same thing about on-premises server farms with cloudy features – they are not cloud computing, because they are not opaque to enduser and they are under enduser’s full control.

Conclusion

All in all, I hope this blog post gets us closer to finally figuring out once and for all what “cloud computing” is and what it isn’t.

Do you remember an old Unix command to create tape backups called dump? Remember its concept of levels? To refresh your memory, in a nutshell level 0 (full backup) includes all files on the filesystem, and any other level corresponds to incremental backup where only files modified since last backup are included.

It turns out somewhat similar concept applies to messaging, specifically to the contents of messages themselves.

A message in general is some piece of information that one system passes to another. On one hand, publisher may make an observation, extract information from it, package entire current state into a blob, and send it out as a message. The same sequence of operations is performed at regular intervals. Examples of this model include sending a message about processes currently running on the system, clients currently connected to a server, current usage of RAM, etc. This model roughly corresponds to dump’s level 0 – consumer needs just a single message to obtain all information that publisher sent, there is no need for consumer to accumulate and merge a series of messages to get the full picture.

On the other hand, a publisher can send a message that contains information about a single event. For example, a new client connected, a new job got submitted to the backend, hard disk failed. This mode is more like incremental backup – a message contains only a delta, its payload doesn’t carry entire state.

Each of these models has its good and bad sides. In full data model, a single message is sufficient to transfer all knowledge about current state from producer to consumer, and consumer can start reading messages at any point in the queue – by design it will catch up once it receives and processes at least one message. The downsides of this model are waste of bandwidth and processing power (if there are no changes, same contents will be transferred over and over again) and the fact that delta must be calculated by consumer (for example, having received 2 “ps auxww” outputs, consumer would have to diff them and parse the result).

Incremental data model clearly provides an easy delta and is less wasteful on resources, but requires consumer to merge multiple messages to get entire picture and as a result is sensitive to a point from which a consumer starts reading the queue.

A potential solution is to do what dump does – send full data once in a while, followed by deltas. This way consumer will catch up eventually – once it gets full data message (which will come sooner or later). Another caveat is that not always does a consumer need a full picture – in a classic scalability scenario of supervisor-workers model, workers rarely need more than contents of their current job contained in an incremental message.

But it’s not the end of it. While working on a problem, I realized that usually I as a developer don’t even get to choose which model to use – it’s dictated to me by the nature of information I am trying to pass from one system to another. Some data can be easily obtained as full and very difficult to obtain as incremental, some vice versa. For example, a list of current processes on Linux is trivial to obtain as full (ps auxww) and quite difficult to obtain as incremental (I would need a notification about when each process starts and dies). Or in case of incoming jobs – it’s easy to obtain delta (one job) but it’s quite difficult to know current status of all jobs.

My conclusion here is that there are 2 main factors to think about:

1. can my publisher get data in full or incremental form?
2. does my consumer need data in full or incremental form?

If the answers to above questions are the same, you are good to go. But if they are different, you need to understand potential issues as discussed above and analyze further. I hope to be able to provide more practical thoughts on this in the future – stay tuned.

And here are my reasons.

Memory Management

Shell scripts are excellent in managing their memory and one has to try real hard to cause a shell script to leak memory. This makes shell a very convenient tool for long running processes, supervisors in multiple-workers models, daemons and so on. There is an easy explanation for this. In shell, there are only a handful of built-in primitives – everything else is an external command, which gets started and then finishes before giving control back to your script. If there is a memory leak in that command, it won’t damage your calling script and will usually be insignificant because it will return quickly.

No Exceptions

This is a double edged sword, and you need to be careful how you exploit this “weakness.” This feature allows me to write compact code which is easy to understand without enclosing every single command in “try… except”. For naysayers, I would like to point out that a strict mode exists, where every error is treated as fatal and causes the script to exit (set -e).

In general, not all unforeseen error conditions warrant a crash, like you get in Python or Ruby when an unhandled exception gets propagated all the way to the top. If a problem is transient, it may be better to ignore it temporarily.

To assure a Ruby or Python script doesn’t crash on some unforeseen transient problem, many people often end up enclosing their entire program in a wildcard try… except block to catch any exception – but to me this approach is dangerous, even though I sometimes end up using it myself.

If you are writing a daemon process to perform some action in a loop, shell is often by far the most stable alternative.

When Not To Use Shell

My personal rule of thumb is don’t use shell when you expect to need high-level data structures like hashes or arrays beyond what for loop can give you, or when you can see potential for code reuse following OOP patterns like inheritance, or when your program needs to participate in some orchestration schemes that go beyond creating and removing files on the filesystem.

Conclusion

I wouldn’t overlook shell if I were you.

When starting to work on a new big feature, always set up 2 branches for it. Say FEATURE_work and FEATURE_integration. Do your regular development in FEATURE_work committing as often as you want. When you reach certain milestones (but entire feature is still not ready yet), squash merge FEATURE_work into FEATURE_integration. When entire feature is finished, merge FEATURE_integration into master.

This gives you a much nicer history of commits, lets you group changes by milestone, and allows to keep big feature as multiple commits in master.