Some background first. Each Erlang program in general consists of multiple Erlang processes that send and receive messages to/from each other. These are not your regular processes – these processes are running inside the Erlang VM, and are not mapped to either processes or threads on the system level in any way. They are cheap to create, and communications between them are fast. Each process has a process ID (pid) associated with it. If multiple Erlang VMs share a cookie (a piece of text which is used as a security token), processes on any Erlang VM can freely talk to each other without any code modification.

RabbitMQ implements each queue as an Erlang process. You can think of AMQP exchanges as pieces of routing logic – when a message arrives, Rabbit applies the routing rules to its routing key and sends the message to all eligible queues. Since each queue is a process and has a pid, in a nutshell exchanges send the message to several pids.

In non-clustered mode, Rabbit routing function will return a list of local PIDs by selecting values stored in a mnesia table (rabbit_durable_queue and rabbit_queue). But since RabbitMQ can also run in clustered mode, developers already implemented a way to send messages to queues on remote nodes. So when I hacked rabbit_durable_queue and rabbit_queue tables and replaced PIDs pointing to local queues with PIDs pointing to remote queues, I got myself a remote queue forwarder.

How is this useful you might ask. RabbitMQ supports remote queues only in clustered mode. The way clustering is implemented today (using mnesia in distributed mode), it’s not recommended to run a rabbit cluster over non-LAN links. This is because if 2 rabbit nodes lose and then regain connectivity to each other, the cluster may enter a “partitioned network” state, which effectively means that rabbit cluster is not functional (in other words, mnesia sacrifices tolerance to partitioning in order to achieve consistency and availability – recall CAP theorem). With remote queue forwarding, you don’t need to set up clustering and hence “partitioned network” state won’t affect you by design – and that’s what mattered to me.

On the other hand, remote queue forwarding can potentially break some AMQP guarantees. For example, if a remote node is temporarily unavailable, Rabbit won’t queue the message for later re-delivery (because such situation is currently impossible in unhacked Rabbit). It means that YOU SHOULD NOT USE THIS HACK UNLESS YOU KNOW WHAT YOU ARE DOING.

The hack works for me though in the following scenario. I publish messages with immediate=true (indicates that payload is time sensitive). Messages are sent to N local queues. These N local queues are forwarded to N remote nodes, with a consumer attached to 127.0.0.1 on each node. In this project, I don’t rely on any AMQP guarantees – I discard basic.return commands and can tolerate occasional message not reaching some or even all consumers.

I doubt anyone will find this hack in its current form useful, but just in case I uploaded it to http://github.com/somic/rabbit_queue_forwarder.

PS. After I started this small project, Tony Garnock-Jones announced that his work on pluggable exchange types has been added to default branch. I still went ahead and published my post, but please note that pluggable exchange types could probably achieve similar effect more cleanly.

When building erlang R13B02-1 on a 64bit non-SMP machine (not sure if it matters), “make -j 2″ somehow resulted in an error which I could not work around. Reverting to simply make (without -j 2) and starting compilation from the very beginning fixed it.

Also, after final make install, I could not start erl – it was complaining about “start.boot not found”. The solution is to symlink boot files like this:


cd /usr/lib/erlang/bin
ln -s /usr/lib/erlang/releases/R13B02/start.boot .
ln -s /usr/lib/erlang/releases/R13B02/start_clean.boot .
ln -s /usr/lib/erlang/releases/R13B02/start_sasl.boot .


I configured it with “./configure –prefix=/usr –disable-x –enable-threads –enable-kernel-poll –disable-hipe”.

Do you remember an old Unix command to create tape backups called dump? Remember its concept of levels? To refresh your memory, in a nutshell level 0 (full backup) includes all files on the filesystem, and any other level corresponds to incremental backup where only files modified since last backup are included.

It turns out somewhat similar concept applies to messaging, specifically to the contents of messages themselves.

A message in general is some piece of information that one system passes to another. On one hand, publisher may make an observation, extract information from it, package entire current state into a blob, and send it out as a message. The same sequence of operations is performed at regular intervals. Examples of this model include sending a message about processes currently running on the system, clients currently connected to a server, current usage of RAM, etc. This model roughly corresponds to dump’s level 0 – consumer needs just a single message to obtain all information that publisher sent, there is no need for consumer to accumulate and merge a series of messages to get the full picture.

On the other hand, a publisher can send a message that contains information about a single event. For example, a new client connected, a new job got submitted to the backend, hard disk failed. This mode is more like incremental backup – a message contains only a delta, its payload doesn’t carry entire state.

Each of these models has its good and bad sides. In full data model, a single message is sufficient to transfer all knowledge about current state from producer to consumer, and consumer can start reading messages at any point in the queue – by design it will catch up once it receives and processes at least one message. The downsides of this model are waste of bandwidth and processing power (if there are no changes, same contents will be transferred over and over again) and the fact that delta must be calculated by consumer (for example, having received 2 “ps auxww” outputs, consumer would have to diff them and parse the result).

Incremental data model clearly provides an easy delta and is less wasteful on resources, but requires consumer to merge multiple messages to get entire picture and as a result is sensitive to a point from which a consumer starts reading the queue.

A potential solution is to do what dump does – send full data once in a while, followed by deltas. This way consumer will catch up eventually – once it gets full data message (which will come sooner or later). Another caveat is that not always does a consumer need a full picture – in a classic scalability scenario of supervisor-workers model, workers rarely need more than contents of their current job contained in an incremental message.

But it’s not the end of it. While working on a problem, I realized that usually I as a developer don’t even get to choose which model to use – it’s dictated to me by the nature of information I am trying to pass from one system to another. Some data can be easily obtained as full and very difficult to obtain as incremental, some vice versa. For example, a list of current processes on Linux is trivial to obtain as full (ps auxww) and quite difficult to obtain as incremental (I would need a notification about when each process starts and dies). Or in case of incoming jobs – it’s easy to obtain delta (one job) but it’s quite difficult to know current status of all jobs.

My conclusion here is that there are 2 main factors to think about:

1. can my publisher get data in full or incremental form?
2. does my consumer need data in full or incremental form?

If the answers to above questions are the same, you are good to go. But if they are different, you need to understand potential issues as discussed above and analyze further. I hope to be able to provide more practical thoughts on this in the future – stay tuned.

Graphite CLI Screenshot

Graphite is a flexible and powerful tool to build charts. It’s also a data series analytics framework. It was developed inside Orbitz by my former colleague, originally for use within a single group (of which I was a part). However, its power did not remain a secret for too long – it quickly spread to entire organization and became an irreplaceable tool for both development and engineering/operations. Graphite was then open-sourced under Apache license. It currently lives at http://graphite.wikidot.com/

The key to Graphite’s power, in addition to dynamic web UI, an improved RRD implementation called “whisper” (read this FAQ – highly recommend!) web-based command line with auto-completion which allows you to overlay any metrics on a single chart, IMHO is the fact that you are in control what kind of data to send to it, how often, and how to set up hierarchies of your metrics – by environment, by machine type, by datacenter, etc. Graphite doesn’t do its own polling that won’t scale to hundreds or thousands of metrics. Nor does it enforce anything but the fact that your metrics are dot-separated hierarchies (as in routing keys of AMQP topic exchanges – my.metric.name) and that their values are numeric (int or float).

If you are still reading this but still are not convinced that it’s the way to go, I’ve got one last argument. If you already use RabbitMQ to publish and consume data, wouldn’t it be nice to get a powerful charts without touching your application AND without installing agents on your publishers or consumers? Recall the duplication pattern of RabbitMQ – you can fork the incoming stream of messages into another queue (without impacting your original consumers and the queues to which they attach) and set up Graphite+RabbitMQ off of this new queue.

If you are planning to run multiple carbon instances, remember that heavy lifting (writing to disk) is actually performed by another process called carbon-persister.py (it’s started by carbon-agent, with communications over a pipe) – try to avoid multiple persisters writing data within the same hierarchy to avoid slow down and possible data corruption. RabbitMQ can help you sort out what messages go where, thus minimizing this risk.

I am very excited about future opportunities that a Graphite-RabbitMQ combination can deliver, and I hope someone finds my scripts useful. Both tools bring a lot of awesomeness to the table, and nicely complement each other forming a great charts and data series analytics solution you have been searching for. Check it out!

The biggest enhancement (IMHO) is ability to increase or decrease the number of backend consumers based on any HTTP request headers. In “table” routing mode (see rabbitbal.yml), you can now specify array of tests against which incoming request headers will be matched. This will cause your request to be published with a matching key (note :key). Your backend consumers use the same YAML file and can bind to all or only some queues, thus giving you flexibility in adjusting the capacity. Old functionality is available by using “topic” routing mode.

Note that I still use topic-based exchange, because I wanted to support a use case where you want to aggregate all incoming requests into separate queues (routing key would be something like “request.#”) for bot detection, access log aggregation, etc. In other words, each request ultimately must end up in a single queue where it will be picked up by backend servers, while at the same time it can also be duplicated into other queues for other purposes.

Are you running broker as user rabbitmq and rabbitmqctl as root or rabbitmq? If not, please stop and fix this. This is not a requirement per se, but this represents canonical installation of rabbitmq broker. You can certainly hack your scripts to work around this requirement, but you are on your own if you do.

Double check that broker is in fact running. Use ps, netstat -lptn (look for port 5672 unless you overrode it in /etc/default/rabbitmq). Telnet to localhost on port 5672, type AMQP and press ENTER several times. You should get response that at least will show AMQ. Check logs at /var/log/rabbitmq to verify that broker saw your connection attempt.

Next step is to start “erl -sname foo -cookie coo” in shell and run this command: “net_adm:names().”

If this command returns ok followed by a list of nodes within 1 or 2 seconds, check if rabbit is there. If it is, it’s very likely that you have users mixed up above. Please double check. If rabbit node is not listed, double check that rabbit broker is still running.

If this command returns {error,address}, there is a problem with your instance of EPMD, erlang naming daemon (man net_adm). First, check if it’s running (it most likely *is* running). Then, in erlang, run “net_adm:localhost().” Exit erlang, and try to connect to exactly the name you got from net_adm:localhost() on port 4369 (epmd). This shouldn’t work and should timeout. If it doesn’t time out, you shouldn’t have gotten {error,address}.

The problem most likely will be that name as returned by net_adm:localhost() is associated with some IP in /etc/hosts or in DNS which is either not accessible from this host, or firewalled off. An entry in /etc/hosts that associates this name with 127.0.0.1 or one of other IPs on this server should fix the problem.

Alternatively, net_adm:names() may time out with {error, timeout}. We have seen it caused by snoopy in the past. Remove snoopy or do not install it system wide in /etc/ld.so.preload, and you should be fine.

If these steps did not help, please leave a comment below or join rabbitmq-discuss and ask your question there, and we’ll help!

UPDATE 2009-04-17: Your host’s name as shown by net_adm:localhost() does not technically need to be defined in /etc/hosts. But if it’s not there, it should not be defined anywhere – when you do “ping name”, you should get “ping: unknown host name”. I have seen at least one case when it worked this way. This is somewhat unverified though.

Here are benefits of AMQP-based approach over traditional HTTP-based reverse proxies taken from rabbitbal README file (in no particular order) as I see them.

1. Model where workers fetch work as fast as they can (each going at its own pace) in theory should provide more efficient load balancing than a model where proxy assigns work based on certain criteria; methods based on round robin, arbitrary weights or least connections become simply unnecessary.
2. RabbitMQ broker implements intelligent failover out of the box – if a web server disconnects before ack’ing, the request will be automagically requeued for another server; all in all, RabbitMQ is way smarter than a bunch of low level TCP connections.
3. Enhanced security of actual web servers – servers behind Rabbitbal do not need inbound connectivity, they only need to be able to establish an outgoing connection to RabbitMQ broker(s).
4. Rabbitbal does not need to know IPs or have direct connectivity into its web servers (use case: Amazon EC2 without Elastic IPs)
5. Using Duplication pattern of RabbitMQ (see Resources below), you could be reading requests and responses off of the same broker in real time (access log aggregation, double-entry book keeping, logging, bot detection)
6. You could relatively easily have one request go to more than 1 web server
7. Add capacity as often and as much as you like – rabbitbal won’t even know
8. By slightly readjusting mapping between queues and URIs, you could add or remove capacity per URI if needed
9. TCP overhead savings compared with HTTP proxies (AMQP uses persistent TCP connections)

I am a big fan of RabbitMQ, an implementation of Advanced Message Queueing Protocol. In this post I am going to provide an overview how RabbitMQ can be used beyond simple queueing and pubsub. For more background on this topic, please see a list of messaging scenarios that RabbitMQ supports.

Queueing

Broker can take a message from producer and keep it until a consumer shows up. To survive broker restarts, queues in this case should be durable, with auto-delete set to false, and messages should be published with delivery mode of 2 (which means persistent). This pattern can also be helpful if consumer is temporarily unable to keep up with incoming message flow -  queueing allows producer and consumer to keep going at their own pace, and will make sure all messages are consumed eventually.

Multiplex

This is when multiple producers publish messages to be routed to a single consumer. You can either have all your producers publish to the same fanout exchange, or to the same direct exchange with the same routing key, or to a topic exchange with a routing key that matches that of consumer. The latter case allows producers to publish “flights.aa.ord” and “flights.ua.sfo”, while the consumer can be reading all of these with “flights.#” (* matches a single word, # matches zero or more words).

Demultiplex

This is when a single producer publishes messages that are routed to multiple different consumers. It can be a topic exchange, producer could publish “order.book” and “order.cd”, while orders for books and CDs are handled by different systems.

Instant Feedback (Queueing Bypass)

Producer can instruct broker not to queue a message at all and return it to sender if a consumer is not currently available to read it (this is achieved by setting immediate flag to true in basic.publish method). Can be helpful in scenarios where message content is time sensitive – it needs to be processed now or an error must be returned.

Duplicating

In AMQP, a message is delivered to all queues bound to a given exchange, if a queue meets routing criteria (for different types of exchanges, these criteria are different). For example, a message published with “prod.server01.disk.full” key can be simultaneously routed to “prod.#” queue (for production logger to keep track of all events in production environment) and “#.disk.full” queue (for an archiver process that removes old logs). Very powerful feature, and it works with direct and fanout exchanges as well.

Load balancing

If multiple consumers read from the same queue, RabbitMQ broker will automatically load balance messages between all available consumers. Each message will be sent to one consumer at a time.

Failover

In no_ack=false mode, a consumer must eventually explicitly acknowledge receipt of each message, individually or as a group (this does not mean that a message must be ack’ed before next one can be received). If a consumer disconnects without acknowledging, unack’ed messages are automatically re-queued for another consumer. This helps achieve consumer failover in response to crashes or loss of network connectivity.

Relaying

If producers and consumers do not have direct line of sight network-wise (for example, they are behind NAT or are located on private subnets), RabbitMQ can provide the connectivity by serving as a message relayer. Both producers and consumers must be able to establish client connections to broker (AMQP official port is TCP 5672) and then they can exchange messages.

Some of these patterns can be mixed and matched, which further expands a set of problems where RabbitMQ can help you achieve a distributed messaging nirvana.