Some background first. Each Erlang program in general consists of multiple Erlang processes that send and receive messages to/from each other. These are not your regular processes – these processes are running inside the Erlang VM, and are not mapped to either processes or threads on the system level in any way. They are cheap to create, and communications between them are fast. Each process has a process ID (pid) associated with it. If multiple Erlang VMs share a cookie (a piece of text which is used as a security token), processes on any Erlang VM can freely talk to each other without any code modification.

RabbitMQ implements each queue as an Erlang process. You can think of AMQP exchanges as pieces of routing logic – when a message arrives, Rabbit applies the routing rules to its routing key and sends the message to all eligible queues. Since each queue is a process and has a pid, in a nutshell exchanges send the message to several pids.

In non-clustered mode, Rabbit routing function will return a list of local PIDs by selecting values stored in a mnesia table (rabbit_durable_queue and rabbit_queue). But since RabbitMQ can also run in clustered mode, developers already implemented a way to send messages to queues on remote nodes. So when I hacked rabbit_durable_queue and rabbit_queue tables and replaced PIDs pointing to local queues with PIDs pointing to remote queues, I got myself a remote queue forwarder.

How is this useful you might ask. RabbitMQ supports remote queues only in clustered mode. The way clustering is implemented today (using mnesia in distributed mode), it’s not recommended to run a rabbit cluster over non-LAN links. This is because if 2 rabbit nodes lose and then regain connectivity to each other, the cluster may enter a “partitioned network” state, which effectively means that rabbit cluster is not functional (in other words, mnesia sacrifices tolerance to partitioning in order to achieve consistency and availability – recall CAP theorem). With remote queue forwarding, you don’t need to set up clustering and hence “partitioned network” state won’t affect you by design – and that’s what mattered to me.

On the other hand, remote queue forwarding can potentially break some AMQP guarantees. For example, if a remote node is temporarily unavailable, Rabbit won’t queue the message for later re-delivery (because such situation is currently impossible in unhacked Rabbit). It means that YOU SHOULD NOT USE THIS HACK UNLESS YOU KNOW WHAT YOU ARE DOING.

The hack works for me though in the following scenario. I publish messages with immediate=true (indicates that payload is time sensitive). Messages are sent to N local queues. These N local queues are forwarded to N remote nodes, with a consumer attached to 127.0.0.1 on each node. In this project, I don’t rely on any AMQP guarantees – I discard basic.return commands and can tolerate occasional message not reaching some or even all consumers.

I doubt anyone will find this hack in its current form useful, but just in case I uploaded it to http://github.com/somic/rabbit_queue_forwarder.

PS. After I started this small project, Tony Garnock-Jones announced that his work on pluggable exchange types has been added to default branch. I still went ahead and published my post, but please note that pluggable exchange types could probably achieve similar effect more cleanly.

When building erlang R13B02-1 on a 64bit non-SMP machine (not sure if it matters), “make -j 2″ somehow resulted in an error which I could not work around. Reverting to simply make (without -j 2) and starting compilation from the very beginning fixed it.

Also, after final make install, I could not start erl – it was complaining about “start.boot not found”. The solution is to symlink boot files like this:


cd /usr/lib/erlang/bin
ln -s /usr/lib/erlang/releases/R13B02/start.boot .
ln -s /usr/lib/erlang/releases/R13B02/start_clean.boot .
ln -s /usr/lib/erlang/releases/R13B02/start_sasl.boot .


I configured it with “./configure –prefix=/usr –disable-x –enable-threads –enable-kernel-poll –disable-hipe”.

Version of Erlang I used was R12B-5. I configured it with ./configure –disable-x –enable-threads –disable-hipe.

But it wouldn’t build, giving me the following error:

Fatal, could not get clock_monotonic value!, errno = 22

This was strange because I had no problems building this version of erlang on Debian Etch, even with an older compiler.

The solution was to edit all instances of config.h in the build tree (in my case, there were 2 – lib/erl_interface/src/i686-pc-linux-gnu/config.h and erts/i686-pc-linux-gnu/config.h) after running ./configure but before starting make and comment out this line:

/* Define if you want to use clock_gettime to simulate gethrtime */
/* #define GETHRTIME_WITH_CLOCK_GETTIME 1 */

Are you running broker as user rabbitmq and rabbitmqctl as root or rabbitmq? If not, please stop and fix this. This is not a requirement per se, but this represents canonical installation of rabbitmq broker. You can certainly hack your scripts to work around this requirement, but you are on your own if you do.

Double check that broker is in fact running. Use ps, netstat -lptn (look for port 5672 unless you overrode it in /etc/default/rabbitmq). Telnet to localhost on port 5672, type AMQP and press ENTER several times. You should get response that at least will show AMQ. Check logs at /var/log/rabbitmq to verify that broker saw your connection attempt.

Next step is to start “erl -sname foo -cookie coo” in shell and run this command: “net_adm:names().”

If this command returns ok followed by a list of nodes within 1 or 2 seconds, check if rabbit is there. If it is, it’s very likely that you have users mixed up above. Please double check. If rabbit node is not listed, double check that rabbit broker is still running.

If this command returns {error,address}, there is a problem with your instance of EPMD, erlang naming daemon (man net_adm). First, check if it’s running (it most likely *is* running). Then, in erlang, run “net_adm:localhost().” Exit erlang, and try to connect to exactly the name you got from net_adm:localhost() on port 4369 (epmd). This shouldn’t work and should timeout. If it doesn’t time out, you shouldn’t have gotten {error,address}.

The problem most likely will be that name as returned by net_adm:localhost() is associated with some IP in /etc/hosts or in DNS which is either not accessible from this host, or firewalled off. An entry in /etc/hosts that associates this name with 127.0.0.1 or one of other IPs on this server should fix the problem.

Alternatively, net_adm:names() may time out with {error, timeout}. We have seen it caused by snoopy in the past. Remove snoopy or do not install it system wide in /etc/ld.so.preload, and you should be fine.

If these steps did not help, please leave a comment below or join rabbitmq-discuss and ask your question there, and we’ll help!

UPDATE 2009-04-17: Your host’s name as shown by net_adm:localhost() does not technically need to be defined in /etc/hosts. But if it’s not there, it should not be defined anywhere – when you do “ping name”, you should get “ping: unknown host name”. I have seen at least one case when it worked this way. This is somewhat unverified though.

Here are some symptoms that you might see:

• net_adm:names() returns {error,timeout} when you do it inside “erl -sname foo”
• if you run epmd -d -d -d -d -d in your terminal and launch erl -sname foo, you will get the registration (ALIVE2_REQ, ALIVE2_RESP) but there will be no more communication between epmd and foo no matter what you do, until you Ctrl+C from foo. Ideally, you should be able to see PORT2_REQ and PORT2_RESP when doing net_adm:ping.
• “erl -sname foo” and “erl -sname bar” on the same machine will not be able to ping each other with “net_adm:ping(’foo@host’).” or communicate with each other.
• inet:gethostbyname(”any.domain.name”) will return {error,timeout} for all domains, even for names in your /etc/hosts, when you do “erl -sname foo” but might work when you do just “erl” (not starting a distributed node).

RabbitMQ related symptoms:

• broker will not start using /etc/init.d/rabbitmq-server start
• rabbitmqctl status will always fail with badrpc error

Or is it not a joke?…