One obstacle was that because we were emulating various failures, at times our local capistrano process, which was driving the tests, had to lose SSH connectivity to its target servers. We quickly discovered that this resulted in exception and cap process would die.

To work around this, I added yet another level on top of cap which uses GNU make (one of my all time favorites). In a nutshell, user controls the testing process via make, and make starts cap. In this case, it’s ok for cap process to occasionally exit.

But then – and we are finally getting to the point of this post – another issue came up: I didn’t want to keep typing password into cap each time it was started by make. Here is how I ended up implementing it to avoid re-typing password.


# in Makefile
USER_PASS := $(shell read -s -p "[make] user's password: " P; echo $$P )
export USER_PASS

all: set_password
# do something here

set_password:
    @test "$(USER_PASS)"


# in Capfile
set :password, lambda { ENV['USER_PASS'] ||
CLI.password_prompt("[cap] #{user}'s password: ") }


And here are my reasons.

Memory Management

Shell scripts are excellent in managing their memory and one has to try real hard to cause a shell script to leak memory. This makes shell a very convenient tool for long running processes, supervisors in multiple-workers models, daemons and so on. There is an easy explanation for this. In shell, there are only a handful of built-in primitives – everything else is an external command, which gets started and then finishes before giving control back to your script. If there is a memory leak in that command, it won’t damage your calling script and will usually be insignificant because it will return quickly.

No Exceptions

This is a double edged sword, and you need to be careful how you exploit this “weakness.” This feature allows me to write compact code which is easy to understand without enclosing every single command in “try… except”. For naysayers, I would like to point out that a strict mode exists, where every error is treated as fatal and causes the script to exit (set -e).

In general, not all unforeseen error conditions warrant a crash, like you get in Python or Ruby when an unhandled exception gets propagated all the way to the top. If a problem is transient, it may be better to ignore it temporarily.

To assure a Ruby or Python script doesn’t crash on some unforeseen transient problem, many people often end up enclosing their entire program in a wildcard try… except block to catch any exception – but to me this approach is dangerous, even though I sometimes end up using it myself.

If you are writing a daemon process to perform some action in a loop, shell is often by far the most stable alternative.

When Not To Use Shell

My personal rule of thumb is don’t use shell when you expect to need high-level data structures like hashes or arrays beyond what for loop can give you, or when you can see potential for code reuse following OOP patterns like inheritance, or when your program needs to participate in some orchestration schemes that go beyond creating and removing files on the filesystem.

Conclusion

I wouldn’t overlook shell if I were you.

The biggest enhancement (IMHO) is ability to increase or decrease the number of backend consumers based on any HTTP request headers. In “table” routing mode (see rabbitbal.yml), you can now specify array of tests against which incoming request headers will be matched. This will cause your request to be published with a matching key (note :key). Your backend consumers use the same YAML file and can bind to all or only some queues, thus giving you flexibility in adjusting the capacity. Old functionality is available by using “topic” routing mode.

Note that I still use topic-based exchange, because I wanted to support a use case where you want to aggregate all incoming requests into separate queues (routing key would be something like “request.#”) for bot detection, access log aggregation, etc. In other words, each request ultimately must end up in a single queue where it will be picked up by backend servers, while at the same time it can also be duplicated into other queues for other purposes.

Here are benefits of AMQP-based approach over traditional HTTP-based reverse proxies taken from rabbitbal README file (in no particular order) as I see them.

1. Model where workers fetch work as fast as they can (each going at its own pace) in theory should provide more efficient load balancing than a model where proxy assigns work based on certain criteria; methods based on round robin, arbitrary weights or least connections become simply unnecessary.
2. RabbitMQ broker implements intelligent failover out of the box – if a web server disconnects before ack’ing, the request will be automagically requeued for another server; all in all, RabbitMQ is way smarter than a bunch of low level TCP connections.
3. Enhanced security of actual web servers – servers behind Rabbitbal do not need inbound connectivity, they only need to be able to establish an outgoing connection to RabbitMQ broker(s).
4. Rabbitbal does not need to know IPs or have direct connectivity into its web servers (use case: Amazon EC2 without Elastic IPs)
5. Using Duplication pattern of RabbitMQ (see Resources below), you could be reading requests and responses off of the same broker in real time (access log aggregation, double-entry book keeping, logging, bot detection)
6. You could relatively easily have one request go to more than 1 web server
7. Add capacity as often and as much as you like – rabbitbal won’t even know
8. By slightly readjusting mapping between queues and URIs, you could add or remove capacity per URI if needed
9. TCP overhead savings compared with HTTP proxies (AMQP uses persistent TCP connections)

This means that if at certain point during the project you adjust validations such that some records that used to be valid are now invalid, expect your code to stop working without any obvious reasons. This will happen because update_attributes will no longer update old records properly, but unless you check its return value, you’ll never know that it’s failing.

I stumbled upon this problem at least twice in the last month, and decided to write it up to finally remember to do it right next time.

Prerequesites:

• RabbitMQ broker configured, up and running on 127.0.0.1 (localhost) on port 5672 (standard AMQP port).
• Apache QPid Ruby library installed within RUBYPATH (svn co http://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/ruby)
• AMQP specification XML form AMQP official site saved as /etc/amqp0-8.xml

You can also download this script from here.

#!/usr/bin/ruby -I/usr/local/qpid-svn/qpid/ruby
#
#
__doc__ = %q(
 
disttailf.rb - distributed "tail -f"
 
Aggregates "tail -f" output from multiple machines and multiple files
into a single RabbitMQ pubsub queue (kind of splunk's log consolidation
function)
 
Usage:
Producer: disttailf.rb [-s broker_host] [-p broker_port] [-x spec_xml] file ...
Consumer: disttailf.rb [-s broker_host] [-p broker_port] [-x spec_xml] -c
 
)
 
require 'qpid'
require 'socket'
 
def consumer(client, ch)
    myqueue = ch.queue_declare()
    ch.queue_bind(:queue=>myqueue.queue, :exchange=>'amq.topic',
                    :routing_key=>'disttailf.#')
    cons = ch.basic_consume(:queue=>myqueue.queue, :no_ack => true)
    ruby_queue = client.queue(cons.consumer_tag)
 
    while true
        raise "Rabbitmq broker disconnected" if client.closed?
        begin
          msg = ruby_queue.pop(non_block=true)
          puts "== #{msg.content.headers[:headers]} " \
                "#{msg.routing_key.split('.')[-1]}"
          puts msg.content.body
        rescue
          sleep(0.5)
        end
    end
end
 
def producer(client, ch, filenames)
    rkey = "disttailf." + Socket.gethostname.split('.')[-1]
    tail_f(filenames) do |filename, line|
        h = {'sent' => Time.now.to_i, 'filename' => filename }
        c = Qpid::Content.new({:headers=>h}, line)
        ch.basic_publish(:routing_key=>rkey, :content=>c,
                            :exchange=>'amq.topic')
        puts "#{filename}: #{line}"
    end
end
 
def tail_f(filenames, &block)
    filedict = Hash.new
    filenames.each { |f| filedict[f] = open_or_nil(f) }
    reopen_counter = 0
    while true:
        if reopen_counter > 120
            reopen_counter = 0
            filenames.reject { |f| filedict[f] }.each {
                |f| filedict[f] = open_or_nil(f) }
        end
 
        filedict.values.reject { |f| not f }.each do |f|
            begin
                raise "trunc" unless File.stat(f.path).size >= f.tell
            rescue
                $stderr << "#{f.path}: removed or truncated\n"
                f.close
                filedict[f.path] = nil
                next
            end
 
            begin
              block.call(f.path,f.readline) while true
            rescue EOFError
              true
            end
        end
 
        reopen_counter += 1
        sleep(0.5)
   end # while true
end
 
def open_or_nil(filename)
    begin
        File.open(filename)
    rescue
        nil
    end
end
 
if __FILE__ == $0
    require 'getoptlong'
 
    server = '127.0.0.1'
    port = 5672
    specxml = '/etc/amqp0-8.xml'
    acts_as_consumer = false
 
    opts = GetoptLong.new(
        ['--server', '-s', GetoptLong::REQUIRED_ARGUMENT],
        ['--port', '-p', GetoptLong::REQUIRED_ARGUMENT],
        ['--specxml', '-x', GetoptLong::REQUIRED_ARGUMENT],
        ['--consume', '-c', GetoptLong::NO_ARGUMENT])
    opts.each do |opt,arg|
      case opt
        when '--server'
            server = arg
        when '--port'
            port = arg.to_i
        when '--specxml'
            specxml = arg
        when '--consume'
            acts_as_consumer = true
      end
    end
 
    # set up connection to rabbitmq broker
    client = Qpid::Client.new(server, port, spec=Spec.load(specxml))
    client.start({ "LOGIN" => "guest", "PASSWORD" => "guest" })
    ch = client.channel(1)
    ch.channel_open()
 
    if acts_as_consumer
        consumer(client, ch)
    else
        if ARGV.length == 0
            puts __doc__
            raise "List of file names is empty - nothing to do"
        end
        producer(client, ch, ARGV)
    end
 
end