The biggest enhancement (IMHO) is ability to increase or decrease the number of backend consumers based on any HTTP request headers. In “table” routing mode (see rabbitbal.yml), you can now specify array of tests against which incoming request headers will be matched. This will cause your request to be published with a matching key (note :key). Your backend consumers use the same YAML file and can bind to all or only some queues, thus giving you flexibility in adjusting the capacity. Old functionality is available by using “topic” routing mode.

Note that I still use topic-based exchange, because I wanted to support a use case where you want to aggregate all incoming requests into separate queues (routing key would be something like “request.#”) for bot detection, access log aggregation, etc. In other words, each request ultimately must end up in a single queue where it will be picked up by backend servers, while at the same time it can also be duplicated into other queues for other purposes.

Let’s say you are running a blog and hosting your own feed. Regular crawler bots from search engines like Google, Yahoo, MSN, etc will always check robots.txt before sending next batch of requests, and they won’t ask for the same URL very frequently.

The second type of bots you are going to deal with, are RSS reader bots. There is no reason for them to hit anything but URL for your feeds. Such bots usually will not check robots.txt. Their user agent string (which you can see if you enable CustomLog /path/to/log combined) will usually have “N subscribers,” which can give you a rough idea how many people subscribe to your feed via this service. These bots may hit you with varying regularity, ranging from half an hour to several hours between requests. Additionally, well behaving RSS reader bots will always include If-Modified-Since in their requests – you will see in your logs that your usual response to these queries is 304 Not Modified, and only once after new post is published you should see 200 OK response.

So what did I see in my logs? First, I noticed a bot requesting / from 2 different IPs every hour, without including If-Modified-Since. Wasteful and negligent. If this bot does not know how to appreciate my server resources, I am sure it won’t miss my content – block!

<Location />
order allow,deny
allow from all
# BadBot1 is from its User-Agent string
SetEnvIf User-Agent BadBot1 DenyBot=1
deny from env=DenyBot
</Location>

Then I noticed several RSS reader bots that were requesting /feed/ way too frequently (every 10 minutes for 1 subscriber and every 60 minutes for one susbcriber) and were very inconsistent with If-Modified-Since – I couldn’t detect logic in their requests, but sometimes I saw 304 and sometimes I saw 200 response, even when there was no new content on my site.

I didn’t feel right blocking these altogether, so instead what I did is I opened a particular 1 hour window during the night when I allow such bots to get the feed’s content – all other times, their requests are blocked.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} .*BadBot2.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*BadBot3.*
RewriteCond %{REQUEST_URI} /feed/
RewriteCond /etc/allow_bad_bots !-f
RewriteRule . - [forbidden]

Voila! If file /etc/allow_bad_bots exists (I create and delete it from cron, it exists on my system between 1am and 2am), requests from these bots will succeed. During the rest of the day, these rude bots are getting 403 Forbidden.

1. An HTTP request comes to Perlbal.
2. Perlbal reverse-proxies it to one of its backend servers.
3. Backend server does some work (in my case, does extensive verification of URL) but instead of returning entire response (status, headers, body), it returns X-REPROXY-URL header which includes a list of URLs.
4. Seamlessly to end user, perlbal attempts to fetch content from one of these URLs, and returns that new content to the user.

The other day I found out that Perlbal can’t reproxy to URLs that require HTTP basic authentication. Here is a part of Perlbal that parses X-REPROXY-URL header and you can clearly see from regex, it treats URLs as (host, port, path) tuples (this is from ClientProxy.pm).

    # construct reproxy_uri list
    if (defined $urls) {
        my @uris = split /\s+/, $urls;
        $self->{currently_reproxying} = undef;
        $self->{reproxy_uris} = [];
        foreach my $uri (@uris) {
            next unless $uri =~ m!^http://(.+?)(?::(\d+))?(/.*)?$!;
            push @{$self->{reproxy_uris}}, [ $1, $2 || 80, $3 || '/' ];
        }
    }

And my backend Apaches do require http auth. What to do?

RTFM to the rescue! Apache provides a very cool feature – Satisfy (all|any) command. Essentially, it means that for a Directory or Location I can specify both http auth and IP based access control, and using Satisfy Any I can allow access if at least one of these conditions are met (default is Satisfy All).

Here is what it looks like in http config:

<Location /foo>
  # http auth
  AuthType basic
  AuthName "protected"
  AuthUserFile /etc/apache2/users
  Require valid-user

  order allow,deny
  # this is subnet where perlbal is running
  # backends see perlbal's reproxy requests from this subnet
  allow from 192.168.4 127.0.0.1
  satisfy any
</Location>

Alternatively, I would need to create a fake URI outside of http auth location and rewrite it with mod_rewrite, or possibly use a symlink – way less transparent.

Here are benefits of AMQP-based approach over traditional HTTP-based reverse proxies taken from rabbitbal README file (in no particular order) as I see them.

1. Model where workers fetch work as fast as they can (each going at its own pace) in theory should provide more efficient load balancing than a model where proxy assigns work based on certain criteria; methods based on round robin, arbitrary weights or least connections become simply unnecessary.
2. RabbitMQ broker implements intelligent failover out of the box – if a web server disconnects before ack’ing, the request will be automagically requeued for another server; all in all, RabbitMQ is way smarter than a bunch of low level TCP connections.
3. Enhanced security of actual web servers – servers behind Rabbitbal do not need inbound connectivity, they only need to be able to establish an outgoing connection to RabbitMQ broker(s).
4. Rabbitbal does not need to know IPs or have direct connectivity into its web servers (use case: Amazon EC2 without Elastic IPs)
5. Using Duplication pattern of RabbitMQ (see Resources below), you could be reading requests and responses off of the same broker in real time (access log aggregation, double-entry book keeping, logging, bot detection)
6. You could relatively easily have one request go to more than 1 web server
7. Add capacity as often and as much as you like – rabbitbal won’t even know
8. By slightly readjusting mapping between queues and URIs, you could add or remove capacity per URI if needed
9. TCP overhead savings compared with HTTP proxies (AMQP uses persistent TCP connections)

On your phone, head over to http://www.google.com/gwt/n and enter URL you are trying to get. GWT (which stands for Google Web Toolkit) will fetch the content and optimize it for your mobile browser. Additionally, it will adjust all links to also go through GWT, which makes Internet surfing with Blazer not painful at all.

For example, I like checking Techmeme on train on my way to work. They offer mini version, which renders well on my Palm Centro from Sprint. But if I want to follow a story and click on a link, I usually get the page not optimized for mobile (there are several exceptions that detect user agent and adjust content formatting). Instead, in my Blazer bookmarks, I have this – http://www.google.com/gwt/n?u=http%3A%2F%2Ftechmeme.com. From this page, I can jump to any story and get the content nicely formatted for my Centro.

UPDATE: It looks like I might have confused Google Wireless Transcoder (GWT) with Google Web Toolkit (GWT).

I did it a couple of weeks ago when I was moving this site to its current address, but noticed today that my old address still shows up in Google at the top of search results. I got curious, and checked both Yahoo and MSN and both of them properly do not display links that have been redirected.

Am I missing anything, or is it a bug in GoogleBot?