Using haproxy with multiple backends a.k.a. content switching
We have been using haproxy as a load balancer for autoki (rails) for a while. The setup was relatively simple. We had two servers full of mongrels and one haproxy was simply throwing requests at them using round robin.
Now our requirements have gotten bigger. We have extracted the game into a separate application that is being deployed on its own server independently. The problem that we want to solve with this is that while we want to deploy changes to our servers whenever we want to, the game application has to keep running all the time without interruptions by a deployment and the problems that might be caused by it. Oh, and at the same time we didn’t want to change any urls. The game has been sitting in the external_games controller, so the url was autoki.com/external_games and since we have given this url to people we wanted it to stay the same.
So now we have a new server called external_games.autoki.com with 10 mongrels running on it and we want all requests to /external_games go there and everything else still go to the old servers. The solution to this is to let the load balancer decide which servers to forward the requests to. Since version 1.3 haproxy has implemented exactly this – and calls it content switching. To get thsi to work you have to tweak the haproxy.cfg file a bit. This is the old listen section with one pool of mongrels:
listen rails *:80
server rails-1 app1.autoki.com:7000 maxconn 1 check
server rails-2 app1.autoki.com:7001 maxconn 1 check
…
First we have split up our single proxy configuration into a frontend and a backend. We will actually be using two backends – one for gaming and one for the rest, and the frontend will decide which backend to use.
backend rails_game
server game-1 app2.autoki.com:7000 maxconn 1 check
server game-2 app2.autoki.com:7001 maxconn 1 check
backend rails_production
server rails-1 app1.autoki.com:7000 maxconn 1 check
server rails-2 app1.autoki.com:7001 maxconn 1 check
frontend rails *:80
The rule to decide which backend to use is implemented using an acl. The following line creates a rule with the name game that evaluates to tue, if the url contains the string external_games. The haproxy documentation has a whole list of keywords you can use to match against all the contents of a http packet.
acl game url_sub external_games
Now all we have to do is tell the frontend when to use which backend. We do this by defining a rule for the gaming backend and set a default backend for all requests not matching our acl.
frontend rails *:80
acl game url_sub external_games
use_backend rails_game if game
default_backend rails_production
That’s it. Now when we go to http://autoki.com/haproxy?stats we’ll see the two backends up and running. (only after adding stats enable to both backends actually)
Tags: acl, backend, content-switching, frontend, haproxy, load-balancer, mongrel, proxy, rails
February 9th, 2009 at 18:09
Thanks for the write-up!
Unfortunately I am running into some issues when I try to replicate this setup. My HAProxy config works great with a single website and a corresponding listen statement. But when I try to set up two websites by using a backend for each as well as a single frontend, switching between them using a url_sub acl very similar to your example, I keep getting 503 errors when I try to hit either of the websites.
Have you ever come across anything like this?
February 9th, 2009 at 21:07
sorry i’m afraid i can’t help you. in the meantime we switched all our apps to phusion passenger/apache which works out of the box.
February 10th, 2009 at 19:03
Ah, no worries.
I’m actually using Passenger as well, but I’m using HAProxy to load balance between multiple Rails app servers (running Passenger). I’m trying to use a single HAProxy instance to handle multiple, different websites, which is what I was trying to achieve with the backend / frontend setup. I ended up using different IP addresses instead, which works great.
February 14th, 2010 at 07:30
keep getting 503 errors <—-does the check.tx exist in the second website. It happened to me and solved it by copying the check.txt to the root directory of all my websites