I know that there are a lot of sysadmin/networking experts reading this Weblog, so I’m appealing for suggestions on the following question: What is the best open-source software for a firewall/load balancer to be used at photo.net?
http://www.photo.net/doc/design/firewall-load-balancer-200611.txt
Explains what we need.
Thanks in advance for your help.
From a brief review of your critera it seems to me that Perlbal (http://www.danga.com/perlbal/) might well solve most of your problem. As designed/used by LiveJournal (http://www.livejournal.com/). You’ll probably need some sort of basic firewall rules on top of that, but that part should be relatively straightforward.
Ewen
To answer you questions:
a) Yes, it is possible to do this with a Linux machine (or any Unix-like system really).
b) Yes, all of it can be done with OSS software.
You can use the built-in iptables firewall software of Linux to accomplish this. (Or any packet filtering system on Unix.)
For the web proxying / load balancing you can look at other software (as mentioned by Ewen) or use a recent (2.1.x, 2.2.x) version of Apache which has modules (mod_proxy, mod_proxy_balancer) that do this. Squid is another proxy software system that can do quite a lot of things. Lighthttpd is another less complicated web server that can also probably handle some of the things you’re talking about.
There isn’t much customization (hacking of source code) necessary, just editing some configuration files to get the set up you want.
c) and d) Any modern multi-gigahertz machine should be able to handle things quite well, but you may want to do some benchmarks to see where / when things start slowing down. ApacheBench is one type of OSS software that you can generate a load of simpel web requests.
e) A single switch shouldn’t be too much of a problem. One possible ‘gotcha’ that I can think of is that if you a special configuration on the switch (VLANs, routing, etc.), then if you change the configuration on the running switch, you risk losing it should the switch die. You would have to be diligent in making sure that the configuration is always backed up somewhere else so you can copy it to the replacement switch if the original one dies. This sounds simply but is easy to let slide–a flash card to save the config on is available for many models of switches.
Other than that, making sure you have a spare handy, and keeping the equipment new enough to be under warranty should be okay. Whether a support contract is needed (i.e., the manufacturer gets you a new part with in four hours) may or may not be useful.
How much you’re willing to spend on redudancy depends on how much downtime costs you and your customers for every unit of time you’re off the air.
Lets break the problem into its distinct parts and see if we can solve them in a very *nix style fashion (by chaining solutions together) (hey… you want low cost OSS… gotta work the system!)
The first problem is an IP Firewall. There’s really very little to say here. iptables comes with linux and works remarkably well. Mix this with the right network architecture, and you’re in business.
The second problem is persistent load balancing. You want to be able to define services based on protocol (port 80/443 to the web servers, 25 to the mail servers, etc, etc) And, preferably, if a user connects to web server #3, and then makes another request within X number of seconds their second request is also routed to web server #3. The LVS project actually manages this quite well (and cheaply) see: http://www.linuxvirtualserver.org including persistence (or affinity in cisco-ese) LVS also supports a couple of different connect and passthrough models for different requirements.
The third problem is that of load balancing your content (Thumbnails on web cluster 2 versus original photos on web cluster 1 for example) Your request was to be able to do that via the URL. This is where the answer gets a bit more ambiguous. You can use a reverse proxying setup (perlbal and pound for example) if you’re still interested in a URL based forwarding pattern. But if you were willing to dedicate an IP address you could use DNS based load balancing. Take the following setup, for example.
Request for server.com comes into the load balancer server.com is a.b.c.d. The load balancer knows that the web1 web2 web3 and web4 servers on the LAN are able to handle web requests for the ip address a.b.c.d. The load balancer hands the request off to web2. Further requests from the same client ip address in the next 180 seconds (because of persistence) also get directed to web2. A thumbnail appears on the page at thumbs.server.com which has a different ip address of a.b.c.e. The load balancer knows that web5 and web6 are set to be the real servers for a.b.c.e and the request is sent off to web6. Because persistence was not configured for a.b.c.e like it was for a.b.c.d the next request might be sent to web5 since all requests are round-robin load balanced for that IP address. Any time a web server goes down the mon (or other monitoring daemon) running on the load balancer notices and takes it out of rotation. and future requests are routed around the down web server until it is brought back up. Heartbeat detects if one of the LVS load balancers goes down and has the other assume its position.
In the above scenario you don’t really get to use the URL as part of the balancing mechanism, but you do get to direct based on content type.
You could also insert into the chain a reverse proxying solution (or cluster) and have the best of all available worlds by load balancing, in a highly available fashion, to the proxy servers
(Request) [ 2 LVS machines (public IPs) ] [ 4 Reverse Proxies (priv) ] [ X “real” web servers (priv) ]
It’s not as elegant as a set of $50k hardware load balancers, I’ll grant you, but it’s workable and a LOT cheaper. And I would definitely recommend not looking for an all-in-one solution for the problem in OSS (if you are) and start thinking in terms of chaining the proper functionality together.
A redundant pair of Cisco/Arrowpoint CSS11501 load balancers costs $15K (with SSL acceleration, it’s double, but you probably don’t need hardware accelerated SSL). At that price (keep in mind Cisco isn’t known to be the low-price leader), it doesn’t make much sense to putz around with a software solution when hardware is more scalable and reliable.
There’s no way to guarantee session persistence without making some application changes. Your problem is users with cookies disabled coming via a megaproxy so that their IP address changes for each request — you simply have nothing to track them by unless you embed some session information in the URL for every internal link on the site.
Mike: A lot of the photo.net services require a reader to be logged in, which means that they already have a cookie.
Fazal: I looked at that Cisco load balancer. It isn’t clear that patterns of the kind that I’m talking about can be implemented. What’s worse, it doesn’t look as though these things provide any firewall action. So we would be introducing an extra point of failure.
Perlbal and iptables on one or more Linux servers is definitely the way to go here. Both are OSS, and Perlbal has great scalability and it’s easy to write plugins to handle any site specific situations you might have. I really can’t recommend it enough.
A decent PC running Linux will be able to handle 150mbps fine. It will probably be able to handle 3x that pretty comfortably but in a commercial environment I wouldn’t be prepared to try it – although I understand photo.net’s situation though and you should be able to find plenty of willing Linux geeks to help tune it.
I have used LEAF (Linux Embedded Application Firewall – http://leaf.sourceforge.net) for border routers and firewalls with excellent results. It comes with Shorewall which makes complex iptables configurations easy and Quagga in case you need to do BGP. There is a module for keepalived which I’m pretty sure would allow you to load-balance your webservers.
Just saw the question at the end of your requirements doc about a smart switch.
It is possible to engineer it so you have two switches and each server has an uplink to each switch. This protects you against a switch failure but introduces additional complexity and things that can go wrong. Unless you have a network geek around, this could end up causing more downtime than would have resulted from a single switch failure anyway.
for #1, I strongly recommend OpenBSD plus their built-in packet filter (pf) for the firewalling. Much easier to understand and configure than iptables.
Anything with 2x gigabit NICs (they have better chipsets than 100mbit NICs) and a P3 processor will be able to handle your load quite easily.
For #2, :
Get a P4 machine for better SSL encrypt/decrypt, and make the proxy machine handle SSL connections as well. Then you can set a header value to let AOLserver know that the connection was SSL-secured.
For reverse proxying, you can use either Pound (www.apsis.ch/pound) or Apache with the mod_proxy and possibly mod_rewrite module.
Given the massive traffic photo.net sees, I doubt that a Perl based program would be the best solution. Everything I have mentioned is open-source.
Some of the stuff could be done inside a switch, but it would be more expensive and not opensource.
patrickg, I think the fact that livejournal is using it is a pretty good certification that perl can handle a pretty large load. The idea that
perl/php/ruby/”insert dynamic/scripting language of choice” can’t
handle big loads and scale well is pretty well disproved at this point
(see livejournal, digg, 37 signals projects for examples of each
language’s ability to scale) It’s not about the language you use, it’s
about having the time, money, and knowledge to work with what you use
enough to understand how to make it scale well.
I didn’t see this post before now, but I hope it’s not too late to offer some good, free advice here. 🙂
LOAD BALANCERS:
My understanding is that Willy Tarreau’s HAProxy is the premier open-source HTTP load balancer today. I have not used it myself, but it has a good reputation and seems well designed (http://haproxy.1wt.eu/). It’s author has written a great article, which gives a solid overview of best practices for load balancers today, at http://1wt.eu/articles/2006_lb/ .
I have used Pound myself, and it’s really nice (www.apsis.ch/pound/). Strong points are SSL support (not a requirement here) and ease of setup. I’m not sure if Pound is the optimal choice for a high-performance scenario, but then again I really don’t know how fast Pound can go. Consider hanging back in the older version numbers of Pound, sometimes the newest versions have less well tested functionality.
I have never used Danga’s PerlBal, but from what else I have seen from those guys I would have no reservations. Their site has a very good presentation about high-performance, dynamic-content web architectures (www.danga.com/words/).
The new kid on the block will probably be Varnish Cache. It’s brand new and not quite polished, but it’s architecture will allow it to scale beyond all the other contenders, and I know it’s main programmer to be really sharp. It’s probably not ready for your use now, but it should grow within the next year (www.varnish-cache.org; http://www.nuug.no/aktiviteter/20060919-varnish/)
Looking towards the future, it seems to me that the trend is to move away from the classic HTTP load balancer. Today HTTP servers have load-balancing built in, and can serve static content directly from the front-end server (the “load-balancer”), while balancing retrieval of dynamic content from back-end app servers via HTTP. Apache 2.2 can do this out of the box, I believe LightTPd can too (www.lighttpd.net), and I know that Litespeed Web Server 2.2 can (www.litespeedtech.com). The Litespeed server has a really good rep in the Ruby On Rails community; Lighttpd has a strong rep for high speed but has been a bit brittle in use. The website of Mongrel HTTPd has good docs on configuring these servers (http://mongrel.rubyforge.org > Documentation).
Performance is really a non-issue of sorts today. If your preferred load balancer / front end HTTPd isn’t fast enough, then you just add more copies of it. Typically you’d go with two cloned LB’s, DNS round-robin between the two, and IP level fail-over. This setup is cheap, and gives you good protection in the event of traffic spikes OR hardware failure (but not both a traffic spike AND a LB failure). There are many IP level fail-over systems, Linux HA, Linux Virtual Servers VRRPd etc., but Whack-a-Mole seems to be a favorite despite it’s age (www.backhand.org).
Another thing is that your usage is atypical. Most LB’s hit their bottleneck in the session creation phase, ie. how many new TCP/HTTP sessions can be established pr. second. Photo.net would create most of it’s load by sending large files in long-running TCP sessions. The internal design and buffering strategy of the balancer would matter much more than usual. I can’t say how this will work out, but my guess is that 150 Mbps in long-running TCP sessions is a piece of cake for a single contemporary server. Benchmarking your specific setup is really the only way to know.
APPLIANCES:
Loads of companies make load-balancing switches, but the prices are all over the map. Radware, Coyote Point, Zeus, Cisco and Juniper are some of the names. Perhaps some of these can be found at reasonable prices in your area, perhaps not. Firewalls are more commonly purchased as appliances, partially for ease of setup and layering of security responsibilities across multiple systems for resilience.
HTTP PERFORMANCE:
There is a small end-user performance benefit to spreading the HTML/CSS/image content over multiple host names, such as http://www.photo.net and static.photo.net, because the end-users browser then opens more parallel TCP connections. See http://yuiblog.com/blog/2006/11/28/performance-research-part-1/ for an introduction. I do not think this is worth optimizing for today, and would personally ignore it for now, but it is real.
MY CONCLUSION:
Load balancing is cheap and attainable today, also at 150 Mbps. The preferred setup really comes down to individual taste and past experiences. Personally, for a new site with those requirements, I’d first try Litespeed HTTPd, and if benchmarks show insufficient performance, then I’d add more HTTP front ends with Whack-a-Mole IP fail over. HAProxy on a dedicated load balancer PC also seems like a good, problem-free approach. Firewalling I would try to keep on a dedicated appliance, mostly so that I don’t accidentally create security holes while mucking around with IP settings on the load balancers.