A Look at Multihoming and BGP

by Iljitsch van Beijnum, author of BGP

Can you even imagine doing your job without being able to send and receive email, or look up information on the Web? Being disconnected from the Internet hurts, even if you don't immediately lose millions of dollars in revenue, as would be the case with eBay or Amazon. But the fiber and copper wires that connect you just sit there in the ground, waiting for a backhoe or a rodent to come along, or for the moisture to finally seep through. Then there is all the equipment, which also doesn't last forever, and which has software bugs and is prone to configuration mistakes. A single typo can bring down a good-size part of an Internet service provider network. Bankruptcy will do it, too. The only way to protect yourself against all of these eventualities is to have more than one connection to the Internet. This is called multihoming.

Multihoming Scenarios

Getting a second connection is just the first step. When the first connection goes down, you will need to start using the second one. There are three ways to accomplish this:

  1. Get two independent connections and configure your equipment with two IP addresses. When the IP address associated with one connection doesn't work anymore, the router or server has to switch over to the other. This approach has several limitations. Obviously, all ongoing communication sessions are disrupted by the change in IP address. Also, this functionality isn't always available in routers, servers, or other equipment. Finally, most applications (such as Web browsers) aren't designed to try multiple IP addresses just in case one is down, so this approach is generally only workable if the multihomed network only has clients and not servers.

  2. Get two connections to the same ISP. This ISP will then make sure you can use the same IP addresses over both connections. How this is exactly configured on your side depends on your ISP and on what your requirements are. A setup where you use two routers, one for each connection, is of course more complex, but this way, you can survive a router going down. In many cases, the actual configuration will be very similar to the next option (BGP), but because you don't need a "real" AS (Autonomous Systems) number or an independent IP address range, it is much easier and cheaper to get this off the ground, and your ISP protects you against most configuration mistakes. Having two connections to the same ISP has one major downside: you still have to depend on a single ISP. Fortunately, ISP-wide network failures are very rare, but all ISPs have points of presence or connections to certain external networks that go down from time to time. And there is always the risk of your ISP going belly-up.

  3. Get two connections to two ISPs, but use the same IP address range over both. This way, you are not only safe from physical problems with telco connections, but also from ISP-wide failures and nearly all types of configuration problems that affect just a single ISP: if one fails, you just use the other. This is done on a per-destination basis: your router can figure out which destinations are reachable over which ISP, and route packets accordingly. This works extremely well, but there is still a downside: all major routers throughout the Internet must know over which ISPs your IP addresses are reachable at all times. This means you have to participate in inter-domain routing using the Border Gateway Protocol (BGP).

The AS Number, IP Addresses, and Configuring BGP

BGP is a routing protocol, just like RIP, EIGRP, OSPF, or IS-IS. But unlike these interior routing (gateway) protocols (IGPs), BGP is an exterior gateway protocol (EGP), designed to exchange routing information between different organizations (or Autonomous Systems) in BGP-speak. Each AS needs its own AS number, along with its own range of IP addresses. Both IP addresses and AS numbers are available from the three Regional Internet Registries:

  • ARIN: serving the Americas and part of Africa.

  • RIPE: serving Europe, the Middle East, the former Soviet Union, and the rest of Africa.

  • APNIC: serving the Asia-Pacific region.

AS numbers aren't hard to get: if you multihome, you qualify. (There is usually a fee.) IP addresses are a bigger problem: because so many networks announce so many IP address ranges over BGP (often many more than necessary), more and more ISPs filter out announcements for small address blocks. Then the user of these addresses is unreachable for that ISP, defeating the purpose of multihoming. The Internet Registries aren't helpful in this regard either, because they won't give you a large block of your own unless you can show you really need that many addresses. This is a good thing because otherwise they'd have run out of addresses years ago.

Related Reading

Building Reliable Networks with the Border Gateway Protocol
By Iljitsch van Beijnum

So in most cases, multihomed organizations don't get their own block, but request a range from one of their ISPs, and then proceed to announce this range over BGP as if it were their own. This announcement will overlap with the one from the ISP the addresses came from, but this isn't a problem: routers always look at the smallest address block that matches a destination. This small block is still subject to being filtered, but this isn't as big a problem as it is with an independent range; whenever the multihomer's announcement is filtered out, the packets still flow towards the ISP announcing the larger block. However, there are some caveats. Most importantly: if you ever stop buying services from the ISP that gave you an address block, you have to give back the addresses.

Configuring BGP on one or more routers isn't hard, but you really need to know what you're doing to avoid mistakes that will come back to haunt you when something goes down. This is not the time you want to find out that you didn't know as much about inter-domain routing and BGP as you thought you did. Training courses (at least the ones I've done) prepare you for this the same way being a passenger prepares you for driving a car: not to any noticeable degree. This is because those courses only focus on simply configuring a few BGP sessions; they don't tell you what will and what will not fly in the real world. You need to read a book or a number of articles on configuring BGP in an Internet environment. Or you can hire someone with experience in this area to handle the entire project (this will cost a lot); to just implement the actual BGP configuration (this will cost a lot less); or something in between.

Once you've decided you really want to do it, follow these steps to multihome to two different ISPs using BGP:

  1. Decide what kind of help you need: a consultant, help from your ISP, or a book, such as BGP.

  2. Learn about IP address ranges, prefixes, and netmask notations, as well as address allocation and assignment policies.

  3. Decide on the type of address space you're going to use.

  4. Select (a second) ISP and get your ISPs on board with your plans, preferably in writing.

  5. Decide on the physical infrastructure: how many routers, what goes where, whether the connections share a physical path, how much bandwidth you need from each ISP.

  6. Request an AS number and IP address space, if necessary.

  7. Select one or more routers with enough CPU capacity and memory (128MB or more, or you need a more complex and less optimal configuration) to handle BGP, and make sure you have a software feature set supporting BGP.

  8. Configure BGP and set filters to avoid improper route announcements (such as routes from ISP A to ISP B, and vice versa).

  9. Register your routing policy in a Routing Registry.

  10. Use looking glasses to see if your announcements are visible elsewhere on the Internet.

  11. Periodically test if everything is as it should be by temporarily disabling a connection and checking if you can still reach all destinations, first for one connection and then the other.

  12. If necessary, adjust BGP properties to balance incoming and outgoing traffic to match the available bandwidth.

I hope I've provided some insight into the complexity of multihoming with BGP. It is not something to be taken lightly, but if you're prepared to invest time and effort in BGP, you can avoid having to depend on a single Internet service provider.

O'Reilly & Associates will soon release (September 2002) BGP.

Iljitsch van Beijnum has been working with BGP in ISP and end-user networks since 1996.

Return to the O'Reilly Network.