High-availability Clusters
and Routing with HA Linux

Sean Reifschneider

tummy.com, ltd.

Introduction

• Unavailability costs money
• Impacts reputation
• High Availability can save the day
• HA Routing is a good first choice

Overview

• Why routing?
• Introduction to Linux HA
• Overview of HA routers
• Example configuration
• Conclusion

Why HA Routers

• "Self-healing"
  • Routing back up seconds after a hardware failure
  • Can route around certain networking problems
• Inexpensive
• Reduces cost impact of failures
• "Easy"
• 0 Downtime maintenance

What is Linux HA

• High-availability cluster
• Two machines
• Service migrated between machines
• Failure detection, Fail-over
• Also on FreeBSD and others
  • Harder than Linux
  • In FreeBSD ports, Thanks to Scott Kleihege

Diagram: Regular Router

Diagram: HA Router

Heartbeat

• Machines exchange heartbeats
• Multiple communication paths
• Runs service on active node
• Loss of heartbeat triggers failover
• Beware of split brain
  • Less of a problem in routing

IPFail

• Ping multiple IP addresses
• Fail-over if one node has better connectivity
• Useful on some router configurations
• Back-up for dynamic routing

STONITH

• Communication failure
• Both nodes believe they are active
• Two machines scribbling on shared disc
• During failover, ensure other node is down
• UPS or serial/network power switch
• External shell script

DRBD

• Shared file-system between nodes
• Multi-system disc mirroring
• Less useful for router configurations
• More for HA web servers or similar

VRRP

• Migrates MAC address between nodes
• Brief downtime for failure
• Less capable in dynamic-routing environment
• Not as mature under Linux as Linux-HA
• Available in other hardware
• Similar results
• Linux-HA is much more capable

HA Router Overview

• BGP sessions/OSPF adjancies
• Router IPs/Gateway IP
• Gratuitous ARP
• 0 Packet loss on clean failover
• BGP timeout takes about 90 seconds on dead node
• OSPF timeout around 45 seconds

Example Setup

• /etc/ha.d/haresources:
  • router1 IPaddr::10.0.0.254
• /etc/ha.d/authkeys:

  • auth 1
    1 md5 cd71992c84b803645836426017aa6fd9

• (dd if=/dev/urandom bs=128 count=1 | md5sum)
• /etc/ha.d/ha.cf:

  • auto_failback on
    node router1.example.com
    node router2.example.com
    ping 10.0.1.254
    ping 10.0.2.254
    ping 10.0.0.1
    respawn /usr/lib/heartbeat/ipfail
    udpport 694
    ucast eth0 10.0.0.253
    ucast eth0 10.0.0.252

Details

• The hardest part
• Does it even make sense for you?
• Finding the right balance of HA
• Check for single points of failure
  • Power
  • Networking
  • Heartbeat Communication
• Testing, testing, and more testing
• Do you need STONITH?

Conclusion

• Linux-HA clusters can inexpensively increase availability
• Routers are an easy way to get your feet wet
• Simple setups are fairly easy
• Beware of the details
• 0 downtime maintenance is a huge benefit

Resources

• http://www.linux-ha.org/ -- Linux-HA Web-site
• http://wiki.linux-ha.org/ -- Linux-HA Wiki
• Blueprints for High Availability by Evan Marcus and Hal Stern
• IBM employs Alan Robertson and offers consulting services
• tummy.com, ltd. also offers Linux HA hosting and consulting