IP FailOver Setup Using KeepAlived on CentOS & Red Hat

By 4 Mins Read

Keepalived is used for IP failover between two servers. It facilities for load balancing and high-availability to Linux-based infrastructures. It works on VRRP ( Virtual Router Redundancy Protocol ) protocol. We have running two load balance servers using HAProxy and now we need to implement VRRP between both servers. This tutorial will help you to configure KeepAlived, Use this tutorial to configure HAProxy on both servers.

Advertisement
Network Scenario:
    1. LB1 Server: 192.168.10.111 ( eth1 )
    2. LB2 Server: 192.168.10.112 ( eth1 )
    3. Virtual IP: 192.168.10.121

Now we are implementing ip failover setup between LB1 and LB2 servers.

Graphical representation of Fail over Setup:

keepalived-vrrp-network

Step 1 – Install Required Packages

Use the following command to install required packages to configure Keepalived on the server.

# yum install gcc kernel-headers kernel-devel
Step 2 – Install Keepalived

Keepalived is available in centos base repository. Install it using yum command line tool.

# yum install keepalived

Keepalived configuration File: /etc/keepalived/keepalived.conf

Step 3 – Configure Keepalived on LB1

Edit Keepalived configuration file on LB1 and add following configuration. Update all red highlighted values with your network and system configuration.

! Configuration File for keepalived

global_defs {
   notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb1@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.121
    }
}
Step 4 – Configure KeepAlived on LB2

Edit Keepalived configuration file on LB2 and add following configuration. While making changes in LB2 configuration file, make sure to set priority values to lower than LB1. For example below configuration is showing 100 priority value than LB1 has it 101.

! Configuration File for keepalived

global_defs {
   notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb2@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.10.121
    }
}
1. Priority value will be higher on Master server, It doesn’t matter what you used in state. If your state is MASTER but your priority is lower than the router with BACKUP, you will lose the MASTER state.
2. virtual_router_id should be same on both LB1 and LB2 servers.
3. By default single vrrp_instance support up to 20 virtual_ipaddress. In order to add more addresses you need to add more vrrp_instance
Step 5 – Start KeepAlived

Start KeepAlived service using the following command and also configure to autostart on system boot.

# service keepalived start
# chkconfig keepalived on
Step 6 – Check Virtual IPs

By default virtual IP will be assigned to master server, In case of master gets down, it will automatically assign to the slave server. Use the following command to show assigned virtual IP on the interface.

# ip addr show eth1

Sample output

2: eth1:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:6f:ed:60 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.111/24 brd 192.168.1.255 scope global eth1
    inet 192.168.10.121/32 scope global eth1
    inet6 fe80::20c:29ff:fe6f:ed60/64 scope link
       valid_lft forever preferred_lft forever
Step 7 – Verify IP Failover

Follow the below process to test keepalived failover is working correctly.

  • Shutdown master server ( LB1 ) and check if ips are automatically assigned to the slave server.
    # ip addr show eth1
  • Now start LB1 and stop slave server ( LB2 ). IPs will automatically be assigned to master server.
    # ip addr show eth1
  • Watch log files to insure its working
    # tailf /var/log/messages

    Sample Output

    Mar 19 17:30:24 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar 19 17:30:25 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar 19 17:30:25 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar 19 17:30:25 localhost Keepalived_healthcheckers[6957]: Netlink reflector reports IP 192.168.10.121 added
Mar 19 17:30:25 localhost avahi-daemon[1407]: Registering new address record for 192.168.10.121 on eth1.IPv4.
Mar 19 17:30:25 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for

I hope this article will help to setup IP failover between two load balance servers.

Share.

Related Posts

16 Comments

  1. Kavshal on

    Hi Rahul,

    I have below setup for mysql:
    Machine #A – 192.168.10.1
    Machine #B – 192.168.10.2
    VIP between Machine #A and Machine #B is 192.168.10.3

    Let say keeapalived is master in Machine #A, possible the application not to use virtual ip to connect to the destination. Sample example is connecting to mysql:
    Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
    tcp 0 0 192.168.10.3:32444 192.168.10.3:3306 ESTABLISHED
    tcp 0 0 192.168.10.3:32442 192.168.10.3:3306 ESTABLISHED

    What I’m expecting is as below:
    Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
    tcp 0 0 192.168.10.1:32444 192.168.10.3:3306 ESTABLISHED
    tcp 0 0 192.168.10.1:32442 192.168.10.3:3306 ESTABLISHED

    Anyway I can achieve above?

    Reply
  5. Ram J on

    Hi,
    First of all this is a post with good explanation. I have a couple of questions regarding sync between servers. As we all know failover/redundancy servers needs to be synchronised to have consistent data across all the servers. One of the site recommended to run rsync to sync data between master and secondary servers periodically. At the worst case consistent data may not be available on a particular server at that point of time when sync hasn’t happened yet. So I was wondering is it possible to feed data from the firewall itself to multiple systems in parallel so that there wont be any sync issues? Kindly let me know. Thanks

    Reply
  6. rcb on

    Hi Jaffar,
    vrrp_script is a script which will be running every 2 s (interval 2), so i dont think that you want that every 2s “killall -0 haproxy” will be executed. Thus i think you should run a command that check the state of your proxy like ps or a special command of your proxy. Anyway, i think that the fail over happens because if the secondary server is the first which runs keepalived, he will runs “killall -0 haproxy”, and increment the priority by 2 (weight 2), so the priority becomes 102 > 101 (the priority of the and increment the priority by 2 (weight 2)primary server) => FAILOVER, after that as the same way, the primary server runs “killall -0 haproxy” and increment the priority by 2 (weight 2), so the priority becomes 103 > 102 => FAILOVER ….

    Reply
  7. Jaffar on

    Hi Guys,
    We are having below config for keepalived to have two HAproxy servers shared the VIP:
    global_defs {
    router_id haproxy0
    }
    vrrp_script haproxy {
    script “killall -0 haproxy”
    interval 2
    weight 2
    }
    vrrp_instance 243 {
    virtual_router_id 243
    advert_int 1
    priority 101

    state BACKUP
    nopreempt

    interface eth0
    virtual_ipaddress {
    192.168.40.41 dev eth0
    }
    track_script {
    haproxy
    }
    }

    Pretty much same config is on the other HAProxy server with exception to the router_id and priority (100).

    Teh config works well in normal scenarios. Once in a while we are observing that Secondary sever tries to claim the VIP (may be the advert messages are not reaching it) and actually claims it and releases it again in few seconds after it get the high priority advert message from Primary Server which has been holding the VIP. The other interesting part is neither HAProxy nor Keepalived ever went down but failover and failback has happend. The servers are under same network. Below are the logs:

    Primary server:
    Dec 9 06:25:02 localhost tripwire[10745]: Integrity Check Failed: File could not be opened.
    Dec 9 12:26:37 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
    Dec 9 12:26:37 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
    Dec 9 12:26:38 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
    Dec 9 12:26:38 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
    Dec 9 12:26:39 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
    Dec 9 12:26:39 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
    Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
    Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
    Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
    Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41

    Secondary server:
    Dec 9 06:25:03 localhost tripwire[31150]: Integrity Check Failed: File could not be opened.
    Dec 9 12:25:06 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Transition to MASTER STATE
    Dec 9 12:25:07 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Entering MASTER STATE
    Dec 9 12:25:07 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) setting protocol VIPs.
    Dec 9 12:25:07 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
    Dec 9 12:25:07 localhost Keepalived_healthcheckers[20741]: Netlink reflector reports IP 192.168.40.41 added
    Dec 9 12:25:09 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Received higher prio advert
    Dec 9 12:25:09 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Entering BACKUP STATE
    Dec 9 12:25:09 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) removing protocol VIPs.
    Dec 9 12:25:09 localhost Keepalived_healthcheckers[20741]: Netlink reflector reports IP 192.168.40.41 removed

    Any help around this would be appreciated.

    Reply
  9. saurabh on

    IP owner case is not working in keepalive and dont know where to use primary ip address ? can anyone see config file and below explanation and let me know is it correct config for owner case or what ????

    Keepalived config for Master host(HOST A) which has its interface IP in the VIP block ::interface ip of (eth1 30.30.30.1/24) sample config::
    eth1 is 30.30.30.1/24

    keepalived.conf file::
    vrrp_instance VI {
    state MASTER
    interface eth1
    virtual_router_id 1
    advert_int 1
    priority 150
    authentication {
    auth_type NONE
    auth_pass 1112
    }
    virtual_ipaddress {
    30.30.30.1/24
    30.30.30.3/24
    }
    }

    In this file there is no field to specifically save the primary address & all configured addresses goes to the VIP list block. Can you please let me know where i can use primary ip address will be configured in above configuration file.
    Moreover if we add Primary address (which is always the interface address as ip owner case says) to the VIP block in KA: we see the following issue::

    We have two hosts which are participating in a VRRP instance. They have same priorities(above see config)::

    Host A – Interface IP “30.30.30.1” (Lower), priority “150” (same), vrrp instance “1” (same).
    Host B – Interface IP “30.30.30.2” (Higher), priority “150” (same), vrrp instance “1” (same).

    Description – Currently we have Host B as “Master” and Host A as “Backup” because Host B’s Interface IP is higher than Host A and their priorities are same.

    We add two entries in the VIP (virtual IP address) block in “Keepalived.conf” file of both the hosts i.e. 30.30.30.3 & 30.30.30.1 (Host A interface IP).

    As per the config if we reload keepalived on both the hosts, Host A interface should become the master as this will be our IP address Owner case. But this does not happens.

    Expected behavior – Host A Interface should become Master as Interface IP address is same as primary IP (IP address owner case).

    Current behavior – Host B interface acts as Master. Host A state is fluctuating between Master & Backup and its Not consistent.
    Secondly when we stop the KA on host A in this scenario it removes the Virtual IP address from the interfaces, since in this case we have added the real interface IP address here thus that IP address is also removed (from the actual interface). This will result in even bad state for a system.

    Reply
  11. zoe_thomas on

    Step 3: Configure Keepalived on LB1.
    Add below configuration in keepalived configuration file on LB1.

    vrrp_instance VI_1 {
    interface eth1
    state MASTER
    virtual_router_id 10
    priority 101 # 101 on master, 100 on backup
    virtual_ipaddress {
    192.168.10.121
    }
    }
    Step 4: Configure Keepalived on LB2.
    Add below configuration in keepalived configuration file on LB2.

    vrrp_instance VI_1 {
    interface eth1
    state MASTER
    virtual_router_id 10
    priority 100 # 101 on master, 100 on backup
    virtual_ipaddress {
    192.168.10.121
    }
    }
    – See more at: http://tecadmin.net/ip-failover-setup-using-keepalived-on-centos-redhat-6/#sthash.32WPFgEa.dpuf

    look lb2 ,state MASTER change state backup?

    Reply
    • Rahul on

      Hi Thomas,

      Thanks to point. But there will not be any issue if you use as like this. Because of state depends on priority. If your state is MASTER but your priority is lower than the router with BACKUP, you will lose the MASTER state…

      Reply

Exit mobile version