Keepalived is used for IP failover between two servers. It facilities for load balancing and high-availability to Linux-based infrastructures. It works on VRRP ( Virtual Router Redundancy Protocol ) protocol. We have running two load balance servers using HAProxy and now we need to implement VRRP between both servers. This tutorial will help you to configure KeepAlived, Use this tutorial to configure HAProxy on both servers.
Network Scenario:
-
1. LB1 Server: 192.168.10.111 ( eth1 )
2. LB2 Server: 192.168.10.112 ( eth1 )
3. Virtual IP: 192.168.10.121
Now we are implementing ip failover setup between LB1 and LB2 servers.
Graphical representation of Fail over Setup:
Step 1 – Install Required Packages
Use the following command to install required packages to configure Keepalived on the server.
# yum install gcc kernel-headers kernel-devel
Step 2 – Install Keepalived
Keepalived is available in centos base repository. Install it using yum command line tool.
# yum install keepalived
Keepalived configuration File: /etc/keepalived/keepalived.conf
Step 3 – Configure Keepalived on LB1
Edit Keepalived configuration file on LB1 and add following configuration. Update all red highlighted values with your network and system configuration.
! Configuration File for keepalived global_defs { notification_email {[email protected] [email protected] } notification_email_from[email protected] smtp_server localhost smtp_connect_timeout 30 } vrrp_instance VI_1 { state MASTER interfaceeth1 virtual_router_id 51 priority 101 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress {192.168.10.121 } }
Step 4 – Configure KeepAlived on LB2
Edit Keepalived configuration file on LB2 and add following configuration. While making changes in LB2 configuration file, make sure to set priority values to lower than LB1. For example below configuration is showing 100 priority value than LB1 has it 101.
! Configuration File for keepalived global_defs { notification_email {[email protected] [email protected] } notification_email_from[email protected] smtp_server localhost smtp_connect_timeout 30 } vrrp_instance VI_1 { state MASTER interfaceeth1 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress {192.168.10.121 } }
2. virtual_router_id should be same on both LB1 and LB2 servers.
3. By default single vrrp_instance support up to 20 virtual_ipaddress. In order to add more addresses you need to add more vrrp_instance
Step 5 – Start KeepAlived
Start KeepAlived service using the following command and also configure to autostart on system boot.
# service keepalived start # chkconfig keepalived on
Step 6 – Check Virtual IPs
By default virtual IP will be assigned to master server, In case of master gets down, it will automatically assign to the slave server. Use the following command to show assigned virtual IP on the interface.
# ip addr show eth1
Sample output
2: eth1:mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000 link/ether 00:0c:29:6f:ed:60 brd ff:ff:ff:ff:ff:ff inet 192.168.10.111/24 brd 192.168.1.255 scope global eth1 inet 192.168.10.121/32 scope global eth1 inet6 fe80::20c:29ff:fe6f:ed60/64 scope link valid_lft forever preferred_lft forever
Step 7 – Verify IP Failover
Follow the below process to test keepalived failover is working correctly.
- Shutdown master server ( LB1 ) and check if ips are automatically assigned to the slave server.
# ip addr show eth1
- Now start LB1 and stop slave server ( LB2 ). IPs will automatically be assigned to master server.
# ip addr show eth1
- Watch log files to insure its working
# tailf /var/log/messages
Sample Output
Mar 19 17:30:24 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) Transition to MASTER STATE Mar 19 17:30:25 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) Entering MASTER STATE Mar 19 17:30:25 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) setting protocol VIPs. Mar 19 17:30:25 localhost Keepalived_healthcheckers[6957]: Netlink reflector reports IP 192.168.10.121 added Mar 19 17:30:25 localhost avahi-daemon[1407]: Registering new address record for 192.168.10.121 on eth1.IPv4. Mar 19 17:30:25 localhost Keepalived_vrrp[6958]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for
I hope this article will help to setup IP failover between two load balance servers.
16 Comments
Hi Rahul,
I have below setup for mysql:
Machine #A – 192.168.10.1
Machine #B – 192.168.10.2
VIP between Machine #A and Machine #B is 192.168.10.3
Let say keeapalived is master in Machine #A, possible the application not to use virtual ip to connect to the destination. Sample example is connecting to mysql:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.10.3:32444 192.168.10.3:3306 ESTABLISHED
tcp 0 0 192.168.10.3:32442 192.168.10.3:3306 ESTABLISHED
What I’m expecting is as below:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.10.1:32444 192.168.10.3:3306 ESTABLISHED
tcp 0 0 192.168.10.1:32442 192.168.10.3:3306 ESTABLISHED
Anyway I can achieve above?
shall i give public IP (142.93.12.140) instead of Local IP(192.168.10.121)..??
Yes, you can use public IP.
will it work in different network ?
waiting for your reply
Yes
Why install gcc + kernel headers/devel ?
Thank you very much for this tutorial. I am a beginner and I found it really easy to follow. Cheers!
Hi,
First of all this is a post with good explanation. I have a couple of questions regarding sync between servers. As we all know failover/redundancy servers needs to be synchronised to have consistent data across all the servers. One of the site recommended to run rsync to sync data between master and secondary servers periodically. At the worst case consistent data may not be available on a particular server at that point of time when sync hasn’t happened yet. So I was wondering is it possible to feed data from the firewall itself to multiple systems in parallel so that there wont be any sync issues? Kindly let me know. Thanks
Hi Jaffar,
vrrp_script is a script which will be running every 2 s (interval 2), so i dont think that you want that every 2s “killall -0 haproxy” will be executed. Thus i think you should run a command that check the state of your proxy like ps or a special command of your proxy. Anyway, i think that the fail over happens because if the secondary server is the first which runs keepalived, he will runs “killall -0 haproxy”, and increment the priority by 2 (weight 2), so the priority becomes 102 > 101 (the priority of the and increment the priority by 2 (weight 2)primary server) => FAILOVER, after that as the same way, the primary server runs “killall -0 haproxy” and increment the priority by 2 (weight 2), so the priority becomes 103 > 102 => FAILOVER ….
Hi Guys,
We are having below config for keepalived to have two HAproxy servers shared the VIP:
global_defs {
router_id haproxy0
}
vrrp_script haproxy {
script “killall -0 haproxy”
interval 2
weight 2
}
vrrp_instance 243 {
virtual_router_id 243
advert_int 1
priority 101
state BACKUP
nopreempt
interface eth0
virtual_ipaddress {
192.168.40.41 dev eth0
}
track_script {
haproxy
}
}
Pretty much same config is on the other HAProxy server with exception to the router_id and priority (100).
Teh config works well in normal scenarios. Once in a while we are observing that Secondary sever tries to claim the VIP (may be the advert messages are not reaching it) and actually claims it and releases it again in few seconds after it get the high priority advert message from Primary Server which has been holding the VIP. The other interesting part is neither HAProxy nor Keepalived ever went down but failover and failback has happend. The servers are under same network. Below are the logs:
Primary server:
Dec 9 06:25:02 localhost tripwire[10745]: Integrity Check Failed: File could not be opened.
Dec 9 12:26:37 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
Dec 9 12:26:37 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
Dec 9 12:26:38 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
Dec 9 12:26:38 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
Dec 9 12:26:39 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
Dec 9 12:26:39 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Received lower prio advert, forcing new election
Dec 9 12:26:40 localhost Keepalived_vrrp[20018]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
Secondary server:
Dec 9 06:25:03 localhost tripwire[31150]: Integrity Check Failed: File could not be opened.
Dec 9 12:25:06 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Transition to MASTER STATE
Dec 9 12:25:07 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Entering MASTER STATE
Dec 9 12:25:07 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) setting protocol VIPs.
Dec 9 12:25:07 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Sending gratuitous ARPs on eth0 for 192.168.40.41
Dec 9 12:25:07 localhost Keepalived_healthcheckers[20741]: Netlink reflector reports IP 192.168.40.41 added
Dec 9 12:25:09 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Received higher prio advert
Dec 9 12:25:09 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) Entering BACKUP STATE
Dec 9 12:25:09 localhost Keepalived_vrrp[20742]: VRRP_Instance(243) removing protocol VIPs.
Dec 9 12:25:09 localhost Keepalived_healthcheckers[20741]: Netlink reflector reports IP 192.168.40.41 removed
Any help around this would be appreciated.
Thanks, its very helpful tutorials
IP owner case is not working in keepalive and dont know where to use primary ip address ? can anyone see config file and below explanation and let me know is it correct config for owner case or what ????
Keepalived config for Master host(HOST A) which has its interface IP in the VIP block ::interface ip of (eth1 30.30.30.1/24) sample config::
eth1 is 30.30.30.1/24
keepalived.conf file::
vrrp_instance VI {
state MASTER
interface eth1
virtual_router_id 1
advert_int 1
priority 150
authentication {
auth_type NONE
auth_pass 1112
}
virtual_ipaddress {
30.30.30.1/24
30.30.30.3/24
}
}
In this file there is no field to specifically save the primary address & all configured addresses goes to the VIP list block. Can you please let me know where i can use primary ip address will be configured in above configuration file.
Moreover if we add Primary address (which is always the interface address as ip owner case says) to the VIP block in KA: we see the following issue::
We have two hosts which are participating in a VRRP instance. They have same priorities(above see config)::
Host A – Interface IP “30.30.30.1” (Lower), priority “150” (same), vrrp instance “1” (same).
Host B – Interface IP “30.30.30.2” (Higher), priority “150” (same), vrrp instance “1” (same).
Description – Currently we have Host B as “Master” and Host A as “Backup” because Host B’s Interface IP is higher than Host A and their priorities are same.
We add two entries in the VIP (virtual IP address) block in “Keepalived.conf” file of both the hosts i.e. 30.30.30.3 & 30.30.30.1 (Host A interface IP).
As per the config if we reload keepalived on both the hosts, Host A interface should become the master as this will be our IP address Owner case. But this does not happens.
Expected behavior – Host A Interface should become Master as Interface IP address is same as primary IP (IP address owner case).
Current behavior – Host B interface acts as Master. Host A state is fluctuating between Master & Backup and its Not consistent.
Secondly when we stop the KA on host A in this scenario it removes the Virtual IP address from the interfaces, since in this case we have added the real interface IP address here thus that IP address is also removed (from the actual interface). This will result in even bad state for a system.
Working perfectly… Thanks
Step 3: Configure Keepalived on LB1.
Add below configuration in keepalived configuration file on LB1.
vrrp_instance VI_1 {
interface eth1
state MASTER
virtual_router_id 10
priority 101 # 101 on master, 100 on backup
virtual_ipaddress {
192.168.10.121
}
}
Step 4: Configure Keepalived on LB2.
Add below configuration in keepalived configuration file on LB2.
vrrp_instance VI_1 {
interface eth1
state MASTER
virtual_router_id 10
priority 100 # 101 on master, 100 on backup
virtual_ipaddress {
192.168.10.121
}
}
– See more at: http://tecadmin.net/ip-failover-setup-using-keepalived-on-centos-redhat-6/#sthash.32WPFgEa.dpuf
look lb2 ,state MASTER change state backup?
Hi Thomas,
Thanks to point. But there will not be any issue if you use as like this. Because of state depends on priority. If your state is MASTER but your priority is lower than the router with BACKUP, you will lose the MASTER state…