负载均衡之lvs简单测试搭建及keepalived高可用实现

7年前 (2017-09-29) gtj linux, lvs, lvs+keepalived 0评论 已收录 1688℃

架构规划：

10.20.23.145 LVS1
10.20.23.144 LVS2
10.20.23.154 RS1
10.20.23.250 RS2
10.20.23.146 VIP

一.安装lvs

1.安装前准备命令：

cat /etc/redhat-release 
uname -rm
lsmod | grep ip_vs
ln -s /usr/src/kernels/2.6.32-431.el6.x86_64/ /usr/src/linux
[root@LVS1 ~]# cat /etc/redhat-release 
CentOS release 6.5 (Final)
[root@LVS1 ~]# uname -r
2.6.32-431.el6.x86_64
[root@LVS1 ~]# lsmod | grep ip_vs      ##lsmod查看内核的模块     ip_vs是LVS的调度器
[root@LVS1 ~]# ls -ld /usr/src/kernels/`uname -r`/    
drwxr-xr-x. 22 root root 4096 1月   3 16:55 /usr/src/kernels/2.6.32-431.el6.x86_64/
[root@LVS1 ~]# ln -s /usr/src/kernels/2.6.32-431.el6.x86_64/ /usr/src/linux
[root@LVS1 ~]# ll /usr/src/
总用量 8
drwxr-xr-x. 2 root root 4096 9月  23 2011 debug
drwxr-xr-x. 3 root root 4096 1月   3 16:55 kernels
lrwxrwxrwx  1 root root   39 4月  17 17:30 linux -> /usr/src/kernels/2.6.32-431.el6.x86_64/
###若没有/usr/src/kernels/2.6.32-431.el6.x86_64/路径，很可能是因为缺少kernel-2.6.32-431.el6.x86_64软件包，可通过yum install kernel-devel -y安装。。

2.安装LVS

cd /home/oldboy/tools
wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.26.tar.gz
tar zxf ipvsadm-1.26.tar.gz
cd ipvsadm-1.26 
yum install libnl* popt* -y
make
echo $?
make install
lsmod | grep ip_vs
modprobe ip_vs  （/sbin/ipvsadm）
lsmod | grep ip_vs
[root@LVS1 ipvsadm-1.26]# lsmod | grep ip_vs
[root@LVS1 ipvsadm-1.26]# modprobe ip_vs #/sbin/ipvsadm
[root@LVS1 ipvsadm-1.26]# lsmod | grep ip_vs
ip_vs                 125220  0 
libcrc32c               1246  1 ip_vs
ipv6                  317340  265 ip_vs
##make时会遇到如下错误：
libipvs.c:1071: error: 'NLM_F_DUMP' undeclared (first use in this function)
libipvs.c:1072: error: too many arguments to function 'ipvs_nl_send_message'
make[1]: *** [libipvs.o] Error 1
make[1]: Leaving directory `/home/oldboy/tools/ipvsadm-1.26/libipvs'
make: *** [libs] Error 2
解决办法：yum install libnl* popt* -y，重新make

3.手动配置LVS负载均衡服务

配置LVS虚拟IP（VIP）

ifconfig eth0:0 10.20.23.146 netmask 255.255.255.0 up   #==>别名的方式
ifconfig eth0:0 10.20.23.146/24 up   ##==>简便写法
route add -host 10.20.23.146 dev eth0    ##==>添加主机路由，也可不加此行
[root@LVS1 ipvsadm-1.26]# ifconfig eth0:0 10.20.23.146/24 up    
[root@LVS1 ipvsadm-1.26]# route add -host 10.20.23.146 dev eth0
[root@LVS1 ipvsadm-1.26]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.20.23.146    0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.20.23.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
0.0.0.0         10.20.23.160    0.0.0.0         UG    0      0        0 eth0

4.手工执行配置添加LVS服务并增加两台RS

ipvsadm --help
ipvsadm -C   ##==>清空所有
ipvsadm --set 30 5 60    #==>设置超时，tcp  tcpfin   udp
ipvsadm -A -t 10.20.23.146:80 -s rr -p 20    #==>-A添加一个虚拟服务，-t指定一个vip地址和端口，-s轮询算法，-p会话保持20s
ipvsadm -a -t 10.20.23.146:80 -r 10.20.23.250 -g -w 1
 #==>-a添加节点，-t在哪一个vip上添加节点，-r添加什么节点，-g是dr工作模式（-a：隧道模式，-m：NAT模式），-w权重
ipvsadm -L -n   ##查看

删除方法：

ipvsadm -D -t 10.20.23.146:80 -s rr 
ipvsadm -D -t 10.20.23.146:80           ##删除Vserver用D
ipvsadm -d -t 10.20.23.146:80 -r 10.20.23.250 <==正确             ##删除节点用d
ipvsadm -d -t 10.20.23.146:80 -r 10.20.23.250 -g -w 1 <==不好用

添加方法

[root@LVS1 ipvsadm-1.26]# ipvsadm -C
[root@LVS1 ipvsadm-1.26]# ipvsadm --set 30 5 60
[root@LVS1 ipvsadm-1.26]# ipvsadm -A -t 10.20.23.146:80 -s rr -p 20
[root@LVS1 ipvsadm-1.26]# ipvsadm -a -t 10.20.23.146:80 -r 10.20.23.250 -g -w 1
[root@LVS1 ipvsadm-1.26]# ipvsadm -a -t 10.20.23.146:80 -r 10.20.23.154 -g -w 1   
[root@LVS1 ipvsadm-1.26]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 rr persistent 20
  -> 10.20.23.154:80              Route   1      0          0         
  -> 10.20.23.250:80              Route   1      0          0

5.手工在RS端绑定VIP

所以下面我们就在两台RS服务器上绑定VIP，抑制ARP
命令：

ifconfig lo:0 10.20.23.146/32 up    <==注意子网掩码的特殊
route add -host 10.20.23.146 dev lo
ifconfig
route -n

6.手工在RS端抑制ARP响应

抑制ARP响应的方法如下：

echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore 
echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore 
echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce

7.arp抑制技术参数说明：

中文说明：
arp_ignore integer
/proc/sys/net/ipv4/conf/lo/arp_ignore
定义对目标地址为IP的ARP询问不同的应答模式
0-（默认值）：回应任何网络接口上对本地任何IP地址的arp查询请求；只要网卡上有这个IP。不一定是接网线的网卡，就应答
1- 只回答目标IP地址是来访网络接口本地地址的ARP查询请求；只有接网线的网络接口（物理网卡）才会应答
2-只回答目标IP地址是来访网络接口本地地址的ARP查询请求，且来访IP必须在该网络接口的子网段内。
3-不回应该网络界面的arp请求，而只对设置的唯一和连接地址做出回应；
4-7 - 保留未使用
8-不回应所有（本地地址）的arp查询

arp_announce integer
/proc/sys/net/ipv4/conf/all/arp_announce
对网络接口上，本地IP地址发出的ARP回应，作出相应级别的限制：
确定不同程度的限制，宣布对来自本地源IP地址发出arp请求的接口
0-（默认）在任意网络接口（eth0，eth1，lo）上的任何本地地址
1-尽量避免不在该网络接口子网段的本地地址做出arp回应。当发起ARP请求的源IP地址是被设置应该经由路由达到此网络接口的时候很有用。此时会检查来访IP是否为所有接口上的子网段内ip之一。如果该来访IP不属于各个网络接口上的子网段内，那么将采用级别2的方式来进行处理。
2-对查询目标使用最适当的本地地址，在此模式下将忽略这个IP数据包的源地址并尝试选择能与该地址通信的本地地址，首要是选择所有的网络接口的子网中外出访问子网中包含该目标IP地址的本地地址。如果没有合适的地址被发现，将选择当前的发送网络接口或其他的有可能接受到该ARP回应的网络接口来进行发送。限制了使用本地的vip地址作为有限的网络接口。

8.LVS端启动LVS脚本：

[root@LVS1 scripts]# cat ipvs_server.sh 
#!/bin/bash
. /etc/init.d/functions

VIP=10.20.23.146
PORT=80
RIP=(
10.20.23.154
10.20.23.250
)
start() {
    ifconfig eth0:0 $VIP/24 up
    route add -host $ViP dev eth0
ipvsadm -C
ipvsadm --set 30 5 60
ipvsadm -A -t $VIP:$PORT -s rr -p 20
for ((i=0;i<${#RIP[*]};i++))
do
ipvsadm -a -t $VIP:$PORT -r ${RIP[$i]} -g -w 1
done
}
stop() {
        ipvsadm -C
        ifconfig eth0:0 $VIP/24 down
        route del -host $ViP dev eth0
}
case "$1" in
    start)
      start
      echo "ipvs is started"
      ;;
    stop)
      stop
      echo "ipvs is stopped"
      ;;
    restart)
      stop
      echo "ipvs is stopped"
      start
      echo "ipvs is started"
      ;;
    *)
       echo "Usage:$0 {start|stop|restart}"
esac

mv  ipvs_server.sh /usr/local/sbin/ipvs
chmod +x ipvs

9.RS端启动脚本：

[root@web02 ~]# cat ipvs_rs.sh 
#!/bin/bash
VIP=(
        10.20.23.146
        10.20.23.147
        10.20.23.148
)

##/etc/rc.d/init.d/functions

case "$1" in
start)
    echo "start LVS of RealServer IP"
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 up
        route add -host ${VIP[$i]} dev $interface
    done
    echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce  
    ;;
stop)
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 down
        route del -host ${VIP[$i]} dev $interface
    done
    echo "SToP LVS of RealServer IP"
    echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
    ;;
*)
    echo "Usage:$0 {start|stop}"
    exit 1
esac

二.使用keepalived管理lvs,实现LVS+keepalived实现负载均衡&高可用。

1.安装keepalived

参考https://bk.devopstack.cn/archives/1081.html

2.开启路由转发

vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
sysctl -p

3.修改keepalived配置文件：

master
cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
  258818040@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 10.0.0.1
   smtp_connect_timeout 30
   router_id LVS_7
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.20.23.146/24
    }
}


virtual_server 10.20.23.146 80 {
	delay_loop 6
	lb_algo wrr
	lb_kind DR
	nat_mask 255.255.255.0
	persistent_timeout 300
	protocol TCP
	#ipvsadm -A -t 10.20.23.146:80 -s wrr -p 20
	real_server 10.20.23.154 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
	real_server 10.20.23.250 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
}
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.145:80 -g -w 1
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.144:80 -g -w 1
backup
修改keepalived配置文件：
cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
  258818040@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 10.0.0.1
   smtp_connect_timeout 30
   router_id LVS_8
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.20.23.146/24
    }
}


virtual_server 10.20.23.146 80 {
	delay_loop 6
	lb_algo wrr
	lb_kind DR
	nat_mask 255.255.255.0
	persistent_timeout 300
	protocol TCP
	#ipvsadm -A -t 10.20.23.146:80 -s wrr -p 20
	real_server 10.20.23.154 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
	real_server 10.20.23.250 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
}
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.145:80 -g -w 1
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.144:80 -g -w 1

4.LVS两台真实服务器客户端即web服务器端要绑定VIP及抑制ARP

cd /usr/local/sbin/
chmod +x ipvs_client
sh ipvs_client start
[root@web02 ~]# cat ipvs_client 
#!/bin/bash
VIP=(
        10.20.23.146
        10.20.23.147
        10.20.23.148
)

##/etc/rc.d/init.d/functions

case "$1" in
start)
    echo "start LVS of RealServer IP"
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 up
        route add -host ${VIP[$i]} dev $interface
    done
    echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce  
    ;;
stop)
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 down
        route del -host ${VIP[$i]} dev $interface
    done
    echo "STOP LVS of RealServer IP"
    #echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
    #echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
    #echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
    #echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
    ;;
*)
    echo "Usage:$0 {start|stop}"
    exit 1
esac

5.测试：

[root@LVS1 ~]# /etc/init.d/keepalived restart
停止 keepalived：                                          [确定]
正在启动 keepalived：                                      [确定]
[root@LVS1 ~]# ip addr | grep 10.20.23.
    inet 10.20.23.145/24 brd 10.20.23.255 scope global eth0
    inet 10.20.23.146/24 scope global secondary eth0
[root@LVS1 ~]# ipvsadm -L -n                 
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0         

[root@LVS2 ~]# /etc/init.d/keepalived restart
停止 keepalived：                                          [确定]
正在启动 keepalived：                                      [确定]
[root@LVS2 ~]# ip addr | grep 10.20.23.
    inet 10.20.23.144/24 brd 10.20.23.255 scope global eth0
[root@LVS2 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0         

[root@LVS1 ~]# /etc/init.d/keepalived stop
停止 keepalived：                                          [确定]
[root@LVS1 ~]# ipvsadm -L -n              
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@LVS1 ~]# ip addr | grep 10.20.23.   
    inet 10.20.23.145/24 brd 10.20.23.255 scope global eth0

[root@LVS2 ~]# ip addr | grep 10.20.23
    inet 10.20.23.144/24 brd 10.20.23.255 scope global eth0
    inet 10.20.23.146/24 scope global secondary eth0
[root@LVS2 ~]# ipvsadm -L -n          
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0      

启动会LVS1就会接管：
[root@LVS1 ~]# /etc/init.d/keepalived start
正在启动 keepalived：                                      [确定]
[root@LVS1 ~]# ip addr | grep 10.20.23.    
    inet 10.20.23.145/24 brd 10.20.23.255 scope global eth0
    inet 10.20.23.146/24 scope global secondary eth0
[root@LVS1 ~]# ipvsadm -L -n               
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0

注：简本配置成功，本文总结有点乱，以后再整理。

AD：让我们在it的道路上越行越远!

如无特殊说明，文章均为本站原创，转载请注明出处
本文标题：负载均衡之lvs简单测试搭建及keepalived高可用实现
本文永久链接地址：https://bk.devopstack.cn/archives/1083.html
本文标签：
博主

这货来去如风，什么鬼都没留下！！！

相关推荐

嗨、骚年、快来消灭0回复。

×
订阅图标按钮
Less is more!!!