负载均衡之lvs简单测试搭建及keepalived高可用实现

3年前 (2017-09-29) gtj linux, lvs, lvs+keepalived 0评论 已收录 273℃

架构规划:

10.20.23.145 LVS1
10.20.23.144 LVS2
10.20.23.154 RS1
10.20.23.250 RS2
10.20.23.146 VIP

一.安装lvs

1.安装前准备命令:

cat /etc/redhat-release 
uname -rm
lsmod | grep ip_vs
ln -s /usr/src/kernels/2.6.32-431.el6.x86_64/ /usr/src/linux
[root@LVS1 ~]# cat /etc/redhat-release 
CentOS release 6.5 (Final)
[root@LVS1 ~]# uname -r
2.6.32-431.el6.x86_64
[root@LVS1 ~]# lsmod | grep ip_vs      ##lsmod查看内核的模块     ip_vs是LVS的调度器
[root@LVS1 ~]# ls -ld /usr/src/kernels/`uname -r`/    
drwxr-xr-x. 22 root root 4096 1月   3 16:55 /usr/src/kernels/2.6.32-431.el6.x86_64/
[root@LVS1 ~]# ln -s /usr/src/kernels/2.6.32-431.el6.x86_64/ /usr/src/linux
[root@LVS1 ~]# ll /usr/src/
总用量 8
drwxr-xr-x. 2 root root 4096 9月  23 2011 debug
drwxr-xr-x. 3 root root 4096 1月   3 16:55 kernels
lrwxrwxrwx  1 root root   39 4月  17 17:30 linux -> /usr/src/kernels/2.6.32-431.el6.x86_64/
###若没有/usr/src/kernels/2.6.32-431.el6.x86_64/路径,很可能是因为缺少kernel-2.6.32-431.el6.x86_64软件包,可通过yum install kernel-devel -y安装。。

2.安装LVS

cd /home/oldboy/tools
wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.26.tar.gz
tar zxf ipvsadm-1.26.tar.gz
cd ipvsadm-1.26 
yum install libnl* popt* -y
make
echo $?
make install
lsmod | grep ip_vs
modprobe ip_vs  (/sbin/ipvsadm)
lsmod | grep ip_vs
[root@LVS1 ipvsadm-1.26]# lsmod | grep ip_vs
[root@LVS1 ipvsadm-1.26]# modprobe ip_vs #/sbin/ipvsadm
[root@LVS1 ipvsadm-1.26]# lsmod | grep ip_vs
ip_vs                 125220  0 
libcrc32c               1246  1 ip_vs
ipv6                  317340  265 ip_vs
##make时会遇到如下错误:
libipvs.c:1071: error: 'NLM_F_DUMP' undeclared (first use in this function)
libipvs.c:1072: error: too many arguments to function 'ipvs_nl_send_message'
make[1]: *** [libipvs.o] Error 1
make[1]: Leaving directory `/home/oldboy/tools/ipvsadm-1.26/libipvs'
make: *** [libs] Error 2
解决办法:yum install libnl* popt* -y,重新make

3.手动配置LVS负载均衡服务

配置LVS虚拟IP(VIP)

ifconfig eth0:0 10.20.23.146 netmask 255.255.255.0 up   #==>别名的方式
ifconfig eth0:0 10.20.23.146/24 up   ##==>简便写法
route add -host 10.20.23.146 dev eth0    ##==>添加主机路由,也可不加此行
[root@LVS1 ipvsadm-1.26]# ifconfig eth0:0 10.20.23.146/24 up    
[root@LVS1 ipvsadm-1.26]# route add -host 10.20.23.146 dev eth0
[root@LVS1 ipvsadm-1.26]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.20.23.146    0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.20.23.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
0.0.0.0         10.20.23.160    0.0.0.0         UG    0      0        0 eth0

4.手工执行配置添加LVS服务并增加两台RS

ipvsadm --help
ipvsadm -C   ##==>清空所有
ipvsadm --set 30 5 60    #==>设置超时,tcp  tcpfin   udp
ipvsadm -A -t 10.20.23.146:80 -s rr -p 20    #==>-A添加一个虚拟服务,-t指定一个vip地址和端口,-s轮询算法,-p会话保持20s
ipvsadm -a -t 10.20.23.146:80 -r 10.20.23.250 -g -w 1
 #==>-a添加节点,-t在哪一个vip上添加节点,-r添加什么节点,-g是dr工作模式(-a:隧道模式,-m:NAT模式),-w权重
ipvsadm -L -n   ##查看

删除方法:

ipvsadm -D -t 10.20.23.146:80 -s rr 
ipvsadm -D -t 10.20.23.146:80           ##删除Vserver用D
ipvsadm -d -t 10.20.23.146:80 -r 10.20.23.250 <==正确             ##删除节点用d
ipvsadm -d -t 10.20.23.146:80 -r 10.20.23.250 -g -w 1 <==不好用

添加方法

[root@LVS1 ipvsadm-1.26]# ipvsadm -C
[root@LVS1 ipvsadm-1.26]# ipvsadm --set 30 5 60
[root@LVS1 ipvsadm-1.26]# ipvsadm -A -t 10.20.23.146:80 -s rr -p 20
[root@LVS1 ipvsadm-1.26]# ipvsadm -a -t 10.20.23.146:80 -r 10.20.23.250 -g -w 1
[root@LVS1 ipvsadm-1.26]# ipvsadm -a -t 10.20.23.146:80 -r 10.20.23.154 -g -w 1   
[root@LVS1 ipvsadm-1.26]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 rr persistent 20
  -> 10.20.23.154:80              Route   1      0          0         
  -> 10.20.23.250:80              Route   1      0          0 

5.手工在RS端绑定VIP

所以下面我们就在两台RS服务器上绑定VIP,抑制ARP
命令:

ifconfig lo:0 10.20.23.146/32 up    <==注意子网掩码的特殊
route add -host 10.20.23.146 dev lo
ifconfig
route -n 

6.手工在RS端抑制ARP响应

抑制ARP响应的方法如下:

echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore 
echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore 
echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce 

7.arp抑制技术参数说明:

中文说明:
arp_ignore integer
/proc/sys/net/ipv4/conf/lo/arp_ignore
定义对目标地址为IP的ARP询问不同的应答模式
0-(默认值):回应任何网络接口上对本地任何IP地址的arp查询请求;只要网卡上有这个IP。不一定是接网线的网卡,就应答
1- 只回答目标IP地址是来访网络接口本地地址的ARP查询请求;只有接网线的网络接口(物理网卡)才会应答
2-只回答目标IP地址是来访网络接口本地地址的ARP查询请求,且来访IP必须在该网络接口的子网段内。
3-不回应该网络界面的arp请求,而只对设置的唯一和连接地址做出回应;
4-7 - 保留未使用
8-不回应所有(本地地址)的arp查询

arp_announce integer
/proc/sys/net/ipv4/conf/all/arp_announce
对网络接口上,本地IP地址发出的ARP回应,作出相应级别的限制:
确定不同程度的限制,宣布对来自本地源IP地址发出arp请求的接口
0-(默认)在任意网络接口(eth0,eth1,lo)上的任何本地地址
1-尽量避免不在该网络接口子网段的本地地址做出arp回应。当发起ARP请求的源IP地址是被设置应该经由路由达到此网络接口的时候很有用。此时会检查来访IP是否为所有接口上的子网段内ip之一。如果该来访IP不属于各个网络接口上的子网段内,那么将采用级别2的方式来进行处理。
2-对查询目标使用最适当的本地地址,在此模式下将忽略这个IP数据包的源地址并尝试选择能与该地址通信的本地地址,首要是选择所有的网络接口的子网中外出访问子网中包含该目标IP地址的本地地址。如果没有合适的地址被发现,将选择当前的发送网络接口或其他的有可能接受到该ARP回应的网络接口来进行发送。限制了使用本地的vip地址作为有限的网络接口。

8.LVS端启动LVS脚本:

[root@LVS1 scripts]# cat ipvs_server.sh 
#!/bin/bash
. /etc/init.d/functions

VIP=10.20.23.146
PORT=80
RIP=(
10.20.23.154
10.20.23.250
)
start() {
    ifconfig eth0:0 $VIP/24 up
    route add -host $ViP dev eth0
ipvsadm -C
ipvsadm --set 30 5 60
ipvsadm -A -t $VIP:$PORT -s rr -p 20
for ((i=0;i<${#RIP[*]};i++))
do
ipvsadm -a -t $VIP:$PORT -r ${RIP[$i]} -g -w 1
done
}
stop() {
        ipvsadm -C
        ifconfig eth0:0 $VIP/24 down
        route del -host $ViP dev eth0
}
case "$1" in
    start)
      start
      echo "ipvs is started"
      ;;
    stop)
      stop
      echo "ipvs is stopped"
      ;;
    restart)
      stop
      echo "ipvs is stopped"
      start
      echo "ipvs is started"
      ;;
    *)
       echo "Usage:$0 {start|stop|restart}"
esac

mv  ipvs_server.sh /usr/local/sbin/ipvs
chmod +x ipvs

9.RS端启动脚本:

[root@web02 ~]# cat ipvs_rs.sh 
#!/bin/bash
VIP=(
        10.20.23.146
        10.20.23.147
        10.20.23.148
)

##/etc/rc.d/init.d/functions

case "$1" in
start)
    echo "start LVS of RealServer IP"
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 up
        route add -host ${VIP[$i]} dev $interface
    done
    echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce  
    ;;
stop)
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 down
        route del -host ${VIP[$i]} dev $interface
    done
    echo "SToP LVS of RealServer IP"
    echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
    ;;
*)
    echo "Usage:$0 {start|stop}"
    exit 1
esac

二.使用keepalived管理lvs,实现LVS+keepalived实现负载均衡&高可用。

1.安装keepalived

参考https://bk.devopstack.cn/archives/1081.html

2.开启路由转发

vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
sysctl -p

3.修改keepalived配置文件:

master
cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
  258818040@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 10.0.0.1
   smtp_connect_timeout 30
   router_id LVS_7
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.20.23.146/24
    }
}


virtual_server 10.20.23.146 80 {
	delay_loop 6
	lb_algo wrr
	lb_kind DR
	nat_mask 255.255.255.0
	persistent_timeout 300
	protocol TCP
	#ipvsadm -A -t 10.20.23.146:80 -s wrr -p 20
	real_server 10.20.23.154 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
	real_server 10.20.23.250 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
}
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.145:80 -g -w 1
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.144:80 -g -w 1
backup
修改keepalived配置文件:
cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
  258818040@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 10.0.0.1
   smtp_connect_timeout 30
   router_id LVS_8
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.20.23.146/24
    }
}


virtual_server 10.20.23.146 80 {
	delay_loop 6
	lb_algo wrr
	lb_kind DR
	nat_mask 255.255.255.0
	persistent_timeout 300
	protocol TCP
	#ipvsadm -A -t 10.20.23.146:80 -s wrr -p 20
	real_server 10.20.23.154 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
	real_server 10.20.23.250 80 {
		weight 1
		TCP_CHECK {
		connect_timeout 8
		nb_get_retry 3
		delay_before_retry 3
		connect_port 80
		}
	}
}
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.145:80 -g -w 1
#ipvsadm -a -t 10.20.23.146 -r 10.20.23.144:80 -g -w 1

4.LVS两台真实服务器客户端即web服务器端要绑定VIP及抑制ARP

cd /usr/local/sbin/
chmod +x ipvs_client
sh ipvs_client start
[root@web02 ~]# cat ipvs_client 
#!/bin/bash
VIP=(
        10.20.23.146
        10.20.23.147
        10.20.23.148
)

##/etc/rc.d/init.d/functions

case "$1" in
start)
    echo "start LVS of RealServer IP"
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 up
        route add -host ${VIP[$i]} dev $interface
    done
    echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore 
    echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce  
    ;;
stop)
    for ((i=0;i<`echo ${#VIP[*]}`;i++))
    do
        interface="lo:`echo ${VIP[$i]}|awk -F . '{print $4}'`"
        /sbin/ifconfig $interface ${VIP[$i]} broadcast ${VIP[$i]} netmask 255.255.255.255 down
        route del -host ${VIP[$i]} dev $interface
    done
    echo "STOP LVS of RealServer IP"
    #echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
    #echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
    #echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
    #echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
    ;;
*)
    echo "Usage:$0 {start|stop}"
    exit 1
esac

5.测试:

[root@LVS1 ~]# /etc/init.d/keepalived restart
停止 keepalived:                                          [确定]
正在启动 keepalived:                                      [确定]
[root@LVS1 ~]# ip addr | grep 10.20.23.
    inet 10.20.23.145/24 brd 10.20.23.255 scope global eth0
    inet 10.20.23.146/24 scope global secondary eth0
[root@LVS1 ~]# ipvsadm -L -n                 
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0         

[root@LVS2 ~]# /etc/init.d/keepalived restart
停止 keepalived:                                          [确定]
正在启动 keepalived:                                      [确定]
[root@LVS2 ~]# ip addr | grep 10.20.23.
    inet 10.20.23.144/24 brd 10.20.23.255 scope global eth0
[root@LVS2 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0         
[root@LVS1 ~]# /etc/init.d/keepalived stop
停止 keepalived:                                          [确定]
[root@LVS1 ~]# ipvsadm -L -n              
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@LVS1 ~]# ip addr | grep 10.20.23.   
    inet 10.20.23.145/24 brd 10.20.23.255 scope global eth0

[root@LVS2 ~]# ip addr | grep 10.20.23
    inet 10.20.23.144/24 brd 10.20.23.255 scope global eth0
    inet 10.20.23.146/24 scope global secondary eth0
[root@LVS2 ~]# ipvsadm -L -n          
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0      

启动会LVS1就会接管:
[root@LVS1 ~]# /etc/init.d/keepalived start
正在启动 keepalived:                                      [确定]
[root@LVS1 ~]# ip addr | grep 10.20.23.    
    inet 10.20.23.145/24 brd 10.20.23.255 scope global eth0
    inet 10.20.23.146/24 scope global secondary eth0
[root@LVS1 ~]# ipvsadm -L -n               
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.23.146:80 wrr persistent 300
  -> 10.20.23.154:0               Route   1      0          0         
  -> 10.20.23.250:0               Route   1      0          0 

注:简本配置成功,本文总结有点乱,以后再整理。

博主

这货来去如风,什么鬼都没留下!!!

相关推荐

嗨、骚年、快来消灭0回复。

×
订阅图标按钮
Less is more!!!