lvs-dr安装笔记-by 2hei
一、准备测试环境，如下图：
# ________
# | |
# | client |
# |________|
# | CIP=eth0 192.168.1.117
# | GW=192.168.1.1
# | __________
# | | |
# |——| director |
# | |__________|
# | VIP=eth0:199 192.168.1.199/32
# | DIP=eth0 192.168.1.191
# |
# |
# ——————-
# | |
# | |
# ______________ ______________
# | | | |
# | realserver1 | | realserver2 |
# |______________| |______________|
# RIP1=eth0 RIP2=eth0
# 192.168.1.192 192.168.1.193
最少需要三台linux机器。
二、软件环境
操作我使用的是 Linux Ad4 kernel：2.6.9-5.ELsmp
需要的软件：
Net-DNS-0.60.tar.gz http://www.perl.com
configure-lvs_0.9.4.tar.gz http://www.austintek.com/LVS/configure-lvs/configure-lvs_0.9.4.tar.gz
heartbeat-1.0.4.tar.gz
httpd-2.2.4.tar.bz2
ipvsadm-1.24.tar.gz http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz
mon-0.99.2.tar.gz
linux-2.6.10.tar.gz http://www.kernel.org/
hidden-2.6.9-1.diff.txt 从http://www.ssi.bg/~ja/ 得到hidden修正包,不同的核心使用相应的版本.请参考下表
Patch Linux 2.4 Created
hidden-2.4.28-1.diff 2.4.28 – 2.4.30 November 18, 2004
hidden-2.4.26-1.diff 2.4.26 – 2.4.27 February 28, 2004
hidden-2.4.25-1.diff 2.4.25 February 19, 2004
hidden-2.4.20pre10-1.diff 2.4.20pre10 – 2.4.24 October 12, 2002
hidden-2.4.19pre5-1.diff 2.4.19pre5 – 2.4.20pre9 April 7, 2002
hidden-2.4.5-1.diff 2.4.5 – 2.4.19pre4 June 2, 2001
hidden-2.4.4-1.diff 2.4.4 April 29, 2001
Patch Linux 2.6 Created
hidden-2.6.9-1.diff 2.6.9 – 2.6.11 October 19, 2004
hidden-2.6.4-1.diff 2.6.4 – 2.6.8 March 12, 2004
hidden-2.6.3-1.diff 2.6.3 February 19, 2004
hidden-2.5.67-1.diff 2.5.67 – 2.6.2 April 9, 2003
重新编译内核，修正arp problem
####################################################################
注意事项：
要用干净的内核，不要发布版的，或者商业版本
一定要使用redhat网站上的内核，否则，哈哈
1. Kernel,IPVS,IPVSadm,ARP hidden之间的版本必须对应。
2. 自己编译内核时，从http://www.kernel.org/下载标准内核源文件，不要使用发行版的内核源文件。
3. Kernel 2.4.28和2.6.10及以上版本已内置IPVS，有些Linux发行版也在其内核里编译了IPVS。
4. ARP hidden可以用arp_ignore/arp_announce或者arptables_jf代替
patch -p1 < hidden-2.6.9-1.diff
三、具体配置顺序：
1、从http://www.kernel.org/下载新内核2.6.10
解压到 /usr/src/
内核文件夹重命名为linux
2、打ARP补丁
cd /usr/src/linux
patch -p1 < hidden-2.6.9-1.diff
3、编译内核：
编译2.6.10的内核 时
#
#make mrproper
#make menuconfig
找到lvs的配置，基本不用修改。
#make clean
#make
#make modules_install
#make install
4、使用新内核：
vi /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda6
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux AS (2.6.10)
root (hd0,0)
kernel /vmlinuz-2.6.10 ro root=LABEL=/
initrd /initrd-2.6.10.img
title Red Hat Enterprise Linux AS (2.6.9-5.ELsmp)
root (hd0,0)
kernel /vmlinuz-2.6.9-5.ELsmp ro root=LABEL=/
initrd /initrd-2.6.9-5.ELsmp.img
title Red Hat Enterprise Linux AS-up (2.6.9-5.EL)
root (hd0,0)
kernel /vmlinuz-2.6.9-5.EL ro root=LABEL=/
initrd /initrd-2.6.9-5.EL.img
5、重新启动系统，使用新的内核
#uname -a
Linux lvs-director 2.6.10 #1 SMP Mon Oct 15 14:22:45 EDT 2007 i686 i686 i386 GNU/Linux
6、安装ipvsadm软件
cd ipvsadm-1.24
make
make install
7、配置 director 和realserver
下载configure文件，运行时需要perl的支持。
我在使用时提示出
Can’t locate Net/DNS.pm in @INC (@INC contains:
说明perl缺少了 Net::DNS模块
#perl -MCPAN -e “shell”
按照提示进行即可
cpan>install Net::DNS
我在安装perl的dns模块没有成功，
t/08-online…………….NOK 57# Failed test (t/08-online.t at line 109)
Failed Test Stat Wstat Total Fail Failed List of Failed
——————————————————————————-
t/08-online.t 4 1024 93 6 6.45% 57-58 92-93
3 tests and 20 subtests skipped.
Failed 1/25 test scripts, 96.00% okay. 4/1181 subtests failed, 99.66% okay.
make: *** [test_dynamic] Error 255
/usr/bin/make test — NOT OK
Running make install
make test had returned bad status, won’t install without force
超时，一直没有make test成功
没有办法，我从http://www.perl.com 上下载了perl的Net-dns模块Net-DNS-0.60.tar.gz单独安装
tar -zxvf Net-DNS-0.60.tar.gz
#cd Net-DNS-0.60
#perl Makefile.PL
#make
#make test
#make install
cp -r lib/Net/DNS* /usr/lib/perl5/5.8.5/Net
8、下载configure脚本configure-lvs_0.9.4.tar.gz http://www.austintek.com/LVS/configure-lvs/configure-lvs_0.9.4.tar.gz，然后执行配置文件
cd configure-lvs_0.9.4
./configure lvs_dr.conf
运行的时候先要执行loader端的脚本。
#以下是在director端的配置
#lvs_dr.conf
#format is (C) 2000-2001 Joseph Mack jmack@wm7d.net
#edited by yu2hei@gmail.com
#released under GPL.
#single network director VS-DR.
#VIP,real-servers on 192.168.1.0/24 on eth0
#There is no router in this setup, all hosts are on the same network
#————————————————————
#
#IPs can be resolvable names (e.g. in /etc/hosts)
#Here’s my /etc/hosts
#
#VIP, the IP the clients connect to.
#192.168.1.199 lvs
#
#DIP is the IP on the director in the real-server network
#192.168.1.191 dip
#
#GW, my test client
#GW 192.168.1.1
#192.168.1.117 client
#
#real-servers
#192.168.1.192 RS1
#192.168.1.193 RS2
#————————————————————-
#setup:
#This script installs the VIP.
#The CIP, DIP and RIPs must be already installed,
#machines must be connected and be able to ping each other.
#CIP, RIPs usually are primary IPs on an interface.
#VIP, DIP are secondary (alias) IPs.
#
#
# ________
# | |
# | client |
# |________|
# | CIP=eth0 192.168.1.117
# |
# | __________
# | | |
# |——| director |
# | |__________|
# | VIP=eth0:199 192.168.1.199/32
# | DIP=eth0 192.168.1.191
# |
# |
# ——————-
# | |
# | |
# ______________ ______________
# | | | |
# | realserver1 | | realserver2 |
# |______________| |______________|
# RIP1=eth0 RIP2=eth0
# 192.168.1.192 192.168.1.193
#
# all realservers
# VIP=lo:199=192.168.1.199 #
#
#———-lvs_dr.conf—————————————-
—— LVSCONF_FORMAT=1.1
—— LVS_TYPE=VS_DR
—— INITIAL_STATE=on
—— CLEAR_IPVS_TABLES=yes
—— #VIP line format – device[:alias] IP netmask broadcast
—— #To help avoid namespace collisions with other VIPs, I set alias=last number of VIP (here 110).
—— #note: for VS-DR, VS-Tun, the IP is in a /32 network
—— VIP=eth0:110 lvs 255.255.255.255 lvs
—— #DIP line format – device[:alias] IP network netmask broadcast
—— DIP=eth0:9 dip 192.168.1.0 255.255.255.0 192.168.1.255
—— #no DIRECTOR_GW for VS-DR or VS-Tun
—— #DIRECTOR_GW=
—— #SERVICE line format – proto port scheduler IP[,weight] [IP[,weight]]
—— SERVICE=t telnet rr RS1 RS2
—— #SERVICE=t ftp rr RS1,1 RS2,2
—— SERVICE=t http rr RS1 RS2
—— SERVER_VIP_DEVICE=lo:110
—— SERVER_NET_DEVICE=eth0
—— #SERVER_GW – packets with src_addr=VIP, dst_addr=0/0 are sent to SERVER_GW
—— #to be forwarded to the outside world.
—— #For standard VS-DR,VS-Tun, this must _NOT_ be the director.
—— #For Julian’s martian modification (see the HOWTO), it will be the director.
—— #If you don’t know about the martian modification, you aren’t using it.
—— #The script will not neccesarily set up the SERVER_GW as the real-servers’s default gw.
—— SERVER_GW=client
#———-end lvs_dr.conf————————————
#——- my lvs_dr.conf in 192.168.1.191——————
LVSCONF_FORMAT=1.1
LVS_TYPE=VS_DR
INITIAL_STATE=on
CLEAR_IPVS_TABLES=yes
#VIP line format – device[:alias] IP netmask broadcast
#To help avoid namespace collisions with other VIPs, I set alias=last number of VIP (here 110).
#note: for VS-DR, VS-Tun, the IP is in a /32 network
VIP=eth0:199 192.168.1.199 255.255.255.255 192.168.1.199
#DIP line format – device[:alias] IP network netmask broadcast
DIP=eth0 192.168.1.191 192.168.1.0 255.255.255.0 192.168.1.255
#no DIRECTOR_GW for VS-DR or VS-Tun
#DIRECTOR_GW=
#SERVICE line format – proto port scheduler IP[,weight] [IP[,weight]]
#SERVICE=t telnet rr 192.168.1.192 192.168.1.193
#SERVICE=t ftp rr 192.168.1.192,1 192.168.1.193,2
SERVICE=t http rr 192.168.1.192 192.168.1.193
SERVER_VIP_DEVICE=lo:199
SERVER_NET_DEVICE=eth0
#SERVER_GW – packets with src_addr=VIP, dst_addr=0/0 are sent to SERVER_GW
#to be forwarded to the outside world.
#For standard VS-DR,VS-Tun, this must _NOT_ be the director.
#For Julian’s martian modification (see the HOWTO), it will be the director.
#If you don’t know about the martian modification, you aren’t using it.
#The script will not neccesarily set up the SERVER_GW as the real-servers’s default gw.
SERVER_GW=192.168.1.1
#——- end of my lvs_dr.conf in 192.168.1.191——————
9、在lvs-dr端配置启动脚本
#—————mini-rc.lvs_dr-director in ldr 192.168.1.191————————#
#!/bin/bash
#mini_rc_lvs_dr.sh
#set ip_forward OFF for lvs-dr director (1 on, 0 off)
#(there is no forwarding in the conventional sense for LVS-DR)
cat /proc/sys/net/ipv4/ip_forward
echo “0” >/proc/sys/net/ipv4/ip_forward
#director is not gw for realservers: leave icmp redirects on
echo ‘setting icmp redirects (1 on, 0 off) ‘
echo “1” >/proc/sys/net/ipv4/conf/all/send_redirects
cat /proc/sys/net/ipv4/conf/all/send_redirects
echo “1” >/proc/sys/net/ipv4/conf/default/send_redirects
cat /proc/sys/net/ipv4/conf/default/send_redirects
echo “1” >/proc/sys/net/ipv4/conf/eth0/send_redirects
cat /proc/sys/net/ipv4/conf/eth0/send_redirects
#add ethernet device and routing for VIP 192.168.1.199
/sbin/ifconfig eth0:110 192.168.1.199 broadcast 192.168.1.199 netmask 255.255.255.255
/sbin/route add -host 192.168.1.199 dev eth0:199
#listing ifconfig info for VIP 192.168.1.199
/sbin/ifconfig eth0:199
#check VIP 192.168.1.199 is reachable from self (director)
/bin/ping -c 1 192.168.1.199
#listing routing info for VIP 192.168.1.199
/bin/netstat -rn
#setup_ipvsadm_table
#clear ipvsadm table
/sbin/ipvsadm -C
#installing LVS services with ipvsadm
#add telnet to VIP with round robin scheduling
/sbin/ipvsadm -A -t 192.168.1.199:http -s rr
#forward telnet to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t 192.168.1.199:http -r 192.168.1.192 -g -w 1
#check realserver reachable from director
ping -c 1 192.168.1.192
#forward telnet to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t 192.168.1.199:http -r 192.168.1.193 -g -w 1
#check realserver reachable from director
ping -c 1 192.168.1.193
#displaying ipvsadm settings
/sbin/ipvsadm
#not installing a default gw for LVS_TYPE vs-dr
#—————mini-rc.lvs_dr-director————————#
#mini-rc.lvs_dr-director 简单的配置 in 192.168.1.191
#/bin/sh
# lv-dr.sh in 192.168.1.191 in simple
# set ip_forward OFF for vs-dr director (1 on, 0 off)
echo 0 > /proc/sys/net/ipv4/ip_forward
echo “1” > /proc/sys/net/ipv4/conf/all/send_redirects
echo “1” > /proc/sys/net/ipv4/conf/default/send_redirects
echo “1” > /proc/sys/net/ipv4/conf/eth0/send_redirects
ifconfig eth0:0 192.168.1.199 netmask 255.255.255.255 broadcast 192.168.1.255 up
route add -host 192.168.1.199 dev eth0:199
ipvsadm -C
ipvsadm -A -t 192.168.1.199:80 -s rr
# Set Real Server
ipvsadm -a -t 192.168.1.199:80 -r 192.168.0.192 -g
ipvsadm -a -t 192.168.1.199:80 -r 192.168.0.193 -g
ipvsadm -Ln
==============================
在loader上执行
[root@lvs-director Net]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.199:80 rr
-> 192.168.1.193:80 Route 1 0 0
-> 192.168.1.192:80 Route 1 0 0
10、realserver的配置
#以下是realserver 端的配置及脚本
#多个realserver的配置都是一样
#realserver的其他前提配置： 需要解决arp的问题和重新编译内核（跟lvs-dr的），配置好http服务。
#!/bin/bash
#———-mini-rc.lvs_dr-realserver——————
#installing default gw 192.168.1.1 for vs-dr
/sbin/route add default gw 192.168.1.1
#showing routing table
/bin/netstat -rn
#checking if DEFAULT_GW 192.168.1.1 is reachable
ping -c 1 192.168.1.1
#set_realserver_ip_forwarding to OFF (1 on, 0 off).
echo “0” >/proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/ip_forward
#looking for DIP 192.168.1.191
ping -c 1 192.168.1.191
#looking for VIP (will be on director)
ping -c 1 192.168.1.199
#install_realserver_vip
/sbin/ifconfig lo:199 192.168.1.199 broadcast 192.168.1.199 netmask 0xffffffff up
#ifconfig output
/sbin/ifconfig lo:199
#installing route for VIP 192.168.1.199 on device lo:199
/sbin/route add -host 192.168.1.199 dev lo:199
#listing routing info for VIP 192.168.1.199
/bin/netstat -rn
#hiding interface lo:199, will not arp
echo “1” >/proc/sys/net/ipv4/conf/all/hidden
cat /proc/sys/net/ipv4/conf/all/hidden
echo “1” >/proc/sys/net/ipv4/conf/lo/hidden
cat /proc/sys/net/ipv4/conf/lo/hidden
===========================
11、测试
client使用浏览器访问： http://192.168.1.199
会有如下的页面出现，如图（需要在realserver端配置httpd服务，设置好不同的显示页面）：
使用不同浏览器或者间隔段时间测试，显示不同的页面则说明配置成功。
安装中仔细阅读 LVS-mini-HOWTO 基本的问题都可解决。
一般来说，为了是负载均衡能够有高可用性，loader需要使用两台，配置heartbeat，等配置成功了弄上来，一起分享。
本文固定链接: https://www.2hei.net/2007/10/16/lvs-dr%e5%ae%89%e8%a3%85%e7%ac%94%e8%ae%b0-by-2hei/ | 2hei.net
最活跃的读者