当前位置: 首页 > others > 正文

lvs-dr安装笔记-by 2hei

一、准备测试环境,如下图:

#               ________

#              |        |

#              | client |

#              |________|

#                  | CIP=eth0 192.168.1.117

#                  | GW=192.168.1.1

#                  |       __________

#                  |      |          |

#                  |——| director |

#                  |      |__________|

#                  |  VIP=eth0:199 192.168.1.199/32

#                  |  DIP=eth0   192.168.1.191

#                  |

#                  |

#         ——————-

#         |                 |

#         |                 |

#   ______________    ______________

#  |              |  |              |

#  | realserver1  |  | realserver2  |

#  |______________|  |______________|

#     RIP1=eth0         RIP2=eth0

#    192.168.1.192       192.168.1.193

最少需要三台linux机器。

二、软件环境

操作我使用的是 Linux Ad4 kernel:2.6.9-5.ELsmp

需要的软件:

Net-DNS-0.60.tar.gz   http://www.perl.com

configure-lvs_0.9.4.tar.gz  http://www.austintek.com/LVS/configure-lvs/configure-lvs_0.9.4.tar.gz

heartbeat-1.0.4.tar.gz

httpd-2.2.4.tar.bz2

ipvsadm-1.24.tar.gz http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz

mon-0.99.2.tar.gz

linux-2.6.10.tar.gz http://www.kernel.org/

hidden-2.6.9-1.diff.txt  从http://www.ssi.bg/~ja/ 得到hidden修正包,不同的核心使用相应的版本.请参考下表

Patch        Linux 2.4        Created

hidden-2.4.28-1.diff         2.4.28 – 2.4.30         November 18, 2004

hidden-2.4.26-1.diff         2.4.26 – 2.4.27         February 28, 2004

hidden-2.4.25-1.diff         2.4.25         February 19, 2004

hidden-2.4.20pre10-1.diff         2.4.20pre10 – 2.4.24         October 12, 2002

hidden-2.4.19pre5-1.diff         2.4.19pre5 – 2.4.20pre9         April 7, 2002

hidden-2.4.5-1.diff         2.4.5 – 2.4.19pre4         June 2, 2001

hidden-2.4.4-1.diff         2.4.4         April 29, 2001

Patch        Linux 2.6        Created

hidden-2.6.9-1.diff         2.6.9 – 2.6.11         October 19, 2004

hidden-2.6.4-1.diff         2.6.4 – 2.6.8         March 12, 2004

hidden-2.6.3-1.diff         2.6.3         February 19, 2004

hidden-2.5.67-1.diff         2.5.67 – 2.6.2         April 9, 2003

重新编译内核,修正arp problem

####################################################################

注意事项:

要用干净的内核,不要发布版的,或者商业版本

一定要使用redhat网站上的内核,否则,哈哈

1. Kernel,IPVS,IPVSadm,ARP hidden之间的版本必须对应。

2. 自己编译内核时,从http://www.kernel.org/下载标准内核源文件,不要使用发行版的内核源文件。

3. Kernel 2.4.28和2.6.10及以上版本已内置IPVS,有些Linux发行版也在其内核里编译了IPVS。

4. ARP hidden可以用arp_ignore/arp_announce或者arptables_jf代替

patch -p1 < hidden-2.6.9-1.diff

三、具体配置顺序:

1、从http://www.kernel.org/下载新内核2.6.10

解压到 /usr/src/

内核文件夹重命名为linux

2、打ARP补丁

cd /usr/src/linux

patch -p1 < hidden-2.6.9-1.diff

3、编译内核:

编译2.6.10的内核 时

#

#make mrproper

#make menuconfig

找到lvs的配置,基本不用修改。

#make clean

#make

#make modules_install

#make install

4、使用新内核:

vi /boot/grub/grub.conf

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You have a /boot partition.  This means that

#          all kernel and initrd paths are relative to /boot/, eg.

#          root (hd0,0)

#          kernel /vmlinuz-version ro root=/dev/sda6

#          initrd /initrd-version.img

#boot=/dev/sda

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title Red Hat Enterprise Linux AS (2.6.10)

        root (hd0,0)

        kernel /vmlinuz-2.6.10 ro root=LABEL=/

        initrd /initrd-2.6.10.img

title Red Hat Enterprise Linux AS (2.6.9-5.ELsmp)

        root (hd0,0)

        kernel /vmlinuz-2.6.9-5.ELsmp ro root=LABEL=/

        initrd /initrd-2.6.9-5.ELsmp.img

title Red Hat Enterprise Linux AS-up (2.6.9-5.EL)

        root (hd0,0)

        kernel /vmlinuz-2.6.9-5.EL ro root=LABEL=/

        initrd /initrd-2.6.9-5.EL.img

5、重新启动系统,使用新的内核

#uname -a

Linux lvs-director 2.6.10 #1 SMP Mon Oct 15 14:22:45 EDT 2007 i686 i686 i386 GNU/Linux

6、安装ipvsadm软件

cd ipvsadm-1.24

make

make install

7、配置 director 和realserver

下载configure文件,运行时需要perl的支持。

我在使用时提示出

Can’t locate Net/DNS.pm in @INC (@INC contains:

说明perl缺少了 Net::DNS模块

#perl -MCPAN -e “shell”

按照提示进行即可

cpan>install Net::DNS

我在安装perl的dns模块没有成功,

t/08-online…………….NOK 57#     Failed test (t/08-online.t at line 109)

Failed Test   Stat Wstat Total Fail  Failed  List of Failed

——————————————————————————-

t/08-online.t    4  1024    93    6   6.45%  57-58 92-93

3 tests and 20 subtests skipped.

Failed 1/25 test scripts, 96.00% okay. 4/1181 subtests failed, 99.66% okay.

make: *** [test_dynamic] Error 255

  /usr/bin/make test — NOT OK

Running make install

  make test had returned bad status, won’t install without force

超时,一直没有make test成功

没有办法,我从http://www.perl.com 上下载了perl的Net-dns模块Net-DNS-0.60.tar.gz单独安装

tar -zxvf Net-DNS-0.60.tar.gz

#cd Net-DNS-0.60

#perl Makefile.PL

#make

#make test

#make install

cp -r lib/Net/DNS* /usr/lib/perl5/5.8.5/Net

8、下载configure脚本configure-lvs_0.9.4.tar.gz  http://www.austintek.com/LVS/configure-lvs/configure-lvs_0.9.4.tar.gz,然后执行配置文件

cd configure-lvs_0.9.4

./configure lvs_dr.conf

运行的时候先要执行loader端的脚本。

#以下是在director端的配置

#lvs_dr.conf

#format is (C) 2000-2001 Joseph Mack [email protected]

#edited by [email protected]

#released under GPL.

#single network director VS-DR.

#VIP,real-servers on 192.168.1.0/24 on eth0

#There is no router in this setup, all hosts are on the same network

#————————————————————

#

#IPs can be resolvable names (e.g. in /etc/hosts)

#Here’s my /etc/hosts

#

#VIP, the IP the clients connect to.

#192.168.1.199 lvs

#

#DIP is the IP on the director in the real-server network

#192.168.1.191 dip

#

#GW, my test client 

#GW 192.168.1.1

#192.168.1.117 client

#

#real-servers

#192.168.1.192 RS1

#192.168.1.193 RS2

#————————————————————-

#setup:

#This script installs the VIP.

#The CIP, DIP and RIPs must be already installed,

#machines must be connected and be able to ping each other.

#CIP, RIPs usually are primary IPs on an interface.

#VIP, DIP are secondary (alias) IPs.

#

#

#               ________

#              |        |

#              | client |

#              |________|

#                  | CIP=eth0 192.168.1.117

#                  |

#                  |       __________

#                  |      |          |

#                  |——| director |

#                  |      |__________|

#                  |  VIP=eth0:199 192.168.1.199/32

#                  |  DIP=eth0   192.168.1.191

#                  |

#                  |

#         ——————-

#         |                 |

#         |                 |

#   ______________    ______________

#  |              |  |              |

#  | realserver1  |  | realserver2  |

#  |______________|  |______________|

#     RIP1=eth0         RIP2=eth0

#    192.168.1.192       192.168.1.193

#

#           all realservers

#       VIP=lo:199=192.168.1.199 #

#

#———-lvs_dr.conf—————————————-

—— LVSCONF_FORMAT=1.1

—— LVS_TYPE=VS_DR

—— INITIAL_STATE=on

—— CLEAR_IPVS_TABLES=yes

—— #VIP line format – device[:alias] IP netmask broadcast

—— #To help avoid namespace collisions with other VIPs, I set alias=last number of VIP (here 110).

—— #note: for VS-DR, VS-Tun, the IP is in a /32 network

—— VIP=eth0:110 lvs 255.255.255.255 lvs

—— #DIP line format – device[:alias] IP network netmask broadcast

—— DIP=eth0:9 dip 192.168.1.0 255.255.255.0 192.168.1.255

—— #no DIRECTOR_GW for VS-DR or VS-Tun

—— #DIRECTOR_GW=

—— #SERVICE line format – proto port scheduler IP[,weight] [IP[,weight]]

—— SERVICE=t telnet rr RS1 RS2

—— #SERVICE=t ftp rr RS1,1 RS2,2

—— SERVICE=t http rr RS1 RS2

—— SERVER_VIP_DEVICE=lo:110

—— SERVER_NET_DEVICE=eth0

—— #SERVER_GW – packets with src_addr=VIP, dst_addr=0/0 are sent to SERVER_GW

—— #to be forwarded to the outside world.

—— #For standard VS-DR,VS-Tun, this must _NOT_ be the director.

—— #For Julian’s martian modification (see the HOWTO), it will be the director.

—— #If you don’t know about the martian modification, you aren’t using it.

—— #The script will not neccesarily set up the SERVER_GW as the real-servers’s default gw.

—— SERVER_GW=client

#———-end lvs_dr.conf————————————

#——- my lvs_dr.conf in 192.168.1.191——————

LVSCONF_FORMAT=1.1

LVS_TYPE=VS_DR

INITIAL_STATE=on

CLEAR_IPVS_TABLES=yes

#VIP line format – device[:alias] IP netmask broadcast

#To help avoid namespace collisions with other VIPs, I set alias=last number of VIP (here 110).

#note: for VS-DR, VS-Tun, the IP is in a /32 network

VIP=eth0:199 192.168.1.199 255.255.255.255 192.168.1.199

#DIP line format – device[:alias] IP network netmask broadcast

DIP=eth0 192.168.1.191 192.168.1.0 255.255.255.0 192.168.1.255

#no DIRECTOR_GW for VS-DR or VS-Tun

#DIRECTOR_GW=

#SERVICE line format – proto port scheduler IP[,weight] [IP[,weight]]

#SERVICE=t telnet rr 192.168.1.192 192.168.1.193

#SERVICE=t ftp rr 192.168.1.192,1 192.168.1.193,2

SERVICE=t http rr 192.168.1.192 192.168.1.193

SERVER_VIP_DEVICE=lo:199

SERVER_NET_DEVICE=eth0

#SERVER_GW – packets with src_addr=VIP, dst_addr=0/0 are sent to SERVER_GW

#to be forwarded to the outside world.

#For standard VS-DR,VS-Tun, this must _NOT_ be the director.

#For Julian’s martian modification (see the HOWTO), it will be the director.

#If you don’t know about the martian modification, you aren’t using it.

#The script will not neccesarily set up the SERVER_GW as the real-servers’s default gw.

SERVER_GW=192.168.1.1

#——- end of my lvs_dr.conf in 192.168.1.191——————

9、在lvs-dr端配置启动脚本

#—————mini-rc.lvs_dr-director in ldr 192.168.1.191————————#

#!/bin/bash

#mini_rc_lvs_dr.sh

#set ip_forward OFF for lvs-dr director (1 on, 0 off)

#(there is no forwarding in the conventional sense for LVS-DR)

cat       /proc/sys/net/ipv4/ip_forward

echo “0” >/proc/sys/net/ipv4/ip_forward

#director is not gw for realservers: leave icmp redirects on

echo ‘setting icmp redirects (1 on, 0 off) ‘

echo “1” >/proc/sys/net/ipv4/conf/all/send_redirects

cat       /proc/sys/net/ipv4/conf/all/send_redirects

echo “1” >/proc/sys/net/ipv4/conf/default/send_redirects

cat       /proc/sys/net/ipv4/conf/default/send_redirects

echo “1” >/proc/sys/net/ipv4/conf/eth0/send_redirects

cat       /proc/sys/net/ipv4/conf/eth0/send_redirects

#add ethernet device and routing for VIP 192.168.1.199

/sbin/ifconfig eth0:110 192.168.1.199 broadcast 192.168.1.199 netmask 255.255.255.255

/sbin/route add -host 192.168.1.199 dev eth0:199

#listing ifconfig info for VIP 192.168.1.199

/sbin/ifconfig eth0:199

#check VIP 192.168.1.199 is reachable from self (director)

/bin/ping -c 1 192.168.1.199

#listing routing info for VIP 192.168.1.199

/bin/netstat -rn

#setup_ipvsadm_table

#clear ipvsadm table

/sbin/ipvsadm -C

#installing LVS services with ipvsadm

#add telnet to VIP with round robin scheduling

/sbin/ipvsadm -A -t 192.168.1.199:http -s rr

#forward telnet to realserver using direct routing with weight 1

/sbin/ipvsadm -a -t 192.168.1.199:http -r 192.168.1.192 -g -w 1

#check realserver reachable from director

ping -c 1 192.168.1.192

#forward telnet to realserver using direct routing with weight 1

/sbin/ipvsadm -a -t 192.168.1.199:http -r 192.168.1.193 -g -w 1

#check realserver reachable from director

ping -c 1 192.168.1.193

#displaying ipvsadm settings

/sbin/ipvsadm

#not installing a default gw for LVS_TYPE vs-dr

#—————mini-rc.lvs_dr-director————————#

#mini-rc.lvs_dr-director 简单的配置 in 192.168.1.191

#/bin/sh

# lv-dr.sh in 192.168.1.191 in simple

# set ip_forward OFF for vs-dr director (1 on, 0 off)

echo 0 > /proc/sys/net/ipv4/ip_forward

echo “1” > /proc/sys/net/ipv4/conf/all/send_redirects

echo “1” > /proc/sys/net/ipv4/conf/default/send_redirects

echo “1” > /proc/sys/net/ipv4/conf/eth0/send_redirects

ifconfig eth0:0 192.168.1.199 netmask 255.255.255.255 broadcast 192.168.1.255 up

route add -host 192.168.1.199 dev eth0:199

ipvsadm -C

ipvsadm -A -t 192.168.1.199:80 -s rr

# Set Real Server

ipvsadm -a -t 192.168.1.199:80 -r 192.168.0.192 -g

ipvsadm -a -t 192.168.1.199:80 -r 192.168.0.193 -g

ipvsadm -Ln

==============================

在loader上执行

[[email protected] Net]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.1.199:80 rr

  -> 192.168.1.193:80             Route   1      0          0        

  -> 192.168.1.192:80             Route   1      0          0 

10、realserver的配置

#以下是realserver 端的配置及脚本

#多个realserver的配置都是一样

#realserver的其他前提配置: 需要解决arp的问题和重新编译内核(跟lvs-dr的),配置好http服务。

#!/bin/bash

#———-mini-rc.lvs_dr-realserver——————

#installing default gw 192.168.1.1 for vs-dr

/sbin/route add default gw 192.168.1.1

#showing routing table

/bin/netstat -rn

#checking if DEFAULT_GW 192.168.1.1 is reachable

ping -c 1 192.168.1.1

#set_realserver_ip_forwarding to OFF (1 on, 0 off).

echo “0” >/proc/sys/net/ipv4/ip_forward

cat       /proc/sys/net/ipv4/ip_forward

#looking for DIP 192.168.1.191

ping -c 1 192.168.1.191

#looking for VIP (will be on director)

ping -c 1 192.168.1.199

#install_realserver_vip

/sbin/ifconfig lo:199 192.168.1.199 broadcast 192.168.1.199 netmask 0xffffffff up

#ifconfig output

/sbin/ifconfig lo:199

#installing route for VIP 192.168.1.199 on device lo:199

/sbin/route add -host 192.168.1.199 dev lo:199

#listing routing info for VIP 192.168.1.199

/bin/netstat -rn

#hiding interface lo:199, will not arp

echo “1” >/proc/sys/net/ipv4/conf/all/hidden

cat       /proc/sys/net/ipv4/conf/all/hidden

echo “1” >/proc/sys/net/ipv4/conf/lo/hidden

cat       /proc/sys/net/ipv4/conf/lo/hidden

===========================

11、测试

client使用浏览器访问: http://192.168.1.199

会有如下的页面出现,如图(需要在realserver端配置httpd服务,设置好不同的显示页面):

192.jpg193.jpg

使用不同浏览器或者间隔段时间测试,显示不同的页面则说明配置成功。

安装中仔细阅读 LVS-mini-HOWTO 基本的问题都可解决。

一般来说,为了是负载均衡能够有高可用性,loader需要使用两台,配置heartbeat,等配置成功了弄上来,一起分享。

本文固定链接: http://2hei.net/lvs-dr%e5%ae%89%e8%a3%85%e7%ac%94%e8%ae%b0-by-2hei.html | 2hei's site

该日志由 2hei 于2007年10月16日发表在 others 分类下,
原创文章转载请注明: lvs-dr安装笔记-by 2hei | 2hei's site

报歉!评论已关闭.