存档

‘OpenSource’ 分类的存档

Get memcached all items and deleted them if needed.

2010年7月14日 2 条评论

    I see there are so many api to operate memcached,such as get and set,but no list all items,so i write a shell just list all the items in memcached and we can delete the items when we don’t know the exactly items key.

#!bin/bash
# get_items_from_memcached.sh
# Usge: sh get_items_from_memcached.sh localhost port
# Exp:  sh get_items_from_memcached.sh 2hei.net 11211
# By: @2hei.net

items=`echo “stats items” | nc $1 $2|grep number|awk -F: ‘{print $2}’|awk ‘{printf(“%s “,$1) }’`
for i in ${items}
  do
    #get delete_items_list
    echo “stats cachedump $i 0” | nc $1 $2|awk -v HOST=$1 -v PORT=$2 ‘{if(length($2)>0) print “echo delete”,$2,” | nc”,HOST,PORT}’ >> $1_$2.txt
    #print all items
    echo “stats cachedump $i 0” | nc $1 $2|awk ‘{if($2) print $2}’
  done

##delete all items by item_list if needed
#/bin/sh $1_$2.txt

##END##

tips:
1. you just can use “flush_all” cmd
   echo “flush_all” | nc $1 $2
   
   “flush_all” is a command with an optional numeric argument. It always
succeeds, and the server sends “OK\r\n” in response (unless “noreply”
is given as the last parameter). Its effect is to invalidate all
existing items immediately (by default) or after the expiration specified.
   flush_all doesn’t actually free all the memory taken up by existing items; that
will happen gradually as new items are stored. The most precise
definition of what flush_all does is the following: it causes all
items whose update time is earlier than the time at which flush_all
was set to be executed to be ignored for retrieval purposes.

2. if your memcached has to many items, this shell will waste a long time, for it will establish a new connection when delete each time.
we can use other tools write by socket and do this in only one connection.

分类: OpenSource 标签:

write nagios nrpe plugin

2010年7月1日 1 条评论

Scripts and executables must do two things (at a minimum) in order to function as Nagios plugins:
1.Exit with one of several possible return values
2.Return at least one line of text output to STDOUT

Plugin Return Code Service State Host State
0 OK UP
1 WARNING UP or DOWN/UNREACHABLE*
2 CRITICAL DOWN/UNREACHABLE
3 UNKNOWN DOWN/UNREACHABLE
Note: If the use_aggressive_host_checking option is enabled, return codes of 1 will result in a host
state of DOWN or UNREACHABLE. Otherwise return codes of 1 will result in a host state of UP.

Plugin Output Spec
At a minimum, plugins should return at least one of text output. Beginning with Nagios 3, plugins can
optionally return multiple lines of output. Plugins may also return optional performance data that can
be processed by external applications. The basic format for plugin output is shown below:
TEXT OUTPUT | OPTIONAL PERFDATA
LONG TEXT LINE 1
LONG TEXT LINE 2

LONG TEXT LINE N | PERFDATA LINE 2
PERFDATA LINE 3

PERFDATA LINE N

this is my python scripts:
#!/usr/bin/evn python
# -*- coding: utf-8 -*-

import sys,getopt
import memcache

memcached_host=’2hei.net’
memcached_port=11211
Warning_item=120
Critical_item=20

def usage():
    print “””
Usage: check_memcached [-h|–help] [-w|–warning curr_items] [-c|–critical curr_items]”
Warning curr_items defaults to 120
Critical curr_items defaults to 20
“””
    sys.exit(3)

#get curr_items from memcache stats
def get_memcache_curr_items(mc):
    #mc = memcache.Client([memcached_host+’:’+str(memcached_port)], debug=0)
    stats = mc.get_stats()[0][1]   
    #for i in xrange(0,100):
    #    mc.set(‘key’+str(i),’value’+str(i))
    #for k,v in stats.items():
    #    print k,v
    items = stats.get(‘curr_items’)
    return items

if __name__ == “__main__”:
    warning_item = 0
    critical_item = 0

    try:
        options, args = getopt.getopt(sys.argv[1:],”h:w:c:”,”–help –warning= –critical=”,)
    except getopt.GetoptError:
        usage()
        sys.exit(3)

    try:
        mc = memcache.Client([memcached_host+’:’+str(memcached_port)], debug=0)
        items = get_memcache_curr_items(mc)
        mc.disconnect_all()
    except Exception:
        print “Cannot get memcache’s curr_items.”,Exception
        sys.exit(3)

    for name, value in options:
        if name in (“-h”, “–help”):
            usage()
            sys.exit(3)
        if name in (“-w”, “–warning”):
            warning_item = value
        if name in (“-c”, “–critical”):
            critical_item = value

    if warning_item == 0:
        warning_item = Warning_item
    if critical_item == 0:
        critical_item = Critical_item

    if int(items) <= int(critical_item):
        print ‘MEMCACHED_ITEM CRITICAL: curr_items is:’,items
        sys.exit(2)
    if int(items) <= int(warning_item):
        print ‘MEMCACHED_ITEM WARNING: curr_items is:’,items
        sys.exit(1)
    else:
        print ‘MEMCACHED_ITEM OK: curr_items is:’,items
        sys.exit(0)

when encounter errors:
CHECK_NRPE: No output returned from daemon.
or
CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.
this shows your plugins return output is null

分类: OpenSource, python 标签:

notice Hostnames should not contain an ‘_’

2010年6月17日 评论已被关闭
notice Hostnames should not contain an ‘_’:
such as vm_test01.2hei.net
Please have a look at RFC952 (http://tools.ietf.org/html/rfc952) Hostnames should not contain an ‘_’!
ASSUMPTIONS:
   1. A “name” (Net, Host, Gateway, or Domain name) is a text string up
   to 24 characters drawn from the alphabet (A-Z), digits (0-9), minus
   sign (-), and period (.).  Note that periods are only allowed when
   they serve to delimit components of “domain style names”. (See
   RFC-921, “Domain Name System Implementation Schedule”, for
   background).  No blank or space characters are permitted as part of a
   name. No distinction is made between upper and lower case.  The first
   character must be an alpha character.  The last character must not be
   a minus sign or period.  A host which serves as a GATEWAY should have
   “-GATEWAY” or “-GW” as part of its name.  Hosts which do not serve as
   Internet gateways should not use “-GATEWAY” and “-GW” as part of
   their names. A host which is a TAC should have “-TAC” as the last
   part of its host name, if it is a DoD host.  Single character names
   or nicknames are not allowed.
分类: OpenSource 标签:

SHELL中的2进制、10进制、8进制、16进制之间的转换

2010年5月4日 评论已被关闭

一、16进制转换成10进制
printf %d 0xF
15
或者
echo $((16#F))
15

二、10进制转换成16进制
printf %x 15
f
或者
echo “obase=16;15″|bc
F

三、10进制转换成8进制
printf %o 9
11

四、8进制转换成10进制
echo $((8#11))
9

五、同理二进制转换成10进制
echo $((2#111))
7

六、10进制转换成二进制
echo “obase=2;15″|bc
1111

分类: linux, OpenSource, shell 标签:

openssh5.4升级后ssh无法登陆

2010年3月30日 评论已被关闭
杯具啊! 新装的服务器,从openssh5.2升级到openssh5.4,重启sshd服务后发现无法登陆了!!!
没办法,找机房人员终端登陆上去开启telnet,又把版本降了回来,汗一个!
谁让手欠呢!
分类: OpenSource 标签:

nginx代理模式下,伪造X-Forwarded-For头的测试

2010年3月18日 6 条评论

测试环境: nginx+resin
IP: 内网:172.16.100.10

客户端IP:123.123.123.123

测试页面: test.jsp
<%
out.println(“x-forwarded-for: ” + request.getHeader(“x-forwarded-for”));
out.println(“remote hosts: ” + request.getRemoteAddr());
%>

nginx 配置一
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

wget测试
wget -O aa –header=”X-Forwarded-For:192.168.0.1″ “http://2hei.net/test.jsp
页面返回结果:
x-forwarded-for: 192.168.0.1, 123.123.123.123
remote hosts: 172.16.100.10

curl测试
curl -H “X-Forwarded-For:192.168.0.1” “http://2hei.net/test.jsp
x-forwarded-for: 192.168.0.1, 123.123.123.123
remote hosts: 172.16.100.10

nginx 配置二
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

wget测试:
wget -O aa –header=”X-Forwarded-For:192.168.0.1″ “http://2hei.net/test.jsp
页面返回结果:
x-forwarded-for: 123.123.123.123
remote hosts: 172.16.100.10

curl测试
curl -H “X-Forwarded-For:192.168.0.1” “http://2hei.net/test.jsp
x-forwarded-for: 123.123.123.123
remote hosts: 172.16.100.10

测试结果:
1、配置  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
增加了一个真实ip X-Forwarded-For,并且顺序是增加到了“后面”。

2、配置  proxy_set_header X-Forwarded-For $remote_addr;
清空了客户端伪造传入的X-Forwarded-For,
保证了使用request.getHeader(“x-forwarded-for”)获取的ip为真实ip,
或者用“,”分隔,截取X-Forwarded-For最后的值。

分类: nginx, OpenSource 标签:

ethtool maybe caused by “TCP checksum offload”

2010年3月3日 评论已被关闭

使用tcpdump抓包时发现有这样的错误信息:

Transmission Control Protocol, Src Port: 44937 (44937), Dst Port: https (443), Seq: 111, Ack: 147, Len: 6

Checksum: 0x5edd [incorrect, should be 0x15db (maybe caused by “TCP checksum offload”?)]
 Good Checksum: False
 Bad Checksum: True

或许是TCP checksum offload的原因

查看本机的网卡信息
2hei# ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off

其他一些帮助
ethtool -K|–offload DEVNAME    Set protocol offload
                [ rx on|off ]
                [ tx on|off ]
                [ sg on|off ]
                [ tso on|off ]
                [ ufo on|off ]
                [ gso on|off ]

关闭rx和tx
#ethtool -K rx off
#ethtool -K tx off
#ethtool -K eth0 rx off tx off tso off gso off

也可用这个命令
#ethtool -K eth0 tx off tso off

打开
#ethtool -K eth0 tx on
#ethtool -K eth0 tso on

关闭tx和rx后,再次抓包,结果显示正常。

CheckSum Offload实际上是将传输层的一部分工作交给了硬件完成,以节约系统的CPU资源。wireshark在本地抓包时看到是系统随机填充的校验和,所以会显示Checksum Offload,实际并不影响数据包的正常传输。
微软的测试表明它可以最多节约30%的CPU资源。IBM里AIX的文档则指出:对于PCI接口的千兆网卡来说还不如让400Mhz以上的CPU来计算校验和,而PCI-X的千兆网卡启用此项后可以达到线路速度,从而节约CPU资源。

详细可见: http://www.microsoft.com/whdc/device/network/taskoffload.mspx

分类: OpenSource 标签:

linux下支持https的压力测试工具

2010年1月24日 2 条评论

测试了linux下的几种压力测试工具,发现有些不支持https,先简单总结如下:

一、apache的ab工具

/home/webadm/bin/ab -c 50 -n 10000 https://2hei.net/mt/index.html
SSL not compiled in; no https support
看样子是说SSL没有编译进来,所以不支持https

二、apache的flood工具

http://httpd.apache.org/test/flood/

wget http://www.apache.org/dist/httpd/flood/flood-0.4.tar.gz

如果要支持https的话,需要添加如下的编译参数:–with-apr –with-apr-util –enable-ssl
具体代码也可以从svn获取。
不过我在编译的时候遇到一些问题,apr-util包在make的时候总是报错!

三、web-bench工具

http://freshmeat.net/projects/web-bench/

./webbench -c 20 -t 10 https://2hei.net/mt/index.html
Webbench – Simple Web Benchmark 1.5
Copyright (c) Radim Kolar 1997-2004, GPL Open Source Software.

Only HTTP protocol is directly supported, set –proxy for others.
明确提示了提示不支持https哦!

四、http_load工具,曾经的最爱,http测试的结果还很令人满意。

http://acme.com/software/http_load/
./http_load -rate 5 -seconds 10 urls
./http_load: unknown protocol – https://2hei.net/mt/index.html
吼吼,看来之前常用的http_load也不支持https

五、siege工具

http://www.joedog.org/index/siege-home

编译使之支持https
./configure –prefix=/home/2hei.net/siege –with-ssl=/usr/include/openssl

基本用法:
./siege
SIEGE 2.69
Usage: siege [options]
       siege [options] URL
       siege -g URL
Options:
  -V, –version           VERSION, prints version number to screen.
  -h, –help              HELP, prints this section.
  -C, –config            CONFIGURATION, show the current configuration.
  -v, –verbose           VERBOSE, prints notification to screen.
  -g, –get               GET, pull down headers from the server and display HTTP
                          transaction. Great for web application debugging.
  -c, –concurrent=NUM    CONCURRENT users, default is 10
  -u, –url=”URL”         Deprecated. Set URL as the last argument.
  -i, –internet          INTERNET user simulation, hits the URLs randomly.
  -b, –benchmark         BENCHMARK, signifies no delay for time testing.
  -t, –time=NUMm         TIME based testing where “m” is the modifier S, M, or H
                          no space between NUM and “m”, ex: –time=1H, one hour test.
  -r, –reps=NUM          REPS, number of times to run the test, default is 25
  -f, –file=FILE         FILE, change the configuration file to file.
  -R, –rc=FILE           RC, change the siegerc file to file.  Overrides
                          the SIEGERC environmental variable.
  -l, –log               LOG, logs the transaction to PREFIX/var/siege.log
  -m, –mark=”text”       MARK, mark the log file with a string separator.
  -d, –delay=NUM         Time DELAY, random delay between 1 and num designed
                          to simulate human activity. Default value is 3
  -H, –header=”text”     Add a header to request (can be many)
  -A, –user-agent=”text” Sets User-Agent in request

siege -c 20 -r 2 -f url
-c 20 并发20个用户
-r 2 重复循环2次
-f url 任务列表:URL列表

结论相当凑合,纵欲找到可以支持https压力测试的工具了。

六、httperf工具,来自hp的工具,不过已经n年没有更新了。

wget ftp://ftp.hpl.hp.com/pub/httperf/httperf-0.9.0.tar.gz

        $ mkdir build
        $ cd build
        $ SRCDIR/configure
        $ make
        $ make install

工具使用方法:
/usr/local/bin/httperf –help
Usage: httperf [-hdvV] [–add-header S] [–burst-length N] [–client N/N]
        [–close-with-reset] [–debug N] [–failure-status N]
        [–help] [–hog] [–http-version S] [–max-connections N]
        [–max-piped-calls N] [–method S] [–no-host-hdr]
        [–num-calls N] [–num-conns N] [–period [d|u|e]T1[,T2]]
        [–port N] [–print-reply [header|body]] [–print-request [header|body]]
        [–rate X] [–recv-buffer N] [–retry-on-failure] [–send-buffer N]
        [–server S] [–server-name S] [–session-cookies]
        [–ssl] [–ssl-ciphers L] [–ssl-no-reuse]
        [–think-timeout X] [–timeout X] [–uri S] [–verbose] [–version]
        [–wlog y|n,file] [–wsess N,N,X] [–wsesslog N,X,file]
        [–wset N,X]
如:
/usr/local/bin/httperf –server www.2hei.net \
   –port 443 –uri /mt/index.html \
   –rate 15 –num-conn 1000 \
   –num-call 1 –timeout 5

发现会有如下错误:
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
以下是解决办法:
# Edit /etc/security/limits.conf and add the line * hard nofile 65535 (or instead of * you can put the username of the user for whom you want to change the limit)
# Edit /usr/include/bits/typesizes.h and change #define __FD_SET_SIZE 1024 to #define __FD_SET_SIZE 65535 (in /usr/include/sys/select.h FD_SETSIZE is defined as __FD_SETSIZE)
重新编译即可。

实例测试如下:
/usr/local/bin/httperf –client=0/1 –ssl –server=www.2hei.net –port=443 –uri=/mt/index.html –rate=1 –num-conns=1000 –rate=50 –num-calls=1 –hog
httperf –hog –client=0/1 –server=www.2hei.net –port=443 –uri=/mt/index.html –rate=50 –send-buffer=4096 –rec
v-buffer=16384 –ssl –num-conns=1000 –num-calls=1
Maximum connect burst length: 1

Total: connections 1000 requests 1000 replies 1000 test-duration 33.758 s

Connection rate: 29.6 conn/s (33.8 ms/conn, <=394 concurrent connections)
Connection time [ms]: min 73.1 avg 6513.5 max 22013.1 median 5371.5 stddev 4176.3
Connection time [ms]: connect 5670.7
Connection length [replies/conn]: 1.000

Request rate: 29.6 req/s (33.8 ms/req)
Request size [B]: 80.0

Reply rate [replies/s]: min 26.6 avg 31.3 max 33.6 stddev 2.4 (6 samples)
Reply time [ms]: response 842.7 transfer 0.0
Reply size [B]: header 331.0 content 163.0 footer 2.0 (total 496.0)
Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0

CPU time [s]: user 9.91 system 23.11 (user 29.4% system 68.5% total 97.8%)
Net I/O: 16.6 KB/s (0.1*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

分类: linux, OpenSource 标签: , , ,

nginx authentication

2010年1月20日 评论已被关闭

今天配置nginx的basic authentication折腾了好半天,开始error_log一直报错:
no user/password was provided for basic authentication

我是按照nginx的wiki配置的:
location  /  {
  auth_basic            “Restricted”;
  auth_basic_user_file  htpasswd;
}

cat htpasswd
2hei:j3M4coizxFLDM

Since version 0.6.7 the filename path is relative to directory of nginx configuration file nginx.conf, but not to nginx prefix directory.

google了半天,发现了nginx作者Igor的一个回复:

Igor Sysoev
The HTTP Basic authentication works as following:
*) A browser requests a page without user/password.
*) A server response with 401 page, sending realm as well.
   At this stage the 401 code appears in access_log and the message
   “no user/password …” appears in error_log.
*) The browser shows a realm/login/password prompt.
*) If a user will press cancel, then the browser will show the received
   401 page.
*) If the user enters login/password, then the browser repeats the request
   with login/password.

Then until you will exit the browser, it will send these login/password
with all requests in protected hierarchy.

赞一个,牛人果然就是牛人!

经过提示找到了罪魁祸首是error_page的配置 401 403 404 /40x.html;

error_page 401 403 404 /40x.html;
实际上是由于40x.html文件并不存在导致的。把文件建好问题得以解决!

分类: OpenSource 标签:

[notice] child pid 19818 exit signal Bus error (7)

2010年1月7日 1 条评论

apache logs:
[Thu Dec 31 12:27:41 2009] [notice] child pid 19818 exit signal Bus error (7)

The first error “Bus Error” is caused by someones (possibly CGI) program crashing. A bus error happens when someone wrote a C program for Intel and re-compiled it for another processor like PPC or Mips that does not support misalligned memory access. Really all it means is someone is running buggy software on your server machine.
The second error could also be a CGI program that hangs longer than the server’s timeout, so its killed. Probably another buggy program.

That sounds like bad RAM or maybe a bad chassis. I would try and build something other the Apache and see if you get similar errors. If they persist then I would ask for a RAM swap to be done and check again. If that does not fix it then request a chassis swap.

There is a small chance this could be OS related if you are running a different Kernel release compared to the OS default or have many patches installed on it but I am leaning towards a hardware issue.

http://www.kernel.org/doc/man-pages/online/pages/man7/signal.7.html

Standard Signals

       Linux supports the standard signals listed below.  Several signal numbers are
       architecture-dependent, as indicated in the “Value” column.  (Where three
       values are given, the first one is usually valid for alpha and sparc, the
       middle one for ix86, ia64, ppc, s390, arm and sh, and the last one for mips.
       A – denotes that a signal is absent on the corresponding architecture.)

       First the signals described in the original POSIX.1-1990 standard.

       Signal     Value     Action   Comment

       ———————————————————————-
       SIGHUP        1       Term    Hangup detected on controlling terminal
                                     or death of controlling process
       SIGINT        2       Term    Interrupt from keyboard
       SIGQUIT       3       Core    Quit from keyboard
       SIGILL        4       Core    Illegal Instruction
       SIGABRT       6       Core    Abort signal from abort(3)
       SIGFPE        8       Core    Floating point exception
       SIGKILL       9       Term    Kill signal
       SIGSEGV      11       Core    Invalid memory reference
       SIGPIPE      13       Term    Broken pipe: write to pipe with no
                                     readers
       SIGALRM      14       Term    Timer signal from alarm(2)
       SIGTERM      15       Term    Termination signal
       SIGUSR1   30,10,16    Term    User-defined signal 1
       SIGUSR2   31,12,17    Term    User-defined signal 2
       SIGCHLD   20,17,18    Ign     Child stopped or terminated
       SIGCONT   19,18,25    Cont    Continue if stopped
       SIGSTOP   17,19,23    Stop    Stop process
       SIGTSTP   18,20,24    Stop    Stop typed at tty
       SIGTTIN   21,21,26    Stop    tty input for background process
       SIGTTOU   22,22,27    Stop    tty output for background process

       The signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored.

       Next the signals not in the POSIX.1-1990 standard but described in SUSv2 and
       POSIX.1-2001.

       Signal       Value     Action   Comment
       ——————————————————————–
       SIGBUS      10,7,10     Core    Bus error (bad memory access)
       SIGPOLL                 Term    Pollable event (Sys V).
                                       Synonym for SIGIO
       SIGPROF     27,27,29    Term    Profiling timer expired
       SIGSYS      12,-,12     Core    Bad argument to routine (SVr4)
       SIGTRAP        5        Core    Trace/breakpoint trap
       SIGURG      16,23,21    Ign     Urgent condition on socket (4.2BSD)
       SIGVTALRM   26,26,28    Term    Virtual alarm clock (4.2BSD)
       SIGXCPU     24,24,30    Core    CPU time limit exceeded (4.2BSD)
       SIGXFSZ     25,25,31    Core    File size limit exceeded (4.2BSD)

分类: OpenSource 标签: