레이블이 memcache인 게시물을 표시합니다. 모든 게시물 표시
레이블이 memcache인 게시물을 표시합니다. 모든 게시물 표시

20130709

PHP Sessions in Memcache – Locking Issues

Actually it is ages since I sat down to scribble something. Well this one could not be avoided. Hence here it is.
In one of our FTE projects, we had faced a complication that Memcached on one node was using 100% cpu and php-cgi was complainging that the same node was not permitting any more memcached connections. The configuration was as what all says, session.save_handler = memcache, session.save_path = “tcp://:11211,tcp://:11211,tcp://:11211″. It was giving jitters to the night support, that this used to happen at the worse time when most of the clients are using the application. And eventually that memcached needed to be restarted, kicking all users out and every one has to login back from the login page. Now during the past weeks it was so horrible that we marked a portion of the ramdisk from one least loaded nodes and used nfs to export this to all the nodes for a file based sessions store.

To explain things more, our application written using the extJS javascript framework wrapped in a custom php framework, was having ajax and frames, with php sessions. Means any one request will lock the sessions till that request is completed, so at some point in the course of work, there was some database deadlocks which might have loaded one or two php requests tieing down the sessions. Memcached is not designed to work like this, and locking the tcp:// would cause the system to block any further php requests to the memcached daemon.
Armed with all these informations, I tried to find a custom solution which may be blocks only the particular connection or to be explicit, the exact browser-php-session would be blocked, and would not affect any other sessions. I expanded a User Contributed Note from php maualsession_set_save_handler to build my first version which is reproduced below.

<?php
 
class SessionSaveHandler {
    protected 
$mc;
 
    public function 
__construct() {
        
session_set_save_handler(
            array(
$this"open"),
            array(
$this"close"),
            array(
$this"read"),
            array(
$this"write"),
            array(
$this"destroy"),
            array(
$this"gc")
        );
    }
    
    public function 
open($savePath$sessionName) {
        
$this->mc = new Memcache();
        
$this->mc->connect('127.0.0.1'11211);
        return 
true;
    }
 
    public function 
close() {
        
$this->mc->close();
        return 
true;
    }
 
    public function 
read($id) {
        return 
$this->mc->get($id);
    }
 
    public function 
write($id$data) {
        return 
$this->mc->set($id$datafalseini_get('session.gc_maxlifetime'));
    }
 
    public function 
destroy($id) {
        return 
$this->mc->delete($id);
    }
 
    public function 
gc($maxlifetime) {
    }
    
}
 
$mcsess = new SessionSaveHandler();session_start();
 
This version should not be used on production, and is provided here only for academic purposes. This can produce unpredicted results. Consider the following scenario.
* We have a page that starts with few data, asks for user login
* Login is ajax processed ** (1)
* On loging, we try to fetch the user preferences from the server ** (2)
The preferences are saved into the session
* Simulataneously a request is sent to the server to check for notifications ** (3)
Notification shown is saved into the session
Now ** (2) takes more time than ** (3) due to some reasons in the production, where as in the test environment the sequnce was okay due to the negligble latency, and low database content volume, as well as very low practical concurrent sessions. In production, a subsequent page will not see any information related to ** (3) in the session. Because, php will read the session on start of ** (2) and then write that back with the updated values when ** (2) completes processing, but ** (3) has already processed and updated the session. This is why php had a session locking. When using the sessions in file method, the locking will affect only the said session, and never affect any other session. Well though the current workaround is much high profile than the default one, still it uses the NFS as well as has the scalablitiy issues so, with the expense of some processor cycles, the above buggy code was extended to the following.

<?php
 
class SessionSaveHandler {
    protected 
$mc;
    protected 
$pid;
 
    public function 
__construct() {
        
session_set_save_handler(
            array(
$this"open"),
            array(
$this"close"),
            array(
$this"read"),
            array(
$this"write"),
            array(
$this"destroy"),
            array(
$this"gc")
        );
        
$this->pid uniqid("");
    }
    
    public function 
open($savePath$sessionName) {
        
$this->mc = new Memcache();
        
$this->mc->connect('127.0.0.1'11211);
        return 
true;
    }
 
    public function 
close() {
        
$this->mc->close();
        return 
true;
    }
 
    public function 
read($id) {
        while((
$locked $this->mc->get($id '-lck2'))){
            if(
$locked == $this->pid or $locked === false){
              break;
            }
            
usleep (10);
        }
        
$this->mc->set($id '-lck2'$this->pidfalseini_get('session.gc_maxlifetime'));
        return 
$this->mc->get($id);
    }
 
    public function 
write($id$data) {
        
$locked $this->mc->get($id '-lck2');
        if(
$locked !== $this->pid){
            return;
        }
        
$this->mc->set($id$datafalseini_get('session.gc_maxlifetime'));
        
$this->mc->set($id '-lck2'falsefalseini_get('session.gc_maxlifetime'));
        return 
true;
    }
 
    public function 
destroy($id) {
        return 
$this->mc->delete($id);
    }
 
    public function 
gc($maxlifetime) {
    }
    
}
 
$mcsess = new SessionSaveHandler();session_start();
 
This one addresses the issue and scenario outlined above by just locking the concerned browser-session. Whereas it provides the scalability as provided by Memcached pool. Theoretically this should work, and to an extend a medium level testing was conducted but still I could not get this code out into production. Since the GA releases are very stringent and this should go in the post chrismas QA release only. Anyway this being a complete detached code, I am providing the second one for download. PHP Sessions in Memcache Non Blocking (267)



20130612

CentOS5.x에 nginx, php-fpm, mariadb, memcache, barcode 등 프로그램 설치

현재 업무 중 많은 부분이 쇼핑몰과 SCM, ERP 등이다..

평균 동접은 약 1,000명 수준이다보니 단일 서버는 어렵고 분산 처리 방식으로 개발이 된다.


그에 맞는 서버 설정을 구성하였고 기록을 남긴다.

울 회사에서 내 포지션은 개발 관련 총책임자로 있어 돈이 들어가는 걸 기본적으로 싫어하는 나는 무료/오픈 소프트웨어 로 주로 사용하며 중국에서 근무하는 상황에서 개발인력 확보가 어렵고 인건비가 상당히 비싼 점을 고려한 세팅이다.

초기 개발 시에는 JAVA도 사용하였지만 현재는 다 걷어낸 상황이다.


대략 정리하면 아래와 같다.

Language : php-fpm(web), python & shell(server inside)

OS : CentOS5.x 64bit

Software : nginx(web or proxy), php,  python, memcache, mmonit, munin, sphinx, mysql or mariadb, imagemagick, genbarcode(barcode), ioncube, goaccess


설치 과정.

1. 우선 CentOS 설치시에 모든 패키지를 제거하고 최소화로 하여 설치를 진행한다.


2. 64bit를 기준으로 설치하였기 때문에 32bit 소프트웨어는 제거.
 # yum remove \*.i\?86

3. yum 으로 설치되는 부분이 있기 때문에 yum.conf 에다가 64bit 소프트웨어만 설치하도록 32bit 소프트웨어는 exclude 처리한다.
# vi /etc/yum.conf

exclude = *.i?86  #add exclude


4. yum으로 기본 패키지 정리.
#  yum -y remove php httpd mysql bind dhclient tftp inetd xinetd ypserv telnet-server rsh-server vsftpd tcsh nfs* samba tftp-server telnet system-config-printer
#  yum -y install system-config-date subversion gcc gcc-c++ g++ cpp make ncurses-devel automake autoconf tcl-devel rdate rsync pcre-devel gd-devel zlib zlib-devel curl curl-devel libcurl-devel openssl openssl-devel libopenssl-devel libtermcap-devel libc-client-devel bzip2-devel

5.ulmiit 설정
# ulimit -SHn 10240
# echo "ulimit -SHn 10240" >> /etc/rc.local

6. imagemagick 설치
# yum -y install ImageMagick*

7. iconv 설치 (source)
# tar zxvf libiconv-1.14.tar.gz
# cd libiconv-1.14
# ./configure --prefix=/usr/local
# gmake
# gmake install
# export LD_PRELOAD=/usr/local/lib/preloadable_libiconv.so

8. mysql or mariadb 설치 (2013년 이전에는 주변에서 데이터베이스 컨설팅 시에 돈 없으면 mysql을 사용하고 아주 아주 잘 설계하면 아주 아주 잘 버텨준다고 했다. 2013년 이후엔 mariadb를 추천해주고 있다. 이유는 explain 을 해보라. 그러면 알 것이다. 어차피 실 서비스는 정규화 설계로는 답없는 경우가 대부분이다.)


# groupadd mysql
# useradd -s /bin/false -g mysql mysql


8.1 mysql 설치 시... (5.6.x 기준)
# tar xvfz mysql-5.6.x.tar.gz
# cd mysql-5.6.x
# cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DWITH_EXTRA_CHARSETS=all -DMYSQL_DATADIR=/usr/local/mysql/data -DENABLED_LOCAL_INFILE=1 -DWITH_INNOBASE_STORAGE_ENGINE=1 -DMYSQL_UNIX_ADDR=/tmp/mysql.sock -DSYSCONFDIR=/usr/local/mysql -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DWITH_EXTRA_CHARSETS=all -DMYSQL_TCP_PORT=3306 -DENABLE_DTRACE=0


 8.2 mariadb 설치 시 (10.x.x 기준)
# tar xvfz mariadb-10.x.x.tar.gz
# cd mariadb-10.x.x
# cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DWITH_EXTRA_CHARSETS=all -DMYSQL_DATADIR=/usr/local/mysql/data -DENABLED_LOCAL_INFILE=1 -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DWITH_ARIA_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DMYSQL__ADDR=/tmp/mysql.sock -DSYSCONFDIR=/usr/local/mysql -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DWITH_EXTRA_CHARSETS=all -DMYSQL_TORT=3306 -DENABLE_DTRACE=0

mysql과 mariadb는 cmake 까지만 다르다..

# gmake
# gmake install
# echo /usr/local/mysql/lib >> /etc/ld.so.conf && ldconfig

설정 파일은 알아서 잘...(여기에 올려진 설정은 메모리 16GB기준의 서버의 설정이다)

# cp example.cnf /usr/local/mysql/my.cnf
# cp support-files/mysql.server /usr/local/mysql/bin/


# chmod 700 /usr/local/mysql/bin/mysql.server
# chown mysql. -R /usr/local/mysql
# /usr/local/mysql/scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data

관리할 폴더 생성
# mkdir /usr/local/mysql/logs
# chmod 755 /usr/local/mysql/tmp
# chmod 755 /usr/local/mysql/logs
# chown -R mysql. /usr/local/mysql

링크 생성
# ln -s /usr/local/mysql/bin/mysql /usr/bin/


# echo "/usr/local/mysql/bin/mysql.server start" >> /etc/rc.local

# /usr/local/mysql/bin/mysql.server start
# /usr/local/mysql/bin/mysql_secure_installation


mysql이나 mariadb의 설치는 동일하다고 보면 된다.



replication (필요시...)
master 에서 실행
# show master status;

slave 에서 실행.
# CHANGE MASTER TO
 MASTER_HOST='master ip',
 MASTER_USER='마스터에 접속할 계정',
 MASTER_PASSWORD='마스터에 접속할 비밀번호',
 MASTER_LOG_FILE='마스터에서 정보 조회 후 설정',
 MASTER_LOG_POS=마스터에서 정보 조회 후 설정;

# START SLAVE;
Slave에서 err를 확인하여 아래의 메시지가 있으면 접속 성공
Slave I/O thread: connected to master 'useid@hostname:3306',replicat
ion started in log 'mysql-bin.00000x' at position xxx




9. php 설치

# tar zxvf php-5.x.x.tar.gz
# cd php-5.x.x
# yum -y install libxml2-devel libpng-devel freetype-devel freetype2-devel gmp-devel libmcrypt libmcrypt-devel gd libjpeg-devel glibc glibc-devel glib2 glib2-devel GeoIP*
# ./configure --prefix=/usr/local/php --with-config-file-path=/usr/local/php/etc \
--with-mysql=/usr/local/mysql --with-mysqli=/usr/local/mysql/bin/mysql_config \
--with-iconv=/usr/local \
--with-freetype-dir --with-jpeg-dir --with-png-dir --with-gd --enable-gd-native-ttf \
--with-mcrypt --with-curl --with-gmp --with-zlib --with-openssl --with-pear --with-bz2 \
--enable-mbstring --enable-mbregex --enable-bcmath --enable-inline-optimization -enable-sockets \
--enable-fpm

# gmake
# gmake install

# mkdir /usr/local/php/logs

# cp php.ini-production /usr/local/php/etc/php.ini
# rm -rf /usr/bin/php
# ln -s /usr/local/php/bin/php /usr/bin/

# cp /usr/local/php/etc/php-fpm.conf.default /usr/local/php/etc/php-fpm.conf
# vim /usr/local/php/etc/php-fpm.conf (알아서설정)

# echo "/usr/local/php/sbin/php-fpm" >> /etc/rc.local

# vim /usr/local/php/etc/php.ini (알아서설정)
extension_dir = "/usr/local/php/lib/php/extensions/no-debug-non-zts-20090626/"

# /usr/local/php/sbin/php-fpm



10. ioncube loader 설치 (소스 중에 주요부분은 사용하고 있어서 사용. 근데 이거 은근 메모리 많이 먹는다.)

# wget http://downloads2.ioncube.com/loader_downloads/ioncube_loaders_lin_x86-64.tar.gz
# tar xfz ioncube_loaders_lin_x86-64.tar.gz
# mv ioncube /usr/local/

# vim /usr/local/php/etc/php.ini
zend_extension = /usr/local/ioncube/ioncube_loader_lin_5.3.so (추가)
date.timezone = Asia/Shanghai (수정)





php 재시작
# killall php-fpm
# /usr/local/php/sbin/php-fpm


11. Zend 설치 (근데 요즘 사용안 하기 때문에 열외)

#  tar xvfpz ZendGuardLoader-php-5.3-linux-glibc23-x86_64.tar.gz
# mkdir /usr/local/zend/
# cp ZendGuardLoader-php-5.3-linux-glibc23-x86_64/php-5.3.x/ZendGuardLoader.so /usr/local/zend/

# vi /usr/local/php/etc/php.ini

[Zend]
zend_loader.enable=1
zend_loader.disable_licensing=0
zend_extension=/usr/local/zend/ZendGuardLoader.so

php 재시작
# killall php-fpm
# /usr/local/php/sbin/php-fpm







12. nginx 설치
# tar xzvf nginx-1.x.x.tar.gz
# cd nginx-1.x.x/

# ./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_gzip_static_module --with-http_flv_module --with-http_mp4_module --with-http_geoip_module --with-http_realip_module --with-http_image_filter_module

# gmake
# gmake install
# echo "/usr/local/nginx/sbin/nginx" >> /etc/rc.local
# vim /usr/local/nginx/conf/nginx.conf (알아서 설정)

# /usr/local/nginx/sbin/nginx


13. GepIP
# yum -y install GeoIP GeoIP-devel GeoIP-data perl-Geo-IP
   
# vim /usr/local/nginx/conf/nginx.conf 에 추가
geoip_country /var/lib/GeoIP/GeoIP.dat;

해당 가상호스트 설정 부분의 php 설정에 추가
fastcgi_param  COUNTRY $geoip_country_code;


14. 바코드 설치

# wget http://mirror.yongbok.net/gnu/barcode/barcode-0.98.tar.gz
# tar xfvz barcode-0.98.tar.gz
# cd barcode-0.98
# ./configure
# gmake
# gmake install
# ldconfig

# wget http://www.ashberg.de/php-barcode/download/files/genbarcode-0.4.tar.gz
# tar xvfz genbarcode-0.4.tar.gz
# cd genbarcode-0.4
# gmake
# gmake install

php 예제....
# wget http://www.ashberg.de/php-barcode/download/files/php-barcode-0.4.tar.gz
# tar xvfpz php-barcode-0.4.tar.gz
# cd php-barcode-0.4


15. munin 설치 (nginx, php, mysql, mariadb 등의 plugin은 알아서들 설치)
# rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
# yum install munin-node -y
# vim /etc/munin/munin-node.conf
# chkconfig munin-node on
# /etc/init.d/munin-node start


16. monit 설치 (munin 이 정상적으로 잘 돌고 있는 지 감시의 목적)
# tar xvfz monit-x.y.z.tar.gz
# cd monit-x.y.z
# ./configure --prefix=/usr/local/monit
# gmake
# gmake install


17. goaccess 설치
# wget http://sourceforge.net/projects/goaccess/files/0.4.2/goaccess-0.4.2.tar.gz/download
# ./configure
# make
# make install

18. sphinx 설치... (이건 워낙 확장들이 많아서... 난 우선 중국에 있어 sphinx-chinese 를 사용)

참조 http://www.sphinx-search.com/ 여기 가면 친절히 설명이 되어있다.




tune 하는 부분은 따로 포스트 할 생각이다.


위의 내용은 아주아주 기본적인 설치이며... pmap으로 각 process를 모니터링 & 튠해야 한다.




Articles