Kill Process based on Start Time (STIME)

One of the server that I am working with has some infinitely running PHP process. Due to incorrect way of cron setup by the development team, it has caused the process hanging and not ended properly. According to them, these processes can be killed if still hang after 12 hours.

Any process which run in server will have start time (STIME). You can check this by using ps command. In this case, following result will appear:

$ ps aux | grep php root 1399 0.0 0.0 61188 740 pts/2 S+ 10:10 0:00 grep php user1 2697 0.0 0.0 100664 8340 ? Ss Jul04 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 5551 0.0 0.4 171052 78832 ? Ss Jun25 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 9913 0.0 0.5 174636 82392 ? Ss Jun22 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 11961 0.0 0.7 223276 131060 ? Ss May25 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 16455 0.0 0.4 171564 79420 ? Ss Jun24 0:01 /usr/local/bin/php /home/user1/cron/sync2server.php user1 17474 0.0 0.5 182060 90016 ? Ss Jun18 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 20094 0.0 0.6 206636 114588 ? Ss Jun03 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 22555 0.0 0.7 213548 121476 ? Ss May30 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 24670 0.0 0.7 214572 122320 ? Ss May30 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 28200 0.0 0.7 220204 127988 ? Ss May26 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 30832 0.0 0.4 170284 78168 ? Ss Jun25 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 30837 0.0 0.4 170114 88508 ? Ss 23:20 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php user1 30848 0.0 0.4 120439 80770 ? Ss 12:20 0:00 /usr/local/bin/php /home/user1/cron/sync2server.php

If you see the STIME value mostly has started long time ago but it is still inside the process list. To kill all the older process which more that 24 hours, I use following command:

$ kill -9 `ps aux | grep php | grep sync2server | awk '$9 !~ /[0-9]:[0-9]/' | awk '{print $2}'`

Explanation:
Using ps command, we grep anything related to “php” and “sync2server” which is the specific process that we want to kill. The 4th argument is checking whether column no 9 (STIME) column has value which NOT in “number:number” format. Process which starts for more than 24 hours, STIME value will contains word for example “Jun23″ or “2010″. The 5th argument is actually print the value of column no 2 which is the PID to be killed.

To kill process which less than 24 hours, you can use following script:

LIMIT=43200 #limit on seconds = 12 hours PROCESS="php" #change to process u want to grep count=`ps aux | grep $PROCESS | awk {'print $9'} | wc -l` for ((i=1;i<=$count;i++)) do ptime=`ps aux | grep $PROCESS | awk {'print $9'} | head -$i | tail -1` psec=`date "+%s" -d "$ptime"` csec=`date "+%s"` exectime=$((csec-psec)) if [ $exectime -gt $LIMIT ] then pid=`ps aux | grep $PROCESS | awk {'print $2'} | head -$i | tail -1` /bin/kill -9 $pid fi done

Above scripts will try to find any PHP process which executed more than 12 hours and will kill it one by one. You can get the script to run as cron twice per day so it will automate your administration job.

Lets share if you have better idea. Cheers!

출처 : http://blog.secaserver.com/2011/08/kill-process-based-start-time-stime/

Piranha as Load Balancer (Direct Routing Method)

am currently working on a web cluster project using CentOS. In this project, I have 2 web servers running on Apache and mounted the same document root to serve the HTTP content. I also have 2 servers in front of it to become the load balancer and failover to increase high availability of the two-node web server cluster. The virtual IP will be hold by load balancer #1 with auto failover to load balancer #2.

You may refer to diagram below to get clearer picture:

I am using following variables:

All servers’ OS: CentOS 6.2 64bit
Web server #1: 192.168.0.221
Web server #2: 192.168.0.222
Load balancer #1: 192.168.0.231
Load balancer #2: 192.168.0.232
Virtual IP: 192.168.0.220

Load Balancer Server

1. All steps should be done in both servers unless specified. We will install Piranha and other required packages using yum:

$ yum install piranha ipvsadm -y

2. Open firewall ports as below:

• Piranha: 3636
• HTTP: 80
• Hearbeat: 539

3. Start all required services and make sure they will auto start if server reboot:

$ service piranha-gui start $ chkconfig piranha-gui on $ chkconfig pulse on

4. Run following command to set password for user piranha. This will be used when accessing the web-based configuration tools:

$ piranha-passwd

5. Turn on IP forwarding. Open /etc/sysctl.conf and make sure following line has value 1:

net.ipv4.ip_forward = 1

And run following command to activate it:

$ sysctl -p

Load Balancer #1

1. Open Piranha web-based configuration tools at http://192.168.0.231:3636 and login as piranha with respective password. We start with configuring Global Settings as below:

2. Then, go to the Redundancy tab and enter the secondary server IP. In this case, we will put load balancer #2 IP as the redundant server in case load balancer #1 is down:

3. Under Virtual Servers tab, click Add and enter required information as below:

4. Now we need to configure the virtual IP and virtual HTTP server to map into the real HTTP server. Go toVirtual Servers > Real Server and add into the list as below:

Make sure you activate the real server once the adding completed by clicking the (DE)ACTIVATE button.

5.  Now copy the configuration file to load balancer #2 to as below:

$ scp /etc/sysconfig/ha/lvs.cf 192.168.0.232:/etc/sysconfig/ha/

6. Restart Pulse service to apply the new configuration:

$ service pulse restart

You can monitor what is happening with Pulse by tailing the /var/log/message output as below:

$ tail -f /var/log/message

Load Balancer #2

No need to configure anything in this server. We just need to restart Pulse service to get affected with the new configuration changes which being copied over from LB1.

$ service pulse restart

If you see the /var/log/message, pulse in this server will report that it will run on BACKUP mode.

Web Servers

1. Since we are using direct-routing method, regards to your Apache installation, we also need to install another package called arptables_jf. Here is some quote from RedHat documentation page:

Using the arptables_jf method, applications may bind to each individual VIP or port that the real server is servicing. For example, the arptables_jf method allows multiple instances of Apache HTTP Server to be running bound explicitly to different VIPs on the system. There are also significant performance advantages to usingarptables_jf over the IPTables option.

However, using the arptables_jf method, VIPs can not be configured to start on boot using standard Red Hat Enterprise Linux system configuration tools.

We will instsall using yum:

$ yum install arptables_jf -y

2. Configure arptables_jf by executing following command:

In web server #1:

$ arptables -A IN -d 192.168.0.220 -j DROP $ arptables -A OUT -d 192.168.0.220 -j mangle --mangle-ip-s 192.168.0.221

In web server #2:

$ arptables -A IN -d 192.168.0.220 -j DROP $ arptables -A OUT -d 192.168.0.220 -j mangle --mangle-ip-s 192.168.0.222

3.  Save the arptables rules and make sure the service is started on boot:

$ service arptables_jf save $ chkconfig arptables_jf on

4.  Add the virtual IP address in the servers:

$ ip addr add 192.168.0.220 dev eth0

5. Since the IP cannot be started during sysinit (boot time), we can automatically start the IP after sysinit complete. Open /etc/rc.local using text editor:

$ vim /etc/rc.local

And add following line:

/sbin/ip addr add 192.168.0.220 dev eth0

Warning: Every time you restart your network service, please make sure to run step #4 to bring up the virtual IP in real server.

Done. You can now point your website to the virtual IP and you will see that the load balancer #1 will report as below:

$ ipvsadm -L   IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.220:http lblc -> 192.168.0.221:http Route 1 0 34 -> 192.168.0.222:http Route 1 0 19

출처 : http://blog.secaserver.com/2012/07/centos-configure-piranha-load-balancer-direct-routing-method/