HowTo's

Script to put order to the massacres performed by the OOM Killer

Script to automate the adjustment of the score of OOM Killer for certain processes, giving more weight to some and saving others.

Script para automatizar el ajuste del score del OOM Killer para ciertos procesos, dandole mas peso a algunos y salvando otros.

Not so long ago I had to recover my VPS via remote console, After checking /var/log/messages I realized that the Out of Memory Killer (OOM Killer) kicked some PID but after the OS ran out of memory . I decided to take a deeper look at the situation and quickly realized that it was a mixture of elements that drove my VPS to this state, Google, Yahoo and Yandex crawling this blog as well as a forum for car enthusiast that I also host here. The script to download torrents was running a long with the torrent tracker and someone was doing doing some flood at the IRC channel on freenode where I had left irssi connected.

The OOM Killer has an algorithm that decides which processes are best to kill but one can manipulate these values ​​and prioritize candidates so that the OOM Killer takes care of this processes first. To automate this task I made a script to help us, this is an early version and a really simple one::
for programas in SCREEN irssi top
do
for pid_of_oomk_candidate in `pidof -x $programas`
do
echo 10 > /proc/$pid_of_oomk_candidate/oom_adj
done
done

As you see the above script increments by 10 to OOMK score of the the processes that I care the less to survive when I run out of memory while the next one does the exact oposite, It will try to prioritize other processes over these ones, meaning that the other processes will be first in the list of candidates to be terminated, lets take a look at this script:
for programas in mysqld portsentry iptables
do
for pid_of_oomk_candidate in `pidof -x $programas`
do
echo -5 > /proc/$pid_of_oomk_candidate/oom_adj
done
done
echo -17 > /proc/`pidof -s sshd`/oom_adj

In this one mysql portsentry and iptables will have five points of advantage over other candidates and if you look at the last line we are making sure that sshd is not a candidate under any circumstances.

Automate the score adjustment of the OOM Killer

Now lets unify these two scripts and give it a little more flexibility, I'm sure there are people who prefer to assign irssi a value different than the one of screen. The first thing we will need is a configuration file, Lets created it in /etc/candidatos_oomk.conf and the contents should be something like this:
# Programas que queremos sacrificar, entre mas alto el valor
# mas peso tienen en la lista de candidatos.
irssi 4
SCREEN 3
smtpd 2
apache2 1
#
# Programas que queremos salvar, estos tendrán valores negativos
portsentry -2
mysqld -5
sshd -17

I named the script as oomk_adj_candidatos.sh and it would look something like this:
#!/bin/bash
CONFIGURACION="/etc/candidatos_oomk.conf"
while read programas
do
proceso=`echo $programas |grep -v \# | grep [0-9]$| awk '{print $1}'`
ajuste=`echo $programas |grep -v \# | grep [0-9]$| awk '{print $2}'`
if [ -z "$proceso" ];
then
continue
else
echo $proceso
for pids_proceso in `/bin/pidof $proceso`
do
echo " echo $ajuste > /proc/$pids_proceso/oom_adj"
done
fi
done < $CONFIGURACION

The script is simple, it first read the configuration file line by line, it then separates the name of the process form the adjustment value and we save those values in $proceso and $ajuste. We then evaluate the variable $proceso to check if it is empty, and if so, continue with the next line, otherwise continue processing the same line. We get the PID of the process using pidof and for each PID we write the value of $ajuste on its oom_adj file within /proc.

You can place this script in your /etc/rc.local or a cronjob and remember that if you want to see the list of candidates and see how your scrores or know a little more about it, can consult this article: The OOM Killer: handling candidates.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comment moderation is enabled. Your comment may take some time to appear.