Simple unix performance reporting with tikz/pgfplot

I became tired using excel to create performance reports and decided to invest some time in creating a template in LaTeX using the fabulous pgfplots package. Since I couldn’t find a suitable example anywhere on the net I’ve decided to share mine here.

Each month I receive bundle of performance statistics from 15 vmware servers which I convert to data looking like this:

2011-12-02 02:00:00, 14.43
2011-12-02 04:00:00, 4.49
2011-12-02 06:00:00, 12.33
2011-12-02 08:00:00, 4.23
2011-12-02 10:00:00, 2.49
2011-12-02 12:00:00, 2.26
2011-12-02 14:00:00, 2.32
2011-12-02 16:00:00, 0.99
2011-12-02 18:00:00, 8.85
2011-12-02 20:00:00, 14.81
...

The extract above is cpu usage of the server every second hour for a month.

\documentclass{minimal}
\usepackage[danish]{babel}
\usepackage[utf8]{inputenc}
\usepackage{palatino}
\usepackage{filecontents}
\usepackage{tikz,pgfplots}
\usepgfplotslibrary{dateplot}
\definecolor{NNITRed}{RGB}{140,0,0}
\definecolor{pengreen}{RGB}{180,204,168}
\begin{document}
\pgfplotstableread[col sep=comma]{/home/regj/code/pyx/data/csv/server.cpud3}
\cputable
\begin{center}
\begin{tikzpicture}
\begin{axis}
[
title=CPU usage,
axis background/.style={fill,bottom color=gray!50,top color=white},
grid=major,
grid style={dotted,black},
width=15cm,
height=8cm,
date coordinates in=x,
date ZERO=2011-12-02 02:00:00,
xmin =2011-12-02 02:00:00,
xmax =2011-12-31 20:00:00,
xtick=,
ymax=100,
ylabel=CPU used \%,
xticklabel style={rotate=90, anchor=near xticklabel}
]
\addplot[blue] table[ y index = 1] from \cputable ;
\addlegendentry{CPU}
\end{axis}
\end{tikzpicture}
\end{center}
\end{document}

Which produces a graph (a very nice one at that IMO):

A simple inotify script to control rtorrent

So, I only want rtorrent to start when a torrent is available for download. I came up with the following script:

while inotifywait -qq -e create /path/to/torrentdir; do
  if pgrep rtorrent > /dev/null; then
     continue
  elif echo /path/to/torrentdir/* | grep torrent; then
     screen -d -m -S rtorrent rtorrent
done

Creating a fail2ban proxy filter

I recently discovered the usefulness of fail2ban whilst researching a way to block proxyheader.php requests on my LAMP server.

The request always came from the same location and the same IP’s:
58.218.199.147
58.218.199.250
58.218.199.227

A whois on these IP’s reveal that they’re based in China – Bejing to be exact – and are probably searching for proxy servers to implement in to the Great FW of China

I’m not sure, but their requests annoyed me and I wanted to block them.

I found this filter: Apache Proxy filter , but I found the jail.conf entry did not work and needed to be updated.

Here’s the filter and working jail.conf entry for fail2ban v0.8.4 …

Filter:
# Fail2Ban configuration file
#
# Author: James Roe
# Use in apache access logs
[Definition]
# Matches lines such as:
# 192.168.1.1 - - "GET http://www.infodownload.info/proxyheader.php ...
failregex = ^(?:(?![0-9\.]* -.*"[A-Z]* (/|.*HTTP/1\.[0-9]" (301|302))))
ignoreregex =

Place the filter in /etc/fail2ban/filter.d/apache-proxy.conf.

/etc/fail2ban/jail.local entry:
[apache-proxy]
enabled = true
filter = apache-proxy
action = iptables-multiport[name=PROXY,port="http,https",protocol=tcp]
logpath = /var/log/httpd/*access_log
maxretry = 0
findtime = 604800
bantime = -1

LVM2 partitioning on an Arch Linux Linode

The default image that comes with an Arch Linux linode install is at the time of writing not fully compatible with LVM2. Slash var, usr, and tmp failed when mounted as LVM2 at boot and some very weird errors when creating new LV’s.

I had to do the following to get it to work properly.

  1. Create a small partition for the standard linode Arch image at around 1024Mb.
  2. Create a disk using the remaining space.
  3. Boot the default partition and install base-devel and init-scripts.
  4. wget https://aur.archlinux.org/packages/li/linux-xen/linux-xen.tar.gz or another xen-kernel of your choice.
  5. tar xvf linux-xen.tar.gz && cd linux-xen && makepkg –asroot
  6. Add lvm2 to HOOKS and xen_blkfront to MODULES in /etc/mkinitcpio.conf.
  7. If compilation is successful install: pacman -U linux-xen-
  8. Install grub and edit /boot/grub/menu.lst as such:

    # (0) Arch Linux
    title Arch Linux [/boot/vmlinuz-linux]
    root (hd0)
    kernel /boot/vmlinuz-linux-xen root=/dev/xvda ro
    initrd /boot/initramfs-linux-xen.img

    Notice that root is on hd0. In Xen boot can not exist on separate partition AFAIK.
  9. In your main profile set kernel to: pv-grub-x86_64 and set “no” on Xenify distro in order to enable proper console access in Arch and change the following line in /etc/inittab:
    c0:12345:respawn:/sbin/agetty 38400 tty0 linux
    to
    c0:12345:respawn:/sbin/agetty 38400 hvc0 linux
  10. Reboot to test if your kernel works – if yes – boot into finnix recovery and create your LVM partitioning the way you see fit. Format your lv’s with ext3 or ext4.
  11. Then:
    mkdir /mnt/root
    mkdir -p /new/{var,srv,home,tmp,usr}
    mount /dev/xvda /mnt
    mount your lv's to /new/
    cp -a /mnt/root/var/* /new/var/ (do this for each new mount point)
    rm -rf /mnt/var/* (remove data from each of the directories you transferred to your new lv's)
  12. Adapt your /etc/fstab for your new mount points. Example:
    # /etc/fstab: static file system information.
    #
    #
    proc /proc proc defaults 0 0
    /dev/xvda / ext3 noatime,errors=remount-ro 0 1
    /dev/xvdb none swap sw 0 0
    /dev/mapper/vglino-lvvar /var ext3 noatime 0 1
    /dev/mapper/vglino-lvtmp /tmp ext3 noatime 0 1
    /dev/mapper/vglino-lvhome /home ext3 noatime 0 1
    /dev/mapper/vglino-lvsrv /srv ext3 noatime 0 1
    /dev/mapper/vglino-lvbck /data/backup ext3 noatime 0 1
    /dev/mapper/vglino-lvusr /usr ext3 noatime 0 1
  13. Reboot … profit.

How to unmount from a script that runs on the mount point that needs to be unmounted?

How would you go about unmounting a mount point from a script that runs on said mount point?

It is clearly not possible with a plain

umount /mnt

since this will immediately unmount the mount point, which will fail with the output

Device busy ... 

In case you need to cut the branch the script is sitting on use

umount -l

which entails:

-l Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is
not busy anymore. (Requires kernel 2.4.11 or later.)

Windows 7 SP1 upgrade – error 0x800f0a12 and the solution

So, I decided to upgrade my Windows 7 installation to SP1 using Windows update.

It did not work and returned the error 0x800f0a12, which means that the updater can not access the system partition (the 100MB partition that Win 7 creates during installation).

After some trial and a lot of error I thought it might be caused by my dual boot setup.

I have grub installed on a separate hard drive from where I boot Windows 7 – I use this setup the preserve the Windows 7 boot block – my hunch is that grub ignores the system partition altogether and that’s why Windows 7 can not see it.The solution was:

  1. Reboot machine.
  2. Choose boot menu in bios (ESC, F12, whatever your bios wants). In other words bypass GRUB.
  3. Choose the disk where Windows 7 is installed.
  4. Restart SP1 upgrade.

Wait an hour.

Done.

Get package size (Arch Linux) with an all inclusive awk command

A post on the Arch forums got me obsessing about running awk without piping anything in to the command i.e. running even external commands from within awk.

The post asked for a way to list installed packages by size and here is what I cooked up:

awk 'BEGIN{while (("pacman -Qi" |getline) > 0){ if ($0 ~ /Name/) {name=$3};\
{if ($0 ~ /Size/) {size=$4/1024;printf "%-25s %d Mb\n", name, size|"sort -k2 -n"}}}}'

It lists packages by size in rounded MB’s sorted by size.

Example output:

gnome-games               51 Mb
virtualbox                51 Mb
gimp                      60 Mb
perl                      61 Mb
texlive-bin               61 Mb
llvm                      62 Mb
gcc                       63 Mb
python                    64 Mb
chromium                  65 Mb
smbclient                 66 Mb
mysql                     67 Mb
python2                   68 Mb
kdelibs                   69 Mb
evolution                 70 Mb
samba                     73 Mb
xulrunner                 80 Mb
jre                       94 Mb
qt                        109 Mb
bin32-jre                 110 Mb
kernel26                  112 Mb
kdebase-workspace         113 Mb
wine                      198 Mb
tesseract                 212 Mb
texlive-core              258 Mb
openoffice-base           356 Mb
texlive-fontsextra        368 Mb
ghc                       649 Mb

Awk battery percentage one-liner

Here’s an awk one-liner to get battery percentage:

awk 'BEGIN{v1=0; v2=0} /rem/ {v1=$3}; /full/ {v2=$4};END {v3=v1*100/v2; printf "%d%\n", v3'} \
/proc/acpi/battery/BAT0/{info,state}

Copy 1000 files out of several millions using while loop.

I got this task where I was supposed to copy exactly 1000 small files from a folder containing ~2 million files.

This turned out to be quite a challenge. First of the find command just hangs when using it with millions of files and ls without the -f switch is very unresponsive.

This is how I solved it:

1. Collected all the file names:

ls -f > lotofiles.txt

2. Extracted 1000 file names:

tail -n 1000 lotofiles.txt > 1000files.txt

3. Ensured that the server I was copying the files to was set up with ssh keys. Important step unless you want to type in a password 1000 times!

4. Created a while loop:

while read file; do scp -p $file servername:/des/ti/nation/; done < 1000files.txt

If you want to avoid banners add a “-q” to your scp command – remember you’ll miss ssh errors as well.

Another way of doing this would be to tar the 1000 files so only one transfer was needed.

Netsh script to change to OpenDNS on the fly

Sometimes it easier to send a script to an IT illiterate than to explain how to do configure stuff in Windows. The script below replaces a ISP’s DNS adresses with OpenDNS entries for the windows user making it much easier for you:

@echo off
rem Adjust the IFACE variable below to the name of the network interface on your system.
set IFACE=Local Area Connection
echo Old DNS:
netsh interface ip show dns
netsh interface ip set dns "%IFACE%" static 208.67.222.222 > NUL
netsh interface ip add dns "%IFACE%" 208.67.220.220 > NUL
echo OpenDNS (Hopefully):
netsh interface ip show dns
echo This window will close in 10 seconds. Please check above if DNS entries have been changed.
rem for /L %%i in (1,1,10) do sleep 1 && (echo|set/p="%%i..") (unrem if there you have sleep.exe)
timeout /T 10