Enable visual and email alerts on Ubuntu

Sometimes, one needs to alert on a condition that is checked using a background process (for example, running a crontab job): here we demonstrate a more user-friendly alternative on Ubuntu Linux desktop than scraping logs.

During Ubuntu 16.04 LTS installation, the partition for the /boot disk is created with a ridiculously tiny amount of space (236M on my system) – which is still plenty for one copy of the kernel and boot files, but runs quickly out of space when newer versions are added (old ones are not removed, even after a reboot).

Without keeping an eye on things, by the time Linux updates the kernel for the fourth time, all hell breaks loose and an ominous “/boot device has no space left” starts to pop up all the time.

In a github gist, we described how to use the Yagmail (Yet Another GMAIL client) API to send email alerts (or anything else, really) when certain events happen.

Following from that, we then decided to add the check on space used on /boot to our crontab list of jobs and hook it up with the alerting scripts, to have both a visual UI notification (in case someone is working on the box and notices such things), as well as an email notification (so things don’t fall through the cracks).

The above github gist contains all the code, here is a short breakdown of what it does:

Use Python and df to check on disk space

Initially, we thought about using a short shell script, but it turns out that writing it in Python is not only just as concise (using the sh module to get access to the df command) but also much more readable than any awk/sed acrobatics (really, if you are still using those, it’s time to realize it’s 2017, folks).

Here is a simplified version:

from sh import df, alert
# The actual script allows to pass this as an optional argument.
threshold = 70
disk_space = df("-h")
for line in disk_space.split('\n'):
    if line.endswith('/boot'):
        items = line.split()
        use_pct = items[-2]
        assert use_pct.endswith('%')
        used_space = int(use_pct[:-1])
        if used_space > threshold:
            alert("/boot is running out of space: {} used`({} bytes left)".format(
                    use_pct, items[-3]))
        exit(0)

Nothing earth-shattering here, just a very basic line-parsing gymnastic (sure, we could have used RegExes with the re module, but in this case it really would have been overkill: we don’t expect the output format of df to change in the, like, next 20 years, probably).

Use a “wrapper” shell script to abstract the invocation of alert.py

Alert “forced” by passing 50% as the threshold

The alert script that gets invoked is a shell wrapper around the alert.py one (see the github gist for details) and it contains something along the lines of:

declare -r ALERT_ICON="/.../icons/dialog-warning.png"
notify-send -i ${ALERT_ICON} "Failure Alert" "$1"
alert.py -f ${HOME}/.yagmail -m \
    "$1" -s "Failure Alert from `hostname`" \
    -d someone@gmail.com

By using Linux’s notify-send command, we also have a nice UI alert if anything goes wrong, which will get noticed if someone is using the desktop at the time the alert is triggered (this is run twice daily from a cronjob).

Run the check at regular intervals

This is really simple:

$ crontab -e

then add the following lines:

# Check twice daily there is enough space left in /boot.
30 8 * * * /home/.../scripts/check_boot.py 70
00 21 * * * /home/.../scripts/check_boot.py 70

which will alert if the disk usage exceeds 70% of available space.

Originally published at codetrips.com on March 29, 2017.