We went to Electromagnetic Field 2024 as a silver sponsor. Whilst there we found a lot of fantastic fun things and missed a vast number of others. The really amazing part of the festival was the massive variety of things the participants brought with them. Lock picking and blacksmithing courses were available. Geodesic domes were very popular and courses on how to build them were at the Maths Village. At least one dome integrated with the API from the bar, so the lighting changed colour based on what type of drinks were currently being ordered.

If you thought that ethernet and IP was a bit too modern there was a fully functioning DECT cordless phone network and you could access the live schedule information over ViewData (Prestel/Minitel).

We missed at least 98% of the talks. Fascinating ones we did see included a comprehensive explanation of the attempt to backdoor ssh with xzutils by Dr Matthew Garrett. Dr Matthew Bothwell gave a guide to Astrophysics for Supervillains covering things like ‘what happens if I crash the moon into the earth?’ (answer: you get a moon). Tim Hunkin of the Secret Life of Machines gave a short history of electric shocks and Ian B Dunne played the Theramin and musical saw. Much to our surprise, this was quite listenable.

Sadly the magic smoke came out of the Tesla Coil before the end so we had to make do with fire and lasers in additional to a traditional light show around the DJ area in the Null zone.

But this is a short summary, there was a fantastic kids creche, swap shop, night market for created things, crochet, hand built guitars, paper rockets, active satellite tracking, a 5km run, a fully stocked bar, a manual version of flappy bird to play and and and [approximately 100,000 further words cut to keep this post merely far beyond reasonable]

The now notorious swap shop gets a special mention. Not only did it have extremely dangerous materials like Linux install CDs from the late 1990s that may permanently corrupt young minds but some things that are rather harder to get hold of.

It has a direct entry on wikipedia and is going to cause a lot of festivals to have to update their terms and conditions to prohibit bringing radioactive materials to the site. Fortunately an attendee was familiar with safe disposal procedure and quickly removed the offending sources.

The last and arguably best thing at Electromagnic Field was an incredibly secretive project, the Great Camp Hexpansion Question (GCHQ.net). This was a series of locations (mostly, but not all static) where you could plug the quest markers into the official badge which would record you’d found them. A cross between a scavenger hunt and a technology preview it encouraged wandering and looking around the whole camp to find many cool things that were tucked away.

We asked the organisers if this was an official GCHQ sanctioned project. They said no. But that’s what you’d expect GCHQ would say.

We like working with talented people be they staff, customers or suppliers. That’s why we give discounts to people who can navigate our jobs challenge even if they don’t want to work for us.

Occasionally we’ve drafted in Gytha Lodge to help us copy write various articles and turn a jumble of thoughts into a coherent and interesting article.

Formerly an aspiring author, her full title is now Richard and Judy book club pick and Sunday Times bestselling author, Gytha Lodge.

We’re also pleased to report that she took our advice on her first book seriously and the new book starts with a murder being watched over a webcam.

Once upon a time @thegyth copy edited material on our blog. Since then she's been a little bit busy publishing her debut novel, which despite being part set in the 1980s doesn't contain a single Sinclair Spectrum or BBC Micro reference. #disappointing https://t.co/wgfmWEsRWa

— Mythic Beasts (@Mythic_Beasts) March 21, 2019

OpenWRT is awesome, as it allows you to run proper Linux tools on your home router. I’m currently using a very old, underspecced TP Link box, with 32MB of RAM, but just 4MB of flash storage. This is just enough to get what I need installed, but one thing I’ve always wanted to do is use iftop to quickly see what’s using all the bandwidth. Unfortunately iftop, with its dependencies on libpcap and libncurses, just won’t fit into a 4MB image.

I recently stumbled across opkg’s install-to-RAM option, allowing me to use the 32MB of RAM to install the package, with the minor and obvious downside that it gets uninstalled when the router gets rebooted. For something like iftop, which is used for ad-hoc diagnostics, this isn’t a big issue.

Installing to RAM puts the packages under /tmp, so a little effort is required to make sure that libraries and other resources can be found. I now have the following shell script which installs iftop if it isn’t already, sets some environment variables and invokes iftop:

#!/bin/sh

if [ ! -f /tmp/usr/bin/iftop ] ; then
  opkg update
  opkg install -d ram iftop
fi

export LD_LIBRARY_PATH=/tmp/usr/lib
export TERM=xterm
export TERMINFO=/tmp/usr/share/terminfo/

/tmp/usr/bin/iftop $@

Fortunately I do have enough free space on flash storage to store the above script.
Obviously a similar approach could be used with other packages that are only needed “on demand”.

(title shamelessly stolen from Tom Steinberg, MySociety founder from his tribute to Chris, Mythic Beasts founder who died ten years ago).

Lots of people have been excited recently about this script, which allows you to remotely reinstall a Linux system with a different version of Linux by giving you a shell in a ramdisk and letting you reinstall the operating system from there.

Chris did this the hard way. Back in 2005 I remember being asked to code review ‘evil.c’, a script that allocated a lot of RAM (800MB!), compressed a filesystem into it, then uncompressed it back to the disk. On reboot it should come up with Debian instead of FreeBSD that it had earlier. It’s really very important not to swap during this process.

Amazingly it worked, and the first test was on a remote box and it saved us a data centre visit. Here’s the code in its full glory.

#include <sys/types.h>

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <zlib.h>

#include <sys/mman.h>
#include <sys/reboot.h>

#define SIZE        ((size_t)896058269)

#define E(...)      fprintf(stderr, __VA_ARGS__)
#define die()       do { fprintf(stderr, "%s; aborted\n", strerror(errno)); exit(1); } while (0)
#define zdie()      do { fprintf(stderr, "%s; aborted\n", Z.msg); exit(1); } while (0)

int main(void) {
    unsigned char *buf, *outbuf, *p;
    int fd;
    FILE *fp;
    z_stream Z = {0};
    unsigned int nin = 0, nout = 0;

    E("size = %lu\n", (unsigned long)SIZE);

    E("open /dev/amrd0... ");
    if (-1 == (fd = open("/dev/amrd0", O_RDWR | O_DIRECT)))
        die();
    E("done\n");
    close(fd);

    E("allocate file buffer... ");
    if (!(buf = malloc(SIZE)))
        die();
    E("done\n");

    E("allocate write buffer... ");
    if (!(outbuf = malloc(1024 * 1024)))
        die();
    E("done\n");

    E("lock into memory... ");
    if (-1 == mlockall(MCL_CURRENT | MCL_FUTURE))
        die();
    E("done\n");

    E("open file... ");
    if (!(fp = fopen("/usr/bitter-first-2100M-of-sda.gz", "rb")))
        die();
    E("done\n");

    E("read file... ");
    p = buf;
    while (nin < SIZE) {
        size_t n;
        n = fread(p, 1, 262144, fp);
        if (n == 0)
            die();
        nin += n;
        p += n;
        E("\rread file... %.2f%% ", 100 * (float)nin / (float)SIZE);
    }
    E("done\n");

    fclose(fp);
    E("zlib version = \"%s\"\n", zlibVersion());

    /* Now we need to walk through the buffer decompressing it into the
     * write buffer, then writing the results to the device. */
    E("initialise inflate object... ");
    Z.next_in = buf;
    Z.avail_in = SIZE;
    if (Z_OK != inflateInit2(&Z, 15 + 32))
        zdie();
    E("done\n");

    while (nout < 2100) {
        int i;
        size_t N;

        Z.next_out = outbuf;
        Z.avail_out = 1024 * 1024;
        i = inflate(&Z, 0);
        if (i != Z_OK && i != Z_STREAM_END)
            zdie();
        if (Z.next_out != outbuf + 1024 * 1024) {
            fprintf(stderr, "\ndidn't get 1MB of output\n");
        }

        /* this is where we'd write the data */
        N = 0;
        p = outbuf;
        while (N < 1024 * 1024) {
            ssize_t n;
            do
                n = write(fd, p, 1024 * 1024 - N);
            while (n == -1 && errno == EINTR);
            if (n == -1)
                die();
            N += n;
            p += n;
        }

        ++nout;
        fprintf(stderr, "\r%d / 2100 MB", nout);
    }

    fprintf(stderr, "\n");

    /* this is where we reboot */
    reboot(RB_NOSYNC);

    E("we should have rebooted by now -- probably best to assume we're completely\n"
      "screwed at this point\n");

    return 0;
}

So last week we built a fort from some old customer servers. Sometimes, though, it’s important to just try a little bit harder.

Hip Hop is not only a style of music, but also the name of a virtual machine written by Facebook which compiles PHP Just In Time to make it go quickly.

Now we receive lots of unsolicited advice about how to run a not very popular wordpress blog and cope with the volume of traffic. Usually this involves ripping and replacing the entire infrastructure from a standard Linux/Apache/MySQL/PHP stack to something different (Nginx/MariaDB/PostgreSQL) which may not even be able to run WordPress at all (e.g. node.js).

At Mythic Beasts we like to understand what we’re doing, rather than blindly installing Magic Go Faster Solution Number 7. So we set up a test 2GB dual core virtual machine, that runs WordPress and a selection of popular plugins ( WordPress SEO, Akismet, Safe Report Comments, Liveblog, Facebook, Yet Another Related Posts Plugin, WordPress Supercache and Jetpack, no endorsement implied). Then we benchmarked with siege and managed the following results.

Apache/mod_php : 5.10 trans/sec

and when you turn supercache on and serve cached pages you get

Apache/mod_php/supercache : 873.50 trans/sec

So this gives us two scenarios, pages which we have to generate content for which can easily cause load issues, and pages served from supercache in which our VM is fast enough for all practical purposes and will easily weather even very big traffic spikes from news websites or television adverts.

Now, it’s very popular to tell us to use Ngnix as it’s faster than Apache. Is it though?

Nginx/php-fpm: 5.70 trans/sec
Nginx/php-fpm/supercache: 2230.58 trans/sec

Wow! Nginx is three times quicker than Apache at serving cached pages. This is amazing, but not very helpful. It means when our webserver is serving pages really quickly, we serve pages at three times really quickly, but when we’re generating pages on demand, it’s about 10% quicker. That’s not very special and doesn’t justify a rip and replace of the whole installation for a 10% performance improvement.

A quick look at the VM during the testing tells us that the bottleneck is executing the PHP code which creates WordPress pages. The choice of webserver is basically irrelevant; almost all the server time is spent executing PHP and reading data from the database.

Enter HipHop Virtual Machine.

This is nothing to do with the HipHop Virtual Machine. But we like tea and Banging Tunes

It has one focus, to execute PHP quickly for Facebook. Facebook have a lot of servers and spend hundreds of millions to billions per year on servers and data centres. A 50% performance improvement in PHP saves them huge sums of money in data centres and servers alone, so it’s clearly worth them trying to optimise as much as possible.

Here’s what happens with Apache/Nginx running HHVM.

Apache/HHVM :           35.93 trans/sec
Apache/HHVM/supercache: 928.70 trans/sec
Nginx/HHVM :            33.78 trans/sec
Nginx/HHVM/supercache : 2137.67 trans/sec

This is a huge improvement for non cached pages – seven times faster. Cached pages are bottlenecked in the webserver so it makes minimal difference, but they were already so fast we weren’t worried about them. Again Apache/Nginx are still pretty much the same speed for generated pages, we’re still dominated by the code execution time but a seven fold performance improvement is worth seriously considering.

All I need to do now is see if I can find someone with a very busy WordPress site and a million complaining users who would like to test it to see if it’s really as good as the lab tests suggest it might be.

---

Very sorry to hear the news that Big Bank Hank who co-wrote the first ever hit Rap track Rappers Delight died earlier this week from kidney complications related to cancer.

You see, he was six foot one, and he was tons of fun