Pi Zero, 2p for scale, not included with prize.

Last week we started a competition to win a Pi Zero. We’ve had a small number of entries, half from school age people eligible to win, the other half from entertained techies. We’ve also been using this as a job filter for some time so we have a rich depth of existing answers.

The competition is very simple, our web-page generates a mathematical sum for you to work out the answer to, and in order to succeed you have to send us the answer within one second. It’s an anti-turing test – a person can’t do it but a computer can. When you’ve succeeded we ask you to send us the code. This gives two important things, a code sample that a candidate wrote and an idea of how long it took them to work out they should automate it.

A text-book answer from an experienced techie is about 15-30 minutes and delivers a short piece of code that fetches the web-page, finds the sum with a regexp or similar, works out the answer with eval, creates the response and sends it back to us. However, people are much more creative than that.

One excellent answer was a single line of shell script, which did the whole thing in bash, but even more cleverly searched the process list for the command that had been typed in, added the escaping back in and then mailed itself to us – a one liner that did the puzzle, recreated its source code and emailed itself to us.

Another excellent answer was someone who guessed that our code had a 32 bit roll-over bug in, tried options until we generated an answer bigger than 2^32 and fixed the otherwise text-book code to implement the bug on our side.

The absolute worst answer we’ve ever seen was someone whose CV listed them as a professional programmer with five years experience. After two whole days of typing answers into the website they finally worked out that the problem had to be automated. After three days of development they wrote a vast piece of Java code that was able to download the page and find the problem but it was only able to work out the answer if there were only three numbers and they all had to be added together. Instead of improving the code for the general case they put it in a loop and repeatedly called the page until by sheer luck they got a page their code could answer. Creative genius but not in a good way.

On to the entrants

So this is a difficult challenge for school age children and teenagers. Most of the entries came from older children 16 and up, and it’s clear that it was difficult and they had to learn new things specifically to solve this. PHP and Python were the preferred languages – the most novice friendly of all the tools available. We were very torn as to who should win. After lots of deliberation our runner up is this php entry from Nils in Germany who was also the first to submit,


// That was fun. There should be more conetsts like this.
// Sorry for the incredibly hacked together code...
<?php

$challenge =
file_get_contents(‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’);

$task = preg_replace(‘/.*

Evaluate /is’, ”, $challenge);
$task = preg_replace(‘/ \, .*/is’, ”, $task);

$id = preg_replace(‘/.*name=\”id\” value\=\”/is’, ”, $challenge);
$id = preg_replace(‘/\”.*/is’, ”, $id);

function calc_string( $mathString ){
$cf_DoCalc = create_function(“”, “return (” . $mathString . “);” );

return $cf_DoCalc();
};

$ans = calc_string($task);

$url = ‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’;
$data = array(‘answer’ => $ans, ‘id’ => $id);

$options = array(
‘http’ => array(
‘header’ => “Content-type: application/x-www-form-urlencoded\r\n”,
‘method’ => ‘POST’,
‘content’ => http_build_query($data),
),
);

$context = stream_context_create($options);
$result = file_get_contents($url, false, $context);

var_dump($result);

?>

Things we particularly like are that all the comments and code and email are in English to make it easier for us, even though it’s the authors second language.

Our winner though goes to 13 year old Nick. From a pure technical standpoint his code isn’t as good as Nils’ entry above, but through the comments it tells the story of slowly working his way towards the solution and appropriately credits the help he received – both personal and online.


#Mythic Beasts - Win a Pi Zero
#Written by Nick Lockhart from Chepstow, Wales, aged 13 3/4
#with help from an old fossil (his dad)
#You will need LXML for Python and Requests

from lxml import html
import requests
debugmode = 1

#get the page and parse all

elements into a Python list.
#For this purpose we should only get one element.
page = requests.get(‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’)
tree = html.fromstring(page.content)
paragraphs = tree.xpath(‘//p/text()’)

#Split out the question.
#First take out everything after the sum (space included)
#And then remove everything before the sum (space again included.)
#And convert to a string. Oddly, after the second time, we have to
reference the second element as there is a blank string in question[0]
#Finally, evaluate it.

question = paragraphs[0].split(‘ ,’)
question = question[0].split(‘Evaluate ‘)
question = question[1]
answer = eval(question)

if (debugmode == 1):
print(“The question was: ” + question)
print(“The answer should be: ” + str(answer))

#There’s a hidden input labeled “id”, which seems to be randomly generated.
#This is probably to track who’s submitting.
#We will also need to extract this.
#I learnt this piece of magic with help from StackOverflow. Thanks,
Mathias Muller!

secretkey = tree.xpath(‘//input[@name=”id”]/@value’)[0]

#That’s all we need to POST. Let’s generate a payload, send it off and
extract the response.
#The server expects the answer to be a string, so we convert it to a string.

payload = {“answer”:str(answer),”id”:secretkey}

if (debugmode == 1):
print payload

response =
requests.post(“http://sphinx.mythic-beasts.com/cgi-bin/job.pl”,
data=payload)
responsetree = html.fromstring(response.content)
print(responsetree.xpath(‘//p/text()’))

Of course the final comment to everyone who entered is if you ever need any kind of hosting, domain name or similar send us an email and include your entry number for a freebie / upgrade /discount. Secondly if you seek summer work or gap year employment, we’d invite you to get in touch and we guarantee that we’ll read your CV and take your application seriously.

Dear LazyWeb,

Yesterday we did a routine disk replacement on a machine with software RAID. It has two mirrored disks, sda and sdb with a RAID 1 partition with software RAID, /dev/md1 mirrored across /dev/sda3 and /dev/sdb3. We took the machine offline and replaced /dev/sda. In netboot recovery mode we set up the partition table on /dev/sda, then set the array off rebuilding as normal:

mdadm --manage /dev/md1 --add /dev/sda3


This expects to take around three hours to complete, so we told the machine too boot up normally and rebuild in the background while being operational. This failed – during bootup in the initrd, the kernel (Debian 3.16) was bringing up the array with /dev/sda3, but not /dev/sdb3, claiming it didn’t have enough disks to start the array and refusing to boot.

Within the initrd if I did:

mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3


the array refused to start claiming that it didn’t have sufficient disks to bring itself online, but if I did:

mdadm --assemble /dev/md1 /dev/sdb3
mdadm --manage /dev/md1 --add /dev/sda3


within the initrd it would bring up the array and start it rebuilding.

Our netboot recovery environment (same kernel) meanwhile correctly identifies both disks, and leaves the array rebuilding happily.

To solve it we ended up leaving the machine to rebuild in the network recovery mode until the array was fully redundant at which point the machine booted without issue. This wasn’t an issue – it’s a member of a cluster so downtime wasn’t a problem – but in general it’s supposed to work better than that.

It’s the first time we’ve ever seen this happen and we’re short on suggestions as to why – we’ve done hundreds of software RAID1 disk swaps before and never seen this issue.

Answers or suggestions in an email or tweet.

With today being Back To The Future day, it’s worth reflecting on two pieces of advice I received in the mid 1980s. The best piece of advice was definitely from the film:

‘If you put your mind to you, you can accomplish anything’.

the worst was my mother:

‘Stop playing on your Spectrum and go and do your piano practice’

I’m not certain this generalises, I think that largely your parents do give better advice than Hollywood script writers.

Unless you really like maths, you’re probably better off just looking at the pictures.

Today is Ada Lovelace day, where we celebrate the achievements of women in the traditionally male dominated fields of Science, Technology, Engineering and Maths.

Mythic Beasts came about as a side project of a bunch of students, most of whom studied at Clare College Cambridge. As a Cambridge student you receive supervisions, hour long tutorials in sets of two or three. You also live in the college for three years and some fellows of the college also have rooms in the same accommodation. As luck would have it, our director Pete was partnered with one of our other founders, Richard for supervisions in Physics, and in the second year they were jointly supervised by Professor Cathie Clarke and it turned out that her college room was directly opposite Pete’s in Clare Old Court.

This led to a slightly unusual arrangement, rather than everyone trekking into the department for supervisions they decided to hold them in Pete’s room at 8am, usually accompanied by a bacon sandwich and strong black coffee before heading off to lectures. Cathie was a superb teacher, neatly covering dynamics, orbits and effortlessly showing why practically the whole of spaceflight involves pointing your rocket motor in obviously the wrong direction in order to get to where you want to go. She also, neatly answered other questions including electro-magnetism that had stumped pretty much the whole year in Clare and all the supervisors in that subject, despite having had about sixty seconds notice and only half a cup of coffee.

Pete’s room was on the top right hand corner of this photograph, the bacon cooking kitchen roughly above the passageway.

There was a particularly memorable supervision, where Richard overslept and arrived very late to Pete’s room, anxious he’d missed the supervision. On waking Pete up, they jointly discovered the note on the door from Cathie apologising that she’d overslept and the supervision would need to be rescheduled. A co-incidence for which all parties were grateful.

So whilst her impressive CV gives away the huge publication list, professorship and that she’s the course coordinator for astrophysics; in person we had the privilege of knowing that she is also a superb prize-winning teacher, gifted researcher and somehow on top of all that, a lovely human being who occasionally oversleeps just like the rest of us.

Happy Incorporation Day to Us
Happy Incorporation Day to Us
Happy Incorporation Day to Mythic Beasts
Happy Incorporation Day to Us

Fifteen years ago today someone with a boring job processed the paperwork and Mythic Beasts Ltd sprang into existence as a legal entity.

We had existed informally for a bit longer than that, we had registered the mythic-beasts.com that April, and our shell server, sphinx, had been running for a while, although it wouldn’t be until early 2001 that we sent our first invoice.

As we all work on t’internet, it’s difficult to all meet in the same pub this Friday for a celebratory drink. That will have to wait until our next full company meetup in September. Slamming the bedroom door, staying in and watching Brazil seems a more apt way for a teenage company to celebrate its bureaucratic anniversary.

One of the Mythic Beasts, Rhosyn has written an article on filtering for trefor.net, a widely read blog on technology and networking.

The part we particularly like this this quote,

As a long standing customer of Mythic Beasts 
(shameless plug; outstanding service and support, 
so good that I switched companies recently to work 
for them)

When you visit a web page, you’ll often see the URL change as it loads.  For example, if you attempt to visit http://mythic-beasts.com you’ll end up at https://www.mythic-beasts.com .   This is achieved using HTTP redirects, a response from a server that tells your browser that the page it is trying to load has moved.

HTTP redirects come in two flavours:

Permanent (301)

This tells the client that the page requested has moved permanently, and crucially, if it wants to load the page again, it needn’t bother checking the old URL to see if the situation has changed. This is a good way of redirecting something that you never want to undo, for example, if you’re permanently moving a website from one domain to another.

Temporary (302)

As the name suggests, this tells the client that the page has moved, but only temporarily, so the client should continue requesting the old URL if it wants to load the page again. This is a good way of telling users that your site is down for maintenance, that they they don’t have enough credit to access a site, or of some other issue that is likely to change.

#makeitwrong

Getting this wrong can be a massive pain for your users. For example, Three use a permanent redirect if you’ve run out of credit on your data plan, or you’re trying to use tethering in the wrong country, or some other temporary problem.

So imagine what happens when you run out of data on your plan. You attempt to visit your favourite website, say, http://www.xkcd.com . Three tell you that that page has been replaced by http://tethering.three.co.uk/TetherNoProductPost. Permanently.

Now find a working internet connection, attempt to load http://www.xkcd.com, and find that your browser quite reasonably takes you straight to the Three fail page, even if you’re no longer using a Three connection. Shift+Reload doesn’t help, even restarting your browser may not help.

Three have told your browser that every page you visited whilst out of credit has moved permanently to their fail page.

Expiring permanent redirects

The example given above is very obviously a place where a temporary 302 redirect should be used, but webmasters are often encouraged to prefer 301s in the name of improving search rankings. 301 redirects allow you to tell search engines that your .co.uk site really is the same site as your .com site, thus accumulating all your google juice in the right place. They also save a small amount of time in loading the page by avoiding an unnecessary HTTP request.

Even when used legitimately, 301 redirects are obviously hazardous, as there’s no way to undo a permanent redirect once it’s been cached by a client.

The safe way to do a 301 redirect is to specify that it will expire, even if you don’t expect to ever change it. This can be done using the Cache-Control header. For example, the redirect that we issue for http://mythic-beasts.com includes the following header:

Cache-Control: max-age=3600

This tells clients that they can remember the redirect for at most one hour, allowing us to change it relatively easily at some point in the future. We use the mod_expires Apache module to create this header, which also produces an equivalent “Expires” header (the old HTTP 1.0 equivalent of Cache-Control).

.htaccess example

The above can be implemented using a .htaccess file as follows:

ExpiresActive on
ExpiresDefault "access plus 1 hour"
Redirect 301 / https://www.mythic-beasts.com/

This example uses mod_alias and mod_expires which may need enabling globally in your web server. In Debian, Ubuntu and similar distributions, this is done by running the following command as root:

a2enmod alias expires

mod_rewrite example

Redirects are often implemented using Apache’s mod_rewrite. Unfortunately, mod_expires doesn’t apply headers to RewriteRules, but mod_headers can be used instead:

RewriteRule ^.* http://www.mythic-beasts.com/ [L,R=301,E=limitcache:1]
Header always set Cache-Control "max-age=3600" env=limitcache

The RewriteRule is used to sent an environment variable which is used to conditionally add a Cache-Control header. Thanks to Mark Kolich’s blog for the inspiration.

Again, you may need to enable mod_rewrite and mod_headers on your web server:

a2enmod rewrite headers

Escaping 301 hell

Fortunately, if you’re unlucky enough to get caught by a broken 301 redirect, such as the one issued by Three, there is an easy way to get to the page you actually wanted: simply append a query string to the end of the URL. For example, http://www.xkcd.com/?foo=bar. Browsers won’t assume that the cached redirect is valid for this new URL and websites will almost always ignore unexpected query parameters.

2015-07-03 – Updated to add mod_rewrite example
2020-03-16 – Updated to note that the relevant modules may need enabling

Here at Mythic Beasts it’s fair to say that our staff hold a wide spectrum of political beliefs, but I think one thing we can all agree on is that all the major political parties have at least some irredeemably stupid policies (and possibly also that some of the minor parties only have stupid policies).

This makes voting for a political party a pretty depressing prospect. So, what about voting for an elected representative who will look after our interests?

Our founders reside in two constituencies with notable MPs: Witney and Cambridge.

The MP for Witney is notable for being the Prime Minister. The MP for Cambridge, Julian Huppert, is notable for being a Liberal Democrat and yet still being highly regarded by a large number of his constituents.

Now, if you want good data on whether your MP is any good or not, you should head over to the excellent They Work For You and find out what they’ve been up to in Parliament on your behalf.

But who wants good data when you can have some anecdotes? Let’s look at two issues that have got us wound up recently.

Firstly, the EU VAT MESS, which causes us an administrative burden far in excess of the value of the affected revenue.

Julian Huppert was very active on behalf of the constituents who contacted him on this issue (Mythic only got as far as a tweet…), including submitting written questions in parliament, which received a predictably useless response.

On the other hand, Paul wrote to David Cameron twice (the first letter went AWOL), and received only a hopeless response which completely failed to address any of the issues raised.

Secondly, banning secure encryption. As a hosting company, the ability to undertake transactions securely online is quite important to our everyday business (see previous notes).

The appalling jeering by other MPs, and the pathetic response given by Theresa May, to Julian Huppert’s questions asked in Parliament demonstrated the he was clearly one of the few MPs who actually grasped the implications of the proposal, rather just resorting to rhetoric that fuels the fear that terrorism relies on.

As for David Cameron, well, it’s his idea.

So what can we conclude from this? Not a lot, except that we’d probably be in a far better place if parliament were full of representatives who listened to and understood their constituents, rather than those who get in on the strength of a party political vote.

At UKNOF31 we presented a talk entitled Catastrophic Unplanned Success, a slightly rushed history of how some of the rapid scale-up of RaspberryPi from the point of view of the hosting provider, detailing some of the issues we’ve dealt with during their extremely rapid scale up, and attempting to educate the teenagers into a proper DDoS rather than the half-hearted ones they’ve tried so far.

https://indico.uknof.org.uk/getFile.py/access?contribId=5&resId=0&materialId=slides&confId=33

We believe this talk was videoed, we’ll put the video up here too once it’s published.

Some time ago we were forwarded a plea by Liz Upton who’s sort of famous on the internet for some sort of cheap computer, on behalf of World Possible, which said

This brings us to good news / bad news.  Last month we pushed through 5TB of
FTP traffic, and over 20TB of FTP traffic on the year.  That's great, about
700 RACHEL downloads - but our web host isn't as excited with our success
and cut us off yesterday.

Liz thought this was the sort of thing we might be able to help with. So we got in contact and we’ve set them up with one of our older inexpensive servers to act as a new host. As it’s an educational project that we’d like to support; we thought we’d donate some bandwidth to help out. Since it nicely coincided with a substantial bandwidth upgrade in our Cambridge data centre we’d put the service there.

So far they seem pleased!

which is handy because some of their other suppliers who pay Amazon rates for bandwidth were a little bit annoyed with them.