PiZero won!

December 10th, 2015 by
Pi Zero, 2p for scale, not included with prize.

Pi Zero, 2p for scale, not included with prize.

Last week we started a competition to win a Pi Zero. We’ve had a small number of entries, half from school age people eligible to win, the other half from entertained techies. We’ve also been using this as a job filter for some time so we have a rich depth of existing answers.

The competition is very simple, our web-page generates a mathematical sum for you to work out the answer to, and in order to succeed you have to send us the answer within one second. It’s an anti-turing test – a person can’t do it but a computer can. When you’ve succeeded we ask you to send us the code. This gives two important things, a code sample that a candidate wrote and an idea of how long it took them to work out they should automate it.

A text-book answer from an experienced techie is about 15-30 minutes and delivers a short piece of code that fetches the web-page, finds the sum with a regexp or similar, works out the answer with eval, creates the response and sends it back to us. However, people are much more creative than that.

One excellent answer was a single line of shell script, which did the whole thing in bash, but even more cleverly searched the process list for the command that had been typed in, added the escaping back in and then mailed itself to us – a one liner that did the puzzle, recreated its source code and emailed itself to us.

Another excellent answer was someone who guessed that our code had a 32 bit roll-over bug in, tried options until we generated an answer bigger than 2^32 and fixed the otherwise text-book code to implement the bug on our side.

The absolute worst answer we’ve ever seen was someone whose CV listed them as a professional programmer with five years experience. After two whole days of typing answers into the website they finally worked out that the problem had to be automated. After three days of development they wrote a vast piece of Java code that was able to download the page and find the problem but it was only able to work out the answer if there were only three numbers and they all had to be added together. Instead of improving the code for the general case they put it in a loop and repeatedly called the page until by sheer luck they got a page their code could answer. Creative genius but not in a good way.

On to the entrants

So this is a difficult challenge for school age children and teenagers. Most of the entries came from older children 16 and up, and it’s clear that it was difficult and they had to learn new things specifically to solve this. PHP and Python were the preferred languages – the most novice friendly of all the tools available. We were very torn as to who should win. After lots of deliberation our runner up is this php entry from Nils in Germany who was also the first to submit,


// That was fun. There should be more conetsts like this.
// Sorry for the incredibly hacked together code...
<?php

$challenge =
file_get_contents(‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’);

$task = preg_replace(‘/.*

Evaluate /is’, ”, $challenge);
$task = preg_replace(‘/ \, .*/is’, ”, $task);

$id = preg_replace(‘/.*name=\”id\” value\=\”/is’, ”, $challenge);
$id = preg_replace(‘/\”.*/is’, ”, $id);

function calc_string( $mathString ){
$cf_DoCalc = create_function(“”, “return (” . $mathString . “);” );

return $cf_DoCalc();
};

$ans = calc_string($task);

$url = ‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’;
$data = array(‘answer’ => $ans, ‘id’ => $id);

$options = array(
‘http’ => array(
‘header’ => “Content-type: application/x-www-form-urlencoded\r\n”,
‘method’ => ‘POST’,
‘content’ => http_build_query($data),
),
);

$context = stream_context_create($options);
$result = file_get_contents($url, false, $context);

var_dump($result);

?>

Things we particularly like are that all the comments and code and email are in English to make it easier for us, even though it’s the authors second language.

Our winner though goes to 13 year old Nick. From a pure technical standpoint his code isn’t as good as Nils’ entry above, but through the comments it tells the story of slowly working his way towards the solution and appropriately credits the help he received – both personal and online.


#Mythic Beasts - Win a Pi Zero
#Written by Nick Lockhart from Chepstow, Wales, aged 13 3/4
#with help from an old fossil (his dad)
#You will need LXML for Python and Requests

from lxml import html
import requests
debugmode = 1

#get the page and parse all

elements into a Python list.
#For this purpose we should only get one element.
page = requests.get(‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’)
tree = html.fromstring(page.content)
paragraphs = tree.xpath(‘//p/text()’)

#Split out the question.
#First take out everything after the sum (space included)
#And then remove everything before the sum (space again included.)
#And convert to a string. Oddly, after the second time, we have to
reference the second element as there is a blank string in question[0]
#Finally, evaluate it.

question = paragraphs[0].split(‘ ,’)
question = question[0].split(‘Evaluate ‘)
question = question[1]
answer = eval(question)

if (debugmode == 1):
print(“The question was: ” + question)
print(“The answer should be: ” + str(answer))

#There’s a hidden input labeled “id”, which seems to be randomly generated.
#This is probably to track who’s submitting.
#We will also need to extract this.
#I learnt this piece of magic with help from StackOverflow. Thanks,
Mathias Muller!

secretkey = tree.xpath(‘//input[@name=”id”]/@value’)[0]

#That’s all we need to POST. Let’s generate a payload, send it off and
extract the response.
#The server expects the answer to be a string, so we convert it to a string.

payload = {“answer”:str(answer),”id”:secretkey}

if (debugmode == 1):
print payload

response =
requests.post(“http://sphinx.mythic-beasts.com/cgi-bin/job.pl”,
data=payload)
responsetree = html.fromstring(response.content)
print(responsetree.xpath(‘//p/text()’))

Of course the final comment to everyone who entered is if you ever need any kind of hosting, domain name or similar send us an email and include your entry number for a freebie / upgrade /discount. Secondly if you seek summer work or gap year employment, we’d invite you to get in touch and we guarantee that we’ll read your CV and take your application seriously.

Testing failure: Raspbian

December 6th, 2015 by
Programmer art, just say no.

Programmer art, just say no.

If you’ve had a look at the Raspbian website today you’ll have noticed the big red !!!FAILOVER TEST!!! logo at the top right corner. That’s because today is officially unimportant for Raspberry Pi, whereas in three weeks time it will be officially very important. Historically Christmas day sees our highest traffic loads as people unwrap their new Raspberry Pis and try them out. The most critical things for us to worry about are some of the educational and getting started resources on the website, and Raspbian and the mirror director so people can download new packages for their existing Raspberry Pis.

The majority of the website has a relatively small amount of data, so pulling an image from backup and redeploying is a relatively quick operation. Raspbian however is a bit harder – it’s a big image with around 4TB of data.

So we picked today to schedule a failover of Raspbian from it’s normal dedicated server to a VM hosted in the Raspberry Pi cloud. This is aiming to check

  • Is the failover server up to date and does it work?
  • Is the failover setup fast enough to keep up with the traffic load?
  • Does every service successfully fail over?

So far we’ve had a very smooth operation, we’ve had to add a couple of missing packages that had been overlooked during setup and testing, but basically we did a DNS flip and the whole site moved over.

If you like to discover that your disaster recovery system works before you have a disaster, have a look at our Managed Services or get in touch – sales@mythic-beasts.com.

Win a PiZero

December 2nd, 2015 by
Pi Zero, 2p for scale, not included with prize.

Pi Zero, 2p for scale, not included with prize.

Thanks to a recent visit to Pi Towers, we’re in possession of a very difficult to get hold of Raspberry Pi Zero. Within Mythic Beasts we don’t have an immediate need for a Pi Zero, so we thought we’d give it away to someone more deserving. So here’s a competition.

Our recruitment page frequently foxes experienced programmers but teenagers often have little difficulty,

http://sphinx.mythic-beasts.com/cgi-bin/job.pl

This competition is open until 9th December at 17:00. In order to enter you must have been born on or after 1st September 1997. Send us a successful answer, and we’ll pick the one we like best and send the winner a Pi Zero.

People over the age of 18 will have to be satisfied merely with the respect of their peers and can go buy their own Pi Zero, for example from our customer Pi Supply.

Detailed Rules


SPONSOR

The Sponsor is Mythic Beasts Ltd, 103 Beche Road, Cambridge, CB5 8HX.

TERM

The Mythic Beasts Contest begins 2nd December 2015 at 17:00:00 UTC and ends 9 December 17:00:00 UTC. By submitting an Entry, each Entrant (or, where appropriate, the Entrant’s parent or legal guardian) agrees to the Official Rules presented here, and warrants that his or her Entry complies with all requirements set out in the Official Rules. This is a skill-based contest and chance plays no part in the determination of winners.

WHO MAY ENTER

The Contest is open only to individuals born on or after 1st September 1997. Employees of the Sponsor and their immediate family members (spouse, parent, child, sibling and their respective spouses, regardless of where they live) or persons living in the same households of such employees, whether or not related, are not eligible. CONTEST IS VOID WHERE PROHIBITED.

HOW TO ENTER

Visit http://sphinx.mythic-beasts.com/cgi-bin/job.pl and follow the instructions. When the challenge is complete submission details will be provided. You need an email address to receive a reply.

CONTEST PRIZES AND JUDGING

The prize is a Raspberry Pi Zero. This will be given to the best entry at the discretion of the judges.

The Sponsor reserves the right to take such steps as it deems necessary to verify the validity and originality of any Entry and/or Entrant (including an Entrant’s age, identity, address and authorship of the Entry), and to disqualify any Entrant who submits an Entry that is not in accordance with these Official Rules.

LICENCE

By entering the Contest, all Entrants grant an irrevocable, perpetual, royalty-free, worldwide non-exclusive licence to the Sponsor, to reproduce, distribute, and display their Entry.

LIMITATION OF LIABILITY

By entering this Contest, the Entrant (or, where appropriate, the Entrant’s parent or legal guardian) agrees to release, discharge, and hold harmless the Sponsor and its partners, affiliates, subsidiaries, advertising agencies, agents and their employees, officers, directors, and representatives from any claims, losses, and damages arising out of their participation in this Contest.

CONDITIONS

This Contest shall be subject to and governed by the laws of England and Wales.

If for any reason the Contest is not capable of running as planned for any cause beyond the control of Sponsor, Sponsor reserves the right, at its sole discretion, to cancel, terminate, or suspend the Contest. The Sponsor reserves the right, at its sole discretion, to amend the Official Rules at any time during the Contest.

Raspberry Pi Zero: Not executing a trillion lines of PHP

November 27th, 2015 by

A number of people noticed that Raspberry Pi had launched their $5 Pi Zero yesterday. We had advance warning that something was going to happen, even if we didn’t know exactly what. When the Pi2 launched we had some difficulties keeping up with comment posting and cache invalidation. We gave a very well received talk on the history and launch at the UK Network Operators Forum which you can see below.


Since then we’ve worked with Ben Nuttall to rebuild the entire hosting setup into an IPv6-only private cloud, hosted on one of our very large servers. This gives us :

  • Containment: One part of the site can’t significantly impact the performance of another.
  • Scalability: We can pull VMs into our public cloud and duplicate them if required.
  • Flexibility: We no longer have to have a single software stack that supports everything.

For the Pi 2 launch we sustained around 4500 simultaneous users before we really started struggling with comment posting and cache invalidation. So our new plan was to be able to manage over 5,000 simultaneous site users before we needed to start adding more VMs. This equates to around 1000 hits per second.

In order to do this, we need to make sure we can serve any of the 90% of the most common requests without touching the disks or the database; and without using more than 10ms of CPU time. We want to reserve all our capacity for pages that have to be dynamic – comment-posting and forums, for example – and make all the common content as cheap as possible.

So we deployed a custom script staticify. This automatically takes the most popular and important pages, renders them to static HTML and rewrites the webserver configuration to serve the static pages instead. It runs frequently so the cache is never more than 60 seconds old, making it appear dynamic.  It also means that we serve a file from filesystem cache (RAM) instead of executing WordPress. During the day we improved and deployed this same code to the MagPi site including some horrid hackery to cache popular GET request combinations.


 


Some very vague back-of-the-envelope calculations give us:

 


It’s fair to say that we exceeded our target of 5,000 simultaneous users,

 


Liz Upton was quite pleased:

 


Not to mention a certain amount of respect from our peers

 


If you deployed the blog unoptimised to AWS and just had auto-magic scaling, we’d estimate the monthly bills to be many tens of thousands of dollars per month, money that instead can be spent on education. In addition you’d still need to make sure you can effortlessly scale to thousands of cores without a single bottleneck somewhere in the stack causing them all to lie idle. The original version of the site (with hopeless analytics plugin that processed the complete site logs on every request) would consume more computer power than has ever existed under the traffic mentioned above. At this scale optimisation is a necessity, and if you’re going to optimise, you might as well optimise well.

That said, we think some of our peers possibly overstated our importance in the big scheme of things,

 


IPv4 is so last century

November 11th, 2015 by
A scary beast that lives in the Fens.

A scary beast that lives in the Fens.

Fenrir is the latest addition to the Mythic Beasts family. It’s a virtual machine in our Cambridge data centre which is running our blog. What’s interesting about it, is that it has no IPv4 connectivity.

eth0 Link encap:Ethernet HWaddr 52:54:00:39:67:12
     inet6 addr: 2a00:1098:0:82:1000:0:39:6712/64 Scope:Global
     inet6 addr: fe80::5054:ff:fe39:6712/64 Scope:Link
     UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

It is fronted by our Reverse Proxy service – any connection over IPv4 or IPv6 arrives at one of our proxy servers and is forwarded on over IPv6 to fenrir which generates and serves the page. If it needs to make an outbound connection to another server (e.g. to embed our Tweets) it uses our NAT64 service which proxies the traffic for it.

All of our standard management services are running: graphing, SMS monitoring, nightly backups, security patches, and the firewall configuration is simpler because we only need to write a v6 configuration. In addition, we don’t have to devote an expensive IPv4 address to the VM, slightly reducing our marketing budget.

For any of our own services, IPv6 only is the new default. Our staff members have to make a justification if they want to use one of our IPv4 addresses for a service we’re building. We now also need to see how many addresses we can reclaim from existing servers by moving to IPv6 + Proxy.

Rebuilding Software RAID 1 refused to boot

October 30th, 2015 by

Dear LazyWeb,

Yesterday we did a routine disk replacement on a machine with software RAID. It has two mirrored disks, sda and sdb with a RAID 1 partition with software RAID, /dev/md1 mirrored across /dev/sda3 and /dev/sdb3. We took the machine offline and replaced /dev/sda. In netboot recovery mode we set up the partition table on /dev/sda, then set the array off rebuilding as normal:

mdadm --manage /dev/md1 --add /dev/sda3

This expects to take around three hours to complete, so we told the machine too boot up normally and rebuild in the background while being operational. This failed – during bootup in the initrd, the kernel (Debian 3.16) was bringing up the array with /dev/sda3, but not /dev/sdb3, claiming it didn’t have enough disks to start the array and refusing to boot.

Within the initrd if I did:

mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3

the array refused to start claiming that it didn’t have sufficient disks to bring itself online, but if I did:

mdadm --assemble /dev/md1 /dev/sdb3
mdadm --manage /dev/md1 --add /dev/sda3

within the initrd it would bring up the array and start it rebuilding.

Our netboot recovery environment (same kernel) meanwhile correctly identifies both disks, and leaves the array rebuilding happily.

To solve it we ended up leaving the machine to rebuild in the network recovery mode until the array was fully redundant at which point the machine booted without issue. This wasn’t an issue – it’s a member of a cluster so downtime wasn’t a problem – but in general it’s supposed to work better than that.

It’s the first time we’ve ever seen this happen and we’re short on suggestions as to why – we’ve done hundreds of software RAID1 disk swaps before and never seen this issue.

Answers or suggestions in an email or tweet.

If you put your mind to it

October 21st, 2015 by



With today being Back To The Future day, it’s worth reflecting on two pieces of advice I received in the mid 1980s. The best piece of advice was definitely from the film:

‘If you put your mind to you, you can accomplish anything’.

the worst was my mother:

‘Stop playing on your Spectrum and go and do your piano practice’

I’m not certain this generalises, I think that largely your parents do give better advice than Hollywood script writers.

IPv6 Graphing

October 15th, 2015 by
it's a server graph!

it’s a server graph!

One of the outstanding tasks for full IPv6 support within Mythic Beasts was to make our graphing server support IPv6 only hosts. In theory this is trivial, in practice it required a bit more work.

Our graphing service uses munin, and we built it on munin 1.4 nearly five years ago; we scripted all the configuration and it has basically run itself ever since. When we added our first IPv6 only server it didn’t automatically get configured with graphs. On investigation we discovered that munin 1.4 just didn’t support IPv6 at all, so the first step was to build a new munin server based on Debian Jessie with munin 2.0.

Our code generates the configuration file by printing a line for each server to monitor which includes the IP address. For IPv4 you print the address as normal, 127.0.0.1, for IPv6 you have to encase the address in square brackets [2a00:1098:0:82:1000:0:1:1]. So a small patch later to spot which type of address is which and we have a valid configuration file.

Lastly we needed to add the IPv6 address of our munin server into the configuration file of all the servers that might be talked to over IPv6. Once this was done, as if by magic, thousands of graphs appeared.

Professor Cathie Clarke, Ada Lovelace day

October 13th, 2015 by

Unless you really like maths, you’re probably better off just looking at the pictures.

Today is Ada Lovelace day, where we celebrate the achievements of women in the traditionally male dominated fields of Science, Technology, Engineering and Maths.

Mythic Beasts came about as a side project of a bunch of students, most of whom studied at Clare College Cambridge. As a Cambridge student you receive supervisions, hour long tutorials in sets of two or three. You also live in the college for three years and some fellows of the college also have rooms in the same accommodation. As luck would have it, our director Pete was partnered with one of our other founders, Richard for supervisions in Physics, and in the second year they were jointly supervised by Professor Cathie Clarke and it turned out that her college room was directly opposite Pete’s in Clare Old Court.

This led to a slightly unusual arrangement, rather than everyone trekking into the department for supervisions they decided to hold them in Pete’s room at 8am, usually accompanied by a bacon sandwich and strong black coffee before heading off to lectures. Cathie was a superb teacher, neatly covering dynamics, orbits and effortlessly showing why practically the whole of spaceflight involves pointing your rocket motor in obviously the wrong direction in order to get to where you want to go. She also, neatly answered other questions including electro-magnetism that had stumped pretty much the whole year in Clare and all the supervisors in that subject, despite having had about sixty seconds notice and only half a cup of coffee.

Pete’s room was on the top right hand corner of this photograph, the bacon cooking kitchen roughly above the passageway.

There was a particularly memorable supervision, where Richard overslept and arrived very late to Pete’s room, anxious he’d missed the supervision. On waking Pete up, they jointly discovered the note on the door from Cathie apologising that she’d overslept and the supervision would need to be rescheduled. A co-incidence for which all parties were grateful.

So whilst her impressive CV gives away the huge publication list, professorship and that she’s the course coordinator for astrophysics; in person we had the privilege of knowing that she is also a superb prize-winning teacher, gifted researcher and somehow on top of all that, a lovely human being who occasionally oversleeps just like the rest of us.

iOS 9 and SSL

September 28th, 2015 by
We're still installing iOS9 for testing reasons onto this Apple Device

We’re still installing iOS9 for testing reasons onto this Apple Device

tl;dr iOS9 applications only work with the newest SHA-256 certificates. If your iOS9 application or website is showing certificate errors and you’d like some help, contact support@mythic-beasts.com

iOS9 was recently released which brings a number of changes. In addition to the widely publicised changes about IPv6 (iOS9 prefers IPv6 and all applications in the App Store must function without issue on an IPv6 only network), Apple has forced obsolescence of older types of SSL certificate.

SSL certificates use hashing functions to provide security. The Secure Hash Algorithm 1 (SHA-1), was published by the NSA in 1995 as the standard for secure authentication. The first theoretical attacks were shown in 2005 leading to a recommendation in 2010 that we abandon SHA-1 and move to SHA-256. In 2014 Google put a sunset date for SHA-1 of December 2016 – if your website trusts an SHA-1 certificate past this date Chrome refuses to regard your site as secure.

With iOS9, Apple pulled the date at which everyday software stops working with SHA-1 forward. If your website or application is secured with a SHA-1 certificate, iOS9 gives warnings and errors. The fix is easy, we can provide or re-issue your existing certificate with an iOS9 compatible – and more importantly more secure – SHA-256 certificate.