glibc 0-day exploit (GHOST), how we’re handling it
I would like to introduce our all new female GHOSTbusting team to tenuously tie in with a new Hollywood movie and gratuitously include a cool staff photo in this blog post, and for marketing reasons I’m going to ignore the reality that Toby did all the updates for GHOST.
Qualys found during a code audit a buffer overflow exploit for gethostbyname() in glibc which they’ve named GHOST. This means that any internet facing software that can be persuaded to do a DNS lookup is potentially vulnerable. To a first approximation that’s everything that’s listening on an internet socket.
The details are in CVE-2015-0235. Note this explains quite comprehensively how to exploit the vulnerability so we are expecting active exploitation to have already started.
The vulnerability was announced at 16:30 on Tuesday, at 16:40 the first ticket was opened in our queue automatically. We started reviewing the information shortly thereafter and deployed the updated packages to our shared hosting servers Tuesday evening. This gives a short window to discover any critical issues with the new packages before we start deploying updates to our managed hosting customers.
At 8:30am on Wednesday, we emailed every managed customer running vulnerable code (which is almost but not quite all of them) explaining the issue and indicating we’d be applying the patches immediately unless otherwise instructed not to. Giving customers a short window to reply before going ahead (some are automatically deploying via Puppet and don’t want us to update for them) we then applied the updates to the customer servers, which involved very brief interruptions to listening services as they restarted.
Subsequently spot auditing some customer machines indicates that the glibc update via the package manager may not have restarted every vulnerable process. We’re now writing some audit tools to check for missing service restarts. Tomorrow morning at 6am, our reporting package will report in lots of data about the status of all our managed customer machines including the complete process list and complete list of listening services, so on our reporting box we can do a complete audit for every listening process that hasn’t been restarted in the last 24 hours and investigate and fix where necessary.
If you aren’t a managed hosting customer of Mythic Beasts we implore you to update your systems as soon as possible, we strongly expect that someone is going to build a very big denial of service botnet very quickly from this vulnerability. If you have no idea how to update and audit your server please get in contact with us at support @ mythic-beasts.com even if you’re not hosted with Mythic Beasts.