Covid 19 update
Covid-19 has dramatically changed life in the UK, and the lock-down announced on Monday has lead to further changes to how we, and our customers, are operating. This page provides an update to our previously announced Covid-19 plan.
We’re happy to report that our staff member who had some of the symptoms of covid-19 is now fully recovered and has returned to our team after a few days of rest.
Data centre access changes
Our data centre suppliers have altered their operation: 24/7 walk in access is now prohibited and every visit needs booking and justification. The “remote hands” service remains available, but at a reduced capacity as they’ve moved as many staff as possible to home working to minimise their risks.
Equinix, who supply two of our core facilities, have completely closed their facilities in Italy, Germany, France and Spain to customers. Changes are only possible via their remote hands service in those countries. We have a significant amount of equipment in two Equinix facilities in London and Amsterdam (LD8 and AM5).
Since this announcement on Sunday, we have been anticipating a similar closure being applied to London. This has now been announced and will be disruptive for us, as our normal operating procedures rely on being able to move spare equipment easily between the London data centres where we have a presence. We have been taking steps to increase the spare hardware that we have at each of our London sites in order to mitigate the impact of this when it comes into force on Tuesday 31st March.
The data centres have well-considered policies to reduce risk, and to handle a confirmed case within the facility with rapid quarantine and deep clean procedures. No access is allowed to customers showing symptoms, and all customers’ temperatures are measured before entry to the facilities. Their operations are also robust in the event they need to manage the facility remotely for a period of time.
We’ve also altered our own policy for data centre access. Customer access to our data-centre space is suspended until further notice, including for documentation-related audits (e.g. ISO27001 compliance). This should have a minimal impact; we only allowed accompanied access previously and visits have always been exceedingly infrequent.
Items shipped to us will be quarantined for 24 hours before opening. Cardboard will self-decontaminate in 24 hours.
Staff members may not meet in a data centre unless it is specifically for a piece of work that requires two people for safety reasons (typically very heavy server deployment), and only if is to maintain an essential service. Staff members may not visit multiple key data centres in a single visit to minimise the risk of transmission between key sites, and may not visit if they are showing symptoms. Data centre visits are being minimised to reduce infection risk. This may limit the range of dedicated servers we are able to provision, and we have decided to stop offering Mac Minis with OS X due to the difficulty of provisioning them remotely.
Customer support
Unsurprisingly, a wholesale shift of the UK to remote working has a significant impact on all kinds of online systems, which are now critical for day to day operations. We’re supporting existing customers to make this transition, as well as provisioning new orders for services that now need to be in the cloud.
We run a system for POhWER that is used by all their advisors to track their cases. This is a critical system; if it’s offline, hundreds of people are unable to work. We maintain this as an active/standby pair split across two of our data centres.
“You’re supporting us to enable our vital work with the most vulnerable people in society to continue in these very trying times and, through your swift upgrade actions, our new fully remote working model is delivering the information, advice and advocacy our clients depend on.”
Sandra Black, Head of Training, Risk and Quality at POhWER.
The shift to remote working means that usage on their system has approximately doubled in ten days and has started to see performance limitations. We identified the bottleneck and proposed a cost-effective upgrade combined with some configuration improvements. We then made staff available to apply the changes in an emergency late night maintenance window, restoring their site to full performance by the next working day.
Direct efforts
We were approached at the weekend by a small team comprising local IT experts and doctors who are building an information website to efficiently distribute information to NHS staff members about how to use and select the correct protective equipment for the environment they are working in. We’re providing the virtual server, security updates, backups and 24/7 monitoring service for this free of charge, which has allowed the volunteer IT experts to concentrate on building the site. We’re expecting go-live in the next day or so once the content is checked.
We’re keen to hear of any other efforts where we may be able to assist.
Adding capacity
We’ve ordered more servers to expand the three busiest VM clouds to support existing customers scaling up, and new customers with urgent needs. We want to avoid cloud full and thankfully our server supplier is fully able to continue to build and deliver servers whilst maintaining 2m spacing between employees.