The blog ·
The OVH fire in 2021: our business continuity plan
Following the fire at OVH, an update on our business continuity plan
During the night of 9 to 10 March 2021, a fire broke out at OVHcloud’s Strasbourg site. There were no casualties amongst OVH staff, the fire service or the security forces. The SBG2 building was completely destroyed.
The following information is not intended as a criticism of OVHcloud. OVH is a long-standing partner, and its teams have done everything possible to restore the service. We will attempt to describe our teams’ experience whilst implementing our disaster recovery plan (DRP), in the face of customers’ expectations and distress.
The BCP
With the support of Sôter Conseil, we had put in place a disaster recovery plan that had been revised shortly beforehand. It included a crisis communication strategy: information published on this blog and shared on our social media channels.
Our disaster recovery plan is relatively straightforward, given the strict compartmentalisation of our internet services:
- connecting to OVH’s services;
- checking that the servers are functioning;
- checking the availability of remote backups;
- ordering new servers;
- restoring backups;
- transferring IP addresses;
- checks and tests.
Plan B:
- if ordering is not possible, partition existing servers and proceed with Plan A;
- if IP addresses are unavailable, amend clients’ DNS records and proceed with Plan A.
Initial assessment: two servers destroyed, but that’s not all…
Two servers completely destroyed is problematic enough in itself, but all of OVH’s services have been affected: the management portal appears to be overloaded, certain operations are significantly slowed down, and the Exchange servers – even those outside Strasbourg – appear to be overloaded.
Our team therefore sprang into action at the crack of dawn to implement the disaster recovery plan that we had reviewed with Sôter at the end of the previous year.
The disaster recovery plan – not as straightforward as expected…
We obviously had remote backups of our servers, but the disruption caused by the fire to OVH’s operations meant we were unable to respond as quickly as we would have liked to our customers’ needs.
For a while, the system for ordering new servers was overwhelmed, the OVH management interface was running terribly slowly, we experienced numerous disconnections, our IP phones were eerily quiet, and the IP address switchover was blocked: we’ll have to adapt the disaster recovery plan to the changing situation.
At 11.40 am, we managed to order a server from the Frankfurt data centre in Germany, with commissioning scheduled for 120 seconds later. At 12.30 pm, it was still being delivered.
Our plan is to migrate all services (hosted applications, websites, emails) to a new server as soon as it becomes available. If OVH is unable to deliver it, Plan B will involve restoring the backups onto servers dedicated to other tasks.
Plan B is underway
10 March 2021, 6.00 pm — Our server has still not been delivered. Octave Klaba, CEO of OVHcloud, has promised replacement servers, but has given no deadline or conditions for their provision. A large proportion of OVH’s services are disrupted, with some completely paralysed, but the OVH teams are doing their utmost to get everything up and running again. This will take several weeks.
At AmaZili, it’s also a race against time for the operations teams, who are restoring services from backups on our remaining servers whilst awaiting a delivery that seems unlikely. We have put in place temporary solutions for customers with urgent matters to deal with.
11 March 2021, 2.00 am — Problems with IP addresses and DNS. As we are unable to transfer the IP addresses, we’ve changed our plan: our teams will reconfigure all DNS records. More to follow after a short night’s sleep.
Update after the first night
11 March 2021, 6.00 am — Our server still hasn’t been delivered.
At AmaZili, all backups have been restored. We set up a temporary installation for one of our clients, who had submitted a tender response the previous day, so that they could receive emails relating to the process. The teams then restored the backups to our least-loaded server and will, if possible, move on to reconfiguring the DNS, as IP addresses remain difficult to transfer from one server to another.
Midday update
11 March 2021, 4.40 pm — Still no replacement server. At midday, Octave Klaba announced that services would return to normal in several weeks’ time.
Update before the second night
11 March 2021, 7.00 pm — Still no replacement server. It’s clear that everyone at OVH is doing their best, but no server is available to replace those that have burnt out.
A backup of the first server has been restored to a temporary space. DNS propagation issues persist, and changes are taking time to take effect. The load on the destination server is increasing, and we have no choice but to make do: we’re proceeding with the utmost caution… We’re attempting to restore the backup of the second burnt-out server.
Midnight update
1 March 2021, 1 am — Still no replacement servers. The server ordered from OVH has still not been delivered. Following OVH’s statements and their responses to our support requests, we have asked for the order to be cancelled and for the burnt-out servers to be replaced.
The backup from a second server has been restored to a temporary location. We were unable to restore the very latest backup: it was corrupted and therefore unusable. The load on the destination server is skyrocketing; we hope it will hold up, but performance will inevitably be poorer for all hosted services. We cannot restore all DNS records: some domains are managed directly by customers, and at this hour, they are asleep… The teams are going to take a well-deserved rest.
Update at the end of the second night
12 March 2021, 5.30 am — Still no news of the replacement servers. Come on, OVH, you must be as tired as we are: a straightforward replacement procedure would be most welcome.
OVH’s services are returning to normal, or nearly so: the response times of the management interfaces are now in line with our teams’ frayed nerves. The e-commerce sites restored from the second set of backups are working. All that remains is to redirect all the DNS records we don’t control. Today will be crucial in determining whether the destination server can handle the load.
Midday update
12 March 2021, 3.30 pm — Still no news on the replacement servers. We have asked OVH to cancel the server ordered in Frankfurt, which has still not been delivered, and have ordered two new servers in Roubaix.
All our clients’ services have been restored: websites are up and running, emails are coming through, and the applications are performing the functions for which they were designed. Backing up the second server proved to be a real challenge, and we were unable to restore the very latest backups. Whilst we await delivery of the new servers, we will prepare for the reverse migration, hoping not to be hampered by delays in DNS updates or IP address transfers. We will also be holding a swift debrief with the teams at Sôter Conseil, in order to learn from this crisis and improve our disaster recovery plan.
What has changed since
This fire has taught us that a remote backup alone is not enough: we must also be able to restore it quickly, elsewhere, when the provider itself is in disarray. Since then, we have reviewed the distribution of our servers, our procedures are audited internally, and we are implementing a more sophisticated backup infrastructure.
To find out how we protect your websites, see our hosting page and our post on the most common security errors.