[RESOLVED] Incident: storage unit [Paris datacenter]
An incident has occurred on one of our storage units in the Parisian datacenter. Our technical team is working to resolve the issue as quickly as possible.
Please do not perform any operations on your virtual machines in the meantime. Services should be restored automatically once the issue has been corrected.
We will update this post as new information arises.
Update Tue Oct 7 19:28:19 UTC: Some faulty hardware has been identified; we’re in the process of swapping it out.
Update Tue Oct 7 22:33:14 UTC: Our technical team is still trying to fix the issue.
Update Tue Oct 7 23:35:44 UTC: A ZIL disk has failed, and its failover also failed. We’re currently performing a manual switchover, and are proceeding very carefully to minimize the risk of data loss.
Most importantly: we understand how disruptive this is for you and we’re working as hard as we can to fix it. We will do our best to make it right.
Update Wed Oct 8 00:39:21 UTC: Our technical team is bringing the storage unit back up. The incident is nearly resolved and services are already beginning to come back online.
Update 02:31:10 UTC: We’re now seeing high loads on the problematic filer. The investigation continues!
Update 04:05:54 UTC: After working all night, our technical team in Paris has resolved the problem. Services should now be back to normal.
A postmortem and compensation details, as described in our IaaS Hosting Contract (section 2.2) will be provided in the days to come.
Update Thu Oct 9 17:31:34 UTC: A postmortem about this incident is available here.
