While it wasn’t a total outage (some VPSs that don’t do a lot of I/O would have been barely affected), many customers were severely impacted as their VPS performance dropped well below acceptable levels.

Issues experienced by customers included:

• Very slow I/O on their VPS
• Very long reboot times (caused by the slow I/O), which often made it look like the VPS was not working at all
• Lack of confirmation from support as to when the rebuild would complete
• Unable to move to a new host due to the degraded I/O

The problem:

The root cause of this problem appears to be a drive in the SAN temporarily locking up early Wednesday morning (2nd May), triggering a failed drive reaction in the array controller, which in turn means an array rebuild onto a hotspare was started. An array rebuild is not a particularly uncommon occurrence, and it normally has less of an impact – we’ve had drive failures in the past and most customers have not even noticed the performance degradation that resulted while the array was rebuilding.

Performance at this point was slightly reduced but generally everything was working as expected. A rebuild usually takes around 24-36 hours.

At approximately 10:30 two hosts that use this SAN, vps1 and vps3 began having issues, presumably due to the I/O load on the SAN from the array rebuild – vps1 became completely unresponsive and vps3 was almost completely unresponsive. We were unable to recover them so had to reset them. At this point we decided to drop the priority of the array rebuild to ‘low’ hoping it would prevent this and further I/O starvation issues during the rebuild.

The next morning as the rebuild slowly progressed we decided to mark the drive that had locked up as failed, to prevent the array from re-using it once the rebuild had completed. (Since it had just locked up temporarily and not actually failed, it was once again marked as healthy by the array controller). This was done at 09:20 on Thursday 3rd May.

Not long after this we noticed severely decreased performance for vps running off the SAN and that several customers were reporting availability and booting issues. We began looking into reasons for this and because of the timing, the marking of that drive as failed seemed to be the most probable cause, even though it seemed unlikely that such an action should have any impact at all since the drive was not part of the array or being used by the SAN at all.

Over the course of the next few hours we attempted various things such as reducing our own load on the SAN as much as possible to cure the degraded performance but nothing was particularly successful. We were reluctant to modify the SAN settings in case it caused additional issues, however since the progress of the rebuild had no slowed right down to a complete crawl and the impact on customers was so high, we felt we had no choice.

So at 14:15 Thursday 3rd May we set the drive previously marked as failed back to the “ready” state – this immediately improved performance. After further discussion this appears to be a firmware bug in the RAID controller.

The array rebuild completed at 06:32 Friday 4th May and normal performance was restored.

What we are doing now:

A number of issues were raised during this outage that we’ll be addressing over the coming months.

• The first priority was to replace the suspect hard disk with a new drive as soon as the rebuild had finished. While it was currently reporting normally, we didn’t want to take any further risks with it. It has now been replaced.
• We’ll update our internal process regarding disk array rebuilds to ensure that we do not manually fail a drive out while the rebuild is still in progress.
• We’ve changed a controller setting which will prevent it from automatically performing a copyback operation as soon as a rebuild is completed (this was not a factor in the outage but may have impacted performance after drive replacement).
• We’ll be looking at implementing a system to allow us to move customers (at their request) to a new host from their most recent backup. This will enable customers to restore service in the event of problems with their host server by migrating to a new host, at the cost of slightly outdated data.

We would also strongly advise customers who have business requirements around site uptime to investigate options that provide proper redundancy. While we make all efforts to ensure our servers are online and utilize redundant network, storage, and power supplies; hardware failures do occasionally occur. A redundant hosting strategy will minimise downtime during such an event. We provide IP failover as a free service for customers who have multiple VPSs.

The cause of the reboot is unknown and we’re investigating further to see if we can determine the exact problem.

Apologies for any inconvenience.

Apologies for the inconvenience.

Update 11:40AM AEST: We are still investigating the problem. The array is rebuilding but much slower than normal, and performance is massively below what it should be (normally a rebuild should only have a small performance hit, so we’re unsure exactly what is causing the massive degradation in performance). We’ll update this as more information comes to light.

Update 1:10PM AEST: We’ve made a few changes to accelerate the rebuild process, which has made a small impact on performance, but it is still slow. Investigations continue, but at this point we’re unsure how long the rebuild process will take.

Update 2:45PM AEST: We’ve made a configuration change which has stopped the crippling performance issue and our monitoring is indicating that things are back to a more normal state for an array rebuild. Customer VPSs should now be in a much more operable state, though please note that the array is still rebuilding so performance won’t be at normal speeds until this has finished.

Now the rebuild is proceeding normally we hope to be able to provide an estimate for how long it will take to be completed; we’ll update this post as soon as we have that information.

Update 5:00PM AEST: The rebuild continues; we have enough information now from the rebuild progress to estimate that it has another 15 hours to go.

Update Friday, May 4, 9:00AM AEST: The array rebuild has finished and performance has returned to normal.

Update Friday, May 4, 1:00PM AEST: The disk responsible for the array build has been swapped out for a new disk.

We will be providing more information about what happened (and what we’re doing to stop it from happening again) to customers very shortly.

The Incident Report is currently being written and our plan is to mail it to affected customers early next week (probably Tuesday at this point).

Update 11:06am AEST: the servers have stopped responding, reason still unknown. The servers are in the process of rebooting; we’ll have updates soon.

Update 11:13am AEST: vps01 host is back up, vps03 still booting. Please note your VPS instance may take a while to boot up after the host node becomes available.

Update 11:24am AEST: vps03 host is back up, servers on it are still starting. All customers on vps01 should be online now.

Update 11:35am AEST: All customers on vps03 should be online now.

Update 11:05 AM AEST: The problem appears to now be resolved. We’re still looking into the root cause with our upstream provider. Apologies for any inconvenience.

0925 We have tracked the problem down to a Denial-of-Service attack against a VPS on this host node. After blocking the attack performance has returned to normal.

0955 Some customer VPS were shut down during the high load; these have now been restarted.

Customer VPSs are unaffected by this outage.

Update 13:15 AEST: The site service has now been restored.

The web servers were stuck in a weird, non-working state. This normally would have been caught by alarming, but it looks like the alarming wasn’t set up correctly to catch this condition. We’ll be fixing this first thing to ensure that this does not happen again.

Again, our sincere apologies to those inconvenienced by this site outage.

We are awaiting an incident report from SOUL and will provide further details as they come to hand.

The effects are currently only barely visible but customers might notice the occasional packet getting dropped or brief period of network loss (a few seconds).

Update: This is now resolved.

During this maintenance period we will changing to a new router configuration to enable the eventual supply of IPv6 to customers later this year. As part of the maintenance period we are also decommissioning an older switch.

Customers should plan for the network to be inaccessible for a period of up to 30 minutes within the maintenance period. There is no other impact; all VPS will continue running and will be accessible without customer intervention following the scheduled maintenance period.

This maintenance period is outside of the 24/5 global trading window. Trading platforms can safely be left running. Individual VPSs will not be affected so you should not need to restart any services after the downtime, but we encourage users to double check just to make sure everything is working as expected after the maintenance window.