Mammoth VPS Status

Scheduled Network Outage

admin — Mon, 16 Jan 2012 22:32:56 +0000

Our upstream network provider has a planned outage scheduled in the early morning on Wednesday the 18th of January.

The expected downtime is around 15 minutes.

The downtime is network only – customer VPSs will stay running during this window and will simply lose network access. Once restored they should be running as they were before the outage.

UPDATE: Actual network downtime was 5 minutes; this status notice has been updated with the actual start and end time of the outage.

Partial network outage

admin — Fri, 23 Dec 2011 05:35:12 +0000

Following our decision to failover to backup router (because of http://status.mammothvps.com.au/view/1781 ) we experienced no network access for some customers. This was identified as being due to the failing router not correctly releasing its routes, at which point the failing router was rebooted. The reboot ended the outage.

Service was interrupted for approximately 11 minutes.

Intermittent High Latency

admin — Fri, 23 Dec 2011 03:55:27 +0000

We are investigating intermittent periods of high latency due to excessive CPU load on a router. mPanel may also be slow or inaccessible for some customers.

Issue has been resolved by failing over to backup router and rebooting primary.

Website maintenance

admin — Mon, 12 Dec 2011 04:32:37 +0000

During this scheduled maintenance period the Mammoth VPS website, including mPanel, will not be accessible.

There is no expected impact on customers’ VPS during this maintenance.

Degraded disk performance on vps1-4

admin — Wed, 07 Dec 2011 00:56:09 +0000

A SAN rebuild is currently in progress. We have observed some periods of quite slow I/O access in the last few hours, apparently related to sustained I/O heavy operations (like backups). These periods may cause slow disk performance for some customers.

We are continuing to monitor the situation and will provide updates as new information comes to light. Based on the progress reporting in the software, we do expect the rebuild to continue until tonight or possibly into tomorrow.

Update 7th December, 2011 12:20 PM AEDT:

Backups are temporarily disabled to alleviate lowered disk performance during the rebuild process.

Update 8th December, 2011 15:47 PM AEDT:

Backups were re-enabled Wednesday, December 7, 2011 23:05 PM AEDT

Disk performance returned to normal levels Thursday, December 8, 2011 06:30 AM AEDT

Data usage reporting inoperative

admin — Mon, 28 Nov 2011 04:05:59 +0000

The data usage reporting in mPanel is currently offline due to a failure in the software service that generates these reports.

We are investigating this now and expect it is a minor problem but will update customers once we have more information.

Update 30th November, 2011 15:19 PM AEDT:

Data usage reporting is now back online.

All data used during the usage reporting outage will not be counted against customers’ monthly data allowances.

Degraded disk performance on vps1-4

admin — Fri, 25 Nov 2011 00:36:48 +0000

A possible hard disk failure has triggered an array rebuild on our primary SAN. The array rebuilding is resulting in increased disk activity which in turn might lead to slower-than-normal I/O for customers that are connected to the SAN. Only customers on vps1, vps2, vps3 or vps4 might be affected.

We are monitoring the situation; at the moment the performance degradation does not seem too severe.

Update 17:15:

The rebuild is still progressing. We have observed some periods of quite slow I/O access in the last few hours, apparently related to sustained I/O heavy operations (like backups). These periods may cause slow disk performance for some customers – if you are affected please contact us to let us know so we can investigate further.

Update 28th November, 2011 15:00 PM AEDT:

The rebuild finished early in the morning yesterday and there are no further errors; performance has returned to normal.

vps02 crash

admin — Tue, 15 Nov 2011 21:21:08 +0000

This server has crashed, causing an outage for all customers on this host.

0810 Attempting to rescue server

0820 Rescue was unsuccessful, reboot in process

0825 Reboot completed, restarting customer VPS

0835 Customer VPS have been restarted

mPanel and Website unavailable

admin — Mon, 31 Oct 2011 23:28:10 +0000

Our primary website and the mPanel control interface will be unavailable for a few minutes during some reconfiguration work.

Network port reconfiguration

admin — Thu, 20 Oct 2011 04:14:57 +0000

An outage is not expected during this scheduled maintenance window.

Between 5:00 and 6:00 Sunday morning, we will be reconfiguring our networking infrastructure as follows:

Customers will be limited to a maximum of 100mbit outgoing traffic (from our network to the internet)
Customers will continue to have full 1gbit speed access to the internal Mammoth LAN

This change is designed to eliminate the potential for a network outage when a hacked VPS is used to attack other servers on the internet, as happened for 20 minutes on September 10 (documented here: http://status.mammothvps.com.au/view/1688 )

We have been testing this change for the past four weeks and have also compared it against our recorded data usage and have concluded that these limits will have no impact on the performance of our VPS infrastructure.