At approximately 12:09 today (SAST/UTC+2), 6 of our hosts suffered intermittent cluster network outages impacting connectivity to Virtual Machines (VMs) on the affected hosts.
The cluster network outages for 5 of the affected hosts was resolved by 12:51, and the issue affecting the 6th host was resolved by 13:06. VMs on all our other hosts were not impacted.
We believe that the root cause of the cluster network outage was an issue with an NFS share of a large internal storage subsystem that initially specifically impacted a single host (with what was most likely related to a stale NFS file handle) - this affected disk IO on the host in question which in turn impacted cluster connectivity from this host to a number of other hosts.
We're currently in the process of moving off this specific NFS storage platform, and anticipate that this will be completed by end-October 2022.
We apologise for any inconvenience caused during this episode, and assure you that (as always) we to strive to maximise uptime, availability and data integrity across our infrastructure at all times.
The CloudAfrica Team.
Posted Oct 03, 2022 - 18:45 SAST
Monitoring
The situation has been resolved and we are monitoring it closely a report will follow soon.
Posted Oct 03, 2022 - 15:06 SAST
Investigating
We are aware that clients are having difficulty accessing their servers.
We are investigating the problem and will revert shortly.
Posted Oct 03, 2022 - 12:19 SAST
This incident affected: Web Sites, API, Storage Services, and Cloud Services.