Hosted website outage

We're pleased to report that all databases have now been restored, and to points in time where any data loss will likely be of no significant consequence. All affected clients will be contacted over the weekend with the specifics of which databases have been restored to which point in time and any further details we wish to raise on a case-by-case basis.

We appreciate the thought of any data loss is unnerving, and we will be conducting a deeper review into the circumstances surrounding this incident and how our backup systems might cater for this highly unusual situation more robustly in future. Thank you once again for your patience and continued business.

We're around half way through restoring all databases on SH1. Some databases have fared better than others, with the recovery date being the 12th or 13th of February for the majority. Some databases have been rolled back to older dates, and we will be contacting everyone with the relevant dates in due course in order that they may consider the best course of action and seek further assistance from us if required.

We will provide another update once all database are completed, and then a further update over the weekend with the relevant dates for all affected customers.

Thank you for your continued patience.

Websites depending on databases are being slowly brought online where database data is available. If you are particularly impacted please contact the Service Desk who will be able to pass a message to Richard and Marc who are working on this and will try to prioritise your site.

We are still working to recover database data through all available means.

Web ports are now open which means customers with static websites, or external databases will now be online. The control panel at https://sh1.wessex.cloud:8443 is also online should customers have a recent copy of their database which they would like to upload.

Our control panel also allows DNS changes so should any customers wish to point their DNS to 3rd party hosting or temporary notice page this is also an option. The Service Desk are happy to assist with DNS changes over the phone from authorised contacts.

We are making some progress but it's too early to share any more than that. We'll provide further updates as soon as we can.

Thank you for your patience.

We are continuing to experience unexpected results restoring the damaged database files. Please bear with us.

We're in the process of restoring the database server from a backup dated 13-Feb-2024 23:00:12 which we believe to not contain corrupt database data files. If successful, service should be restored tonight, but data loss of 9 days within databases only will have occurred. This does not affect website files, scripts, emails or any other web hosting functionality.

Newer recovery points will be locked for a period of time in case customers need us to attempt restore of specific databases or tables from later dates but we expect unreliable results and incomplete data.

Apologies for the inconvenience due to this silent corruption issue. We haven't seen this occur in our web hosting environments in 19 years.

We have a full server recovery operation running in the background which should finish in the early hours of the morning. There is no guarantee this will resolve the issue if the database corruption is present in this full server backup. In the meantime we are stepping through backups to extract the last good database data as this may provide a faster path to resolution.

We will share further updates as they become available. Thank you for your patience.

As mentioned in our previous update this outage is affecting only a subset of our shared hosting customers hosted on node SH1. If your service is provisioned on SH1 you will see "SH1" in the URL for your control panel login. The IP address of your web hosting / website will also be address 185.160.167.27. This is the only node with issues at the moment, so if your website is offline that strongly indicates your services are provisioned on SH1.

Email services on SH1 were previously affected but service was restored earlier this afternoon.

Websites on SH1 and the hosting control panel itself remain offline. We are working diligently to restore the MariaDB databases on this server with all the appropriate resources allocated. A corrupt database is currently stopping the database service from running reliably, and to avoid any lost transactions or data we need to keep websites offline until the problem is 100% resolved.

We expect to have this resolved this evening, but cannot give a more accurate ETA at this time. Thank you for your patience.

Our Hosting team are currently working on an outage affecting some Shared hosting customers websites.

The issue is limited to a single server meaning virtual server customers aren't affected and not all websites are impacted.

A resolution is expected today but at this stage we do not have a firm timeframe for this.