Webarchitects Service Status #

Tue, 11 May 2021 07:36:28 +0100 #

All the services have been restored on older servers while we look into the issue with the new server. We believe that everything is running without issues but there wil be a email backlog for a while, if you find any problems please contact us.

Tue, 11 May 2021 07:18:46 +0100 #

We have brought up most servers effected by the issue with the connection to the disk array on other, older front facing servers, there are three servers still down, these include git.coop and webarch.email, we hope to be able to bring these back up soon.

Tue, 11 May 2021 06:55:46 +0100 #

We appear to have a problem with the mounting of the storage array on our latest front facing server so we are going to restart it, this should solve the issue.

Mon, 10 May 2021 22:07:30 +0100 #

We are upgrading all out hosted Discourse servers so they will be unavailable for a little while.

Fri, 30 Apr 2021 18:55:24 +0100 #

We are upgrading all the Discourse servers we host so they will be unavailable for a little while.

Fri, 30 Apr 2021 08:45:36 +0100 #

The webarch.email server is up and running again, very sorry about the downtime this morning.

Fri, 30 Apr 2021 08:13:05 +0100 #

We believe that all services have been restored.

Fri, 30 Apr 2021 07:55:16 +0100 #

We are sorry that our new front facing servers connection to the new disk array failed overnight, this resulted in the virtual servers it is running having errors / being out of action, we are restarting everything now.

Thu, 29 Apr 2021 22:18:16 +0100 #

We are updating adding some additional resources to the webarch.email server and this requires a restart so it will be unavailable for a little while.

Thu, 29 Apr 2021 10:33:43 +0100 #

We are upgrading GitLab on git.coop so it will be offline for a little while.

Sat, 24 Apr 2021 14:24:33 +0100 #

We are moving some servers to new hardware, this will result in a little downtime for each server as we power it down on the old computer and then bring it up again on the new one.

Fri, 23 Apr 2021 13:02:36 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Tue, 20 Apr 2021 20:09:37 +0100 #

We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.

Thu, 15 Apr 2021 20:40:48 +0100 #

We are upgrading Discourse on all our hosted servers so they will be unavailable for a little while.

Thu, 15 Apr 2021 09:09:13 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Mon, 12 Apr 2021 22:14:56 +0100 #

We are upgrading Docker across all our servers and this will result in a little downtime for services that run via Docker, specifically Mailcow and Discourse.

Sat, 10 Apr 2021 14:11:51 +0100 #

We are upgrading all the Discourse servers we host so they will be unavailabe for a little while.

Thu, 01 Apr 2021 11:26:26 +0100 #

We are upgrading GitLab on git.coop, so it will be unavailable for a little while.

Tue, 30 Mar 2021 19:42:53 +0100 #

We are upgrading Mailcow at webarch.email, this will result in the service being unavailable for a little while.

Mon, 29 Mar 2021 17:47:20 +0100 #

The upgrade for GitLab at git.coop has been completed.

Mon, 29 Mar 2021 17:36:19 +0100 #

We are shutting down Gitlab at git.coop in order to extend one of the server disks, this upgrade shouldn't take long.

Sun, 28 Mar 2021 20:41:14 +0100 #

We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.

Fri, 26 Mar 2021 14:49:25 +0000 #

We have had the following response from Cloudmark: 'I have reset the reputation of your IP, so you should see delivery improve shortly. Please note that updates do not occur instantly but should generally happen within an hour of receiving this response.'

Fri, 26 Mar 2021 14:32:27 +0000 #

We are aware of email delivery issues to servers that use Cloudmark Sender Intelligence, we don't know what the reason for the CSI RBL listing is but we have requested removal.

Tue, 23 Mar 2021 09:26:36 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Thu, 18 Mar 2021 08:25:30 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Sun, 14 Mar 2021 10:27:10 +0000 #

We need to shutdown git.coop and for a little while to extend disk sizes, they won't be offline for long.

Thu, 11 Mar 2021 08:15:13 +0000 #

We are upgrading Discourse on all our servers so there will be a little downtime for these services.

Tue, 09 Mar 2021 08:52:22 +0000 #

There is a new containerd.io Docker package available and this means that we will need to restart Docker across all our servers, so Mailcow on webarch.email and all our Discourse servers will be unavailable while this package is upgraded.

Mon, 08 Mar 2021 21:35:05 +0000 #

We are upgrading a Docker package on all our servers and this will result in a little downtime for all services that run on Docker.

Fri, 05 Mar 2021 13:48:23 +0000 #

We are upgrading GitLab on git.coop so it will be offline for a little while.

Wed, 03 Mar 2021 07:58:47 +0000 #

We are updating Docker on all our servers so services that depend on it, Mailcow on webarch.email and all the Discourse servers will be unavailable for a little while.

Fri, 26 Feb 2021 20:22:42 +0000 #

The webarch4.co.uk server is back up.

Fri, 26 Feb 2021 20:16:14 +0000 #

We are shutting down the webarch4.co.uk shared hosting server to extend one of it's disks, it shouldn't be offline for long.

Fri, 26 Feb 2021 16:40:05 +0000 #

We are upgrading Docker on all our servers and this will result in a little downtime for services that run on Docker, specifically Mailcow and Discourse.

Wed, 24 Feb 2021 09:20:51 +0000 #

We are upgrading GitLab on git.coop so it will be offline for a little while.

Mon, 22 Feb 2021 15:35:52 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Fri, 19 Feb 2021 11:53:47 +0000 #

We are upgrading Discourse on all our servers and this will result in them being offline for a little while.

Sun, 14 Feb 2021 17:09:32 +0000 #

Migrating git.coop to our new SSD file array is taking a little longer than expected, the biggest 200G disk in 65% done, we hope the transfer will be completed sometime after 6pm.

Sun, 14 Feb 2021 16:06:10 +0000 #

We are going to shutdown GitLab at git.coop in order to migrate it to the new SSD based storage array, it might be offiline for around an hour or so as a result, but will be faster when it is back up again.

Sun, 14 Feb 2021 15:52:52 +0000 #

We are shutting down GitLab at git.coop to grow the partition containing the Docker registry, it should only be offline for a little while.

Sun, 14 Feb 2021 10:15:04 +0000 #

We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.

Fri, 12 Feb 2021 13:34:36 +0000 #

The GitLab runner server, runner.git.coop is now up and running again.

Fri, 12 Feb 2021 13:00:38 +0000 #

We are migrating the GitLab runner server, runner.git.coop from our old DHH storage array to the new SSD storage array so it will be offline for around another half hour.

Fri, 12 Feb 2021 12:56:14 +0000 #

We are shutting down runner.git.coop to add additional space to the disks.

Fri, 12 Feb 2021 10:26:22 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Wed, 10 Feb 2021 18:38:10 +0000 #

We need to shutdown host2.webarch.net for a little time to expand a disk, it shouldn't be offline for long.

Sat, 06 Feb 2021 11:03:24 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Tue, 02 Feb 2021 08:40:05 +0000 #

We are upgradinmg GitLab on git.coop so it will be unavailable for a little while.

Tue, 02 Feb 2021 08:30:27 +0000 #

We are upgrading Docker on all our servers and this will cause a little downtime for services such as Mailcow on webarch.email and all the Discourse servers.

Fri, 29 Jan 2021 07:47:44 +0000 #

We are upgrading Discourse on all the managed Discourse servers we host so they will be unavailable for a little while.

Thu, 28 Jan 2021 09:28:13 +0000 #

We have discovered an issue with our anti-spam gateway, mx.webarch.net this appears to have resulted in some email being delayed or returned, we are now adding some additional capacity to the server and it should be fully functional again shortly.

Tue, 26 Jan 2021 21:30:56 +0000 #

We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.

Sat, 23 Jan 2021 11:35:32 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Fri, 22 Jan 2021 20:11:48 +0000 #

We are upgarding all our managed Discourse servers so they will be unavailable for a little while.

Fri, 15 Jan 2021 10:33:06 +0000 #

The upgrade of GitLab on git.coop has completed and the servers is up and running as normal, sorry for the downtime.

Fri, 15 Jan 2021 09:57:08 +0000 #

We are extending one of the disks on git.coop prior to the upgrade of GitLab, we expect it will be unavailable for another 10 minutes or so.

Fri, 15 Jan 2021 09:37:32 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Sat, 09 Jan 2021 11:56:41 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Fri, 08 Jan 2021 09:22:14 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Wed, 06 Jan 2021 15:57:33 +0000 #

Over the new few hours we are upgrading all Discourse servers to use PostgreSQL 13 and this migration process will result in a little downtime for each server.

Tue, 05 Jan 2021 08:04:10 +0000 #

We are upgrading Docker on all our servers, this will result in a little downtime for services that depend on it like Discourse and Mailcow on webarch.email

Thu, 31 Dec 2020 20:51:06 +0000 #

We are shutting down webarch4.co.uk to enable a disk to be extended, this shouldn't take long.

Thu, 24 Dec 2020 10:17:20 +0000 #

We are upgrading GitLab on git.coop, it will be offline for a little while.

Wed, 23 Dec 2020 11:06:46 +0000 #

We are upgrading GitLab at git.coop so it will be unavailable for a little while.

Sun, 20 Dec 2020 06:07:31 +0000 #

We are restarting webarch.email, sorry for te time it took to move the server and the inconvience caused.

Sat, 19 Dec 2020 11:47:10 +0000 #

50% of the main 1TiB webarch.email disk has now been copied, we anticipate that the mail service will be operational again at some point on Sunday morning, once again we are sorry about this downtime and the fact that it is lasting longer than expected.

Sat, 19 Dec 2020 08:03:19 +0000 #

We are afraid that copying the webarch.emal server from our old disk array to the new one is taking longer than expected, 33% of the main 1TB disk has been copied to the new server, at this rate we don't anticipate that the server will be back online until Sunday morning, we are very sorry for the inconvience this will cause people today.

Fri, 18 Dec 2020 23:17:46 +0000 #

The webarch.email file systems have been checked and we have started copying the disks to the new SSD based file server.

Fri, 18 Dec 2020 23:05:46 +0000 #

The webarch.email server is now shutdown and we are running checks on the disk image prior to copying them to the new file server.

Fri, 18 Dec 2020 22:44:21 +0000 #

We are shutting down webarch.email at 23:00 UTC in order to migrate it to a new server, due to the size of the server we anticipate that this will take around 7 hours. No email should be lost, other mail servers should queue the email for delivery when the server is up and running again.

Tue, 15 Dec 2020 11:25:19 +0000 #

We are upgrading Docker on all our servers, this will result in a little downtime for services that run in Docker containers, specifically Mailcow on webarch.email and all Discourse servers.

Fri, 11 Dec 2020 11:32:52 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Wed, 09 Dec 2020 20:17:49 +0000 #

We are upgrading Docker on a few servers that were not updated this morning.

Wed, 09 Dec 2020 13:50:46 +0000 #

The issue with the EHLO/HELO on webarch.email has been resolved and we believe that delayed emails will now start to get through.

Wed, 09 Dec 2020 13:29:07 +0000 #

There is an issue with webarch.email related to this mornings Docker upgrade, it is currently not using the correct EHLO/HELO and this is resulting is some outgoing email being delayed, we are working with the Mailcow developer to resolve this problem ASAP.

Wed, 09 Dec 2020 10:09:24 +0000 #

The issue with the certificate for mail.webarch.email has been resolved, sorry for any problems this caused.

Wed, 09 Dec 2020 10:03:56 +0000 #

We are sorry about the certificate issues with mail.webarch.email, we are sorting that out now.

Wed, 09 Dec 2020 08:39:03 +0000 #

We are upgrading Docker across all our servers and this will cause a little downtime for services that depends on iy like webarch.email and Discourse servers.

Tue, 08 Dec 2020 09:59:23 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Wed, 02 Dec 2020 19:13:59 +0000 #

We are upgrading Docker, Discourse and Mailcow over the next hour so services that run using these projects will have a little downtime.

Tue, 24 Nov 2020 09:31:55 +0000 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Mon, 23 Nov 2020 06:41:54 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Fri, 20 Nov 2020 20:06:42 +0000 #

We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.

We are upgrading Docker on all our servers, this will results in services that are run in Docker containers restarting and they will be unavailable for a little while, this includes webarch.email and all our Discourse servers.

Tue, 15 Sep 2020 10:17:23 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Sat, 05 Sep 2020 09:14:23 +0100 #

We are upgading GitLab on git.coop so it will be unavailable for a little while.

Thu, 03 Sep 2020 09:39:07 +0100 #

We are uogading GitLab on git.coop so it will be unavailable for a little while.

Sun, 30 Aug 2020 09:01:27 +0100 #

We are upgrading GitLab on git.coop and also Mailcow on webarch.email so both services will be unavailable for a little while.

Tue, 25 Aug 2020 15:54:02 +0100 #

We are updating GitLab at git.coop, so it will be unavailable for a little while.

Fri, 21 Aug 2020 10:07:40 +0100 #

We are currently upgrading all the Discourse servers we host, they will be unavailable for a little while.

Wed, 19 Aug 2020 10:16:21 +0100 #

We are upgrading GitLab on git.coop so it will be offline for a little while.

Tue, 18 Aug 2020 11:41:25 +0100 #

We are upgrading GitLab on git.coop so it will be offline for a little while.

Wed, 12 Aug 2020 06:00:28 +0100 #

We are upgrading GitLab on git.coop so it will be offline for a little while.

Thu, 06 Aug 2020 00:08:33 +0100 #

We have completed the move to the new file server for webarch7.co.uk.

Wed, 05 Aug 2020 23:34:27 +0100 #

We have completed the move for webarch6.co.uk and it is up and running again. We are now shutting down webarch7.co.uk to move it to the new file server.

Wed, 05 Aug 2020 22:52:38 +0100 #

We are moving webarch6.co.uk to our new file server so it will be offline for a while from 11pm BST.

Sat, 01 Aug 2020 12:11:08 +0100 #

We are shutting host3.webarch.net down for a short time to extend a disk.

Sat, 01 Aug 2020 10:58:15 +0100 #

The upgrade and reconfiguration of webarch4.co.uk has completed.

Sat, 01 Aug 2020 02:55:31 +0100 #

The upgrade and reconfiguration of webarch5.co.uk has completed.

Sat, 01 Aug 2020 00:43:26 +0100 #

The upgrade and reconfiguration of webarch2.co.uk has completed.

Fri, 31 Jul 2020 22:52:27 +0100 #

All four servers we are upgrading this weekend are now running the latest stable version of Debian, Buster, we are now in the process of reconfiguring all the user accounts on the servers and regenerating the Apache and PHP config for the switch from mod_php to php-fpm.

Fri, 31 Jul 2020 21:27:49 +0100 #

We are starting a planned upgrade on webarch5.co.uk so it will be unavailable for a while.

Fri, 31 Jul 2020 20:39:36 +0100 #

We are starting a planned upgrade on webarch4.co.uk so it will be unavailable for a while.

Fri, 31 Jul 2020 19:58:10 +0100 #

We are starting a planned upgrade on webarch3.co.uk so it will be unavailable for a while.

Fri, 31 Jul 2020 18:07:57 +0100 #

We are starting a planned upgrade on webarch2.co.uk so it will be unavailable for a while.

Fri, 31 Jul 2020 11:19:20 +0100 #

We are currently having an issue restarting MySQL on webarch4.co.uk, we hope to have this resolved soon.

Thu, 30 Jul 2020 18:24:57 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Sat, 25 Jul 2020 14:13:53 +0100 #

We have completed the upgrade work on host3.webarch.net and webarch1.co.uk.

Sat, 25 Jul 2020 08:06:11 +0100 #

We are currently configuring users accounts on host3.webarch.net and upgrading webarch1.co.uk.

Sat, 25 Jul 2020 07:47:51 +0100 #

The upgrade work on host2.webarch.net has been completed.

Sat, 25 Jul 2020 07:46:41 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Fri, 24 Jul 2020 20:35:02 +0100 #

We are shutting down webarch1.co.uk in order to copy it to our new file server, once it has been copied we will be upgrading it, so it will probably be unavailable for some time.

Fri, 24 Jul 2020 19:16:40 +0100 #

We are shutting down host3.webarch.net in order to copy it to our new file server, once it has been copied we will be upgrading it, so it will probably be unavailable for some time.

Fri, 24 Jul 2020 18:01:57 +0100 #

We are shutting down host2.webarch.net in order to copy it to our new file server, once it has been copied we will be upgrading it, so it will probably be unavailable for some time.

Wed, 22 Jul 2020 17:03:19 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Sun, 12 Jul 2020 22:01:03 +0100 #

We are upgrading Mailcow on webarch.email so email services be unavailable for a little while.

Fri, 10 Jul 2020 10:30:58 +0100 #

We are upgrading GitLab on git.coop and as a result it will be unavailable for a little while.

Wed, 08 Jul 2020 10:24:56 +0100 #

The issues with our old disk array this morning have been resolved. We have a new SSD based disk array in place and shall be migrating service to it out of office hours over the next few weeks.

Wed, 08 Jul 2020 10:03:21 +0100 #

We are upgrading Mailcow on webarch.email as it won't take much longer than a restart and there have been a few users who have has issues that appear to be related to the issues with the disk array earlire. It will be offline for a little while.

Wed, 08 Jul 2020 08:35:26 +0100 #

There is currently degraded performance on our old disk array and this is effecting most of our Sheffield based systems, we shall be migrating services to our new disk array as quickly as possible but we are afraid that this situation is probably going to take some time to fully resolve.

Tue, 07 Jul 2020 23:49:15 +0100 #

The members discourse forum and nextcloud are available again

Tue, 07 Jul 2020 21:32:21 +0100 #

The members discourse forum and office nextcloud is offline for maintenance

Tue, 07 Jul 2020 08:43:47 +0100 #

We are upgradeing GitLab on git.coop, it will be unavailable for a little while.

Mon, 06 Jul 2020 10:04:41 +0100 #

Scheduled maintenance this evening (20:00-24:00) is cancelled

Mon, 06 Jul 2020 01:50:16 +0100 #

The migration of git.coop back to the old disk array has completed.

Sun, 05 Jul 2020 22:03:50 +0100 #

We have having to migrate the GitLab server at git.coop back to our old storage array as we have found that the new array needs rebuilding before it can be used in production and we are sorry to say that we anticipate that this is going to result in the git.coop server being offline for a little over two hours.

Sun, 05 Jul 2020 20:40:32 +0100 #

git.coop will be offline between 20:00 - 24:00 06/07/2020 for maintenance

Wed, 01 Jul 2020 22:41:40 +0100 #

The migration and upgrade of git.coop has completed, sorry it took so long.

Wed, 01 Jul 2020 20:11:00 +0100 #

The git.coop GitLab server will be unavailable for a little while as we are upgrading it and migrating it to the new storage array.

Sun, 28 Jun 2020 09:53:50 +0100 #

We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.

Sat, 27 Jun 2020 19:53:22 +0100 #

We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.

Wed, 24 Jun 2020 17:08:40 +0100 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Tue, 23 Jun 2020 08:45:47 +0100 #

We are in the process of upgrading Docker on all our server and this will result in services that depend on Docker being upavailable for a little while, sorry for any problems caused by this.

Tue, 23 Jun 2020 07:53:29 +0100 #

We will be stopping and restarting webarch.email to allow Docker to be upgraded, this will result in a little downtime.

Mon, 22 Jun 2020 16:18:07 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Mon, 22 Jun 2020 09:59:04 +0100 #

Sorry for the disruption to the webarch.email service this morning, it has now been updadet and allocated additional resources.

Mon, 22 Jun 2020 09:32:11 +0100 #

We are having some issues with webarch.email and as a result we are going to upgrade and restart it, it will be unavailable for a little while.

Wed, 17 Jun 2020 21:39:00 +0100 #

We are going to be shutting down webarch1.co.uk, webarch2.co.uk, webarch4.co.uk and webarch5.co.uk for short periods this evening to extend the the size of their disks.

Sat, 13 Jun 2020 12:36:20 +0100 #

We are upgrading all our hosted Discourse servers, they will be unavailble for a little while.

Thu, 11 Jun 2020 09:42:05 +0100 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Sun, 07 Jun 2020 23:26:18 +0100 #

We are upgrading Mailcow on webarch.email and extending a disk on webarch3.co.uk so these services will be unavilable for a little while.

Thu, 04 Jun 2020 20:07:31 +0100 #

We are upgrading GitLab on git.coop.

Wed, 03 Jun 2020 21:53:39 +0100 #

We are upgrading GitLab on git.coop.

Tue, 02 Jun 2020 07:52:39 +0100 #

We are upgrading Docker on our servers, this will result in services such as webarch.email being unavailable for a little while.

Fri, 29 May 2020 15:52:56 +0100 #

We are upgrading GitLab on git.coop so it will be down for a little while.

Fri, 29 May 2020 06:31:20 +0100 #

We are upgrading Docker on all our servers, this is going to result in a little downtime for everything that depends on it including webarch.email.

Wed, 27 May 2020 20:11:44 +0100 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Tue, 26 May 2020 21:08:39 +0100 #

We are manually rebuilding the Discourse servers we host to upgrade PostgreSQL, so there will be a little downtime this evening.

Fri, 22 May 2020 16:09:14 +0100 #

We are uugrading GitLab on git.coop so it will be unavailable for a little while.

Tue, 19 May 2020 08:31:45 +0100 #

A new version of Docker has been released so we are going to upgrade it across all our servers, this will cause a little downtime for services that run on Docker, including webarch.email.

Sat, 16 May 2020 09:26:38 +0100 #

We are upgrading GitLab on git.coop to version 12.10.6.

Fri, 15 May 2020 08:25:14 +0100 #

This mornings upgrades have completed, in a little while webarch.email should have fully restarted, all other services should already be up and running without issues.

Fri, 15 May 2020 07:55:15 +0100 #

We are about to update the Docker containerd.io package on all our servers, this will cause all the services running via Docker to be restarted so there will be some downtime for some things like webarch.email for a little while.

Thu, 14 May 2020 15:31:20 +0100 #

We are about to upgrade git.coop so it will be down for a little while.

Mon, 04 May 2020 19:05:18 +0100 #

We are upgrading GitLab on git.coop.

Thu, 30 Apr 2020 21:07:46 +0100 #

We are upgrading GitLab on git.coop.

Sun, 26 Apr 2020 09:21:55 +0100 #

We are sorry that the Apache web server stopped running on webarch3.co.uk just before 1:20am BST this morning, we haven't found the cause of this but the service was restored at 9:10am.

Sat, 25 Apr 2020 20:53:24 +0100 #

We are rebooting webarch6.co.uk to add additional CPUs to the server, it will be unavailable for a little while.

Fri, 24 Apr 2020 14:04:33 +0100 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Thu, 23 Apr 2020 22:54:49 +0100 #

We are upgrading webarch.email, it will be unavailable for a little while.

Wed, 22 Apr 2020 16:27:56 +0100 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Mon, 20 Apr 2020 11:17:05 +0100 #

We are upgrading GitLab on git.coop so it will be offline to a little time.

Fri, 17 Apr 2020 22:19:00 +0100 #

We are going to be upgrading webarch.email shortly and this will result in it being unavailable for a little while.

Wed, 15 Apr 2020 07:58:38 +0100 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Wed, 01 Apr 2020 07:20:35 +0000 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Sun, 29 Mar 2020 10:53:07 +0100 #

We are going to upgrade webarch.email so it will be unavailable for a little while from around 11:00 UTC.

Sat, 28 Mar 2020 23:22:23 +0000 #

The webarch6.co.uk server is up now, we are very sorry for the downtime of services today.

Sat, 28 Mar 2020 20:37:06 +0000 #

We believe that all our servers are now back up apart from webarch6.co.uk, we are still working on this one. Please contact us at support@webarchitects.coop if you are having an issue with a service not on webarch6.co.uk.

Sat, 28 Mar 2020 19:12:46 +0000 #

We are having issues with MariaDB InnoDB recovery on several shared hosting servers.

Sat, 28 Mar 2020 18:09:28 +0000 #

We are now leaving the data centre, we will bring up the remaining servers remotly.

Sat, 28 Mar 2020 18:06:14 +0000 #

We have started bringing up the virtual servers, it will be a little while before they are all running again.

Sat, 28 Mar 2020 17:48:39 +0000 #

There appears to be no hardware problem with the file server, we are now decrypting the disks and bringing it back up, once this has been done we will be able to mount the filesystem from the front facing servers and then we will be able to start bringing all the virtual servers back up, this is going to probably take a couple of hours in total, very sorry for the inconvienience caused.

Sat, 28 Mar 2020 17:40:31 +0000 #

Our main file server is still booting.

Sat, 28 Mar 2020 17:15:29 +0000 #

Our main file server was not responding on the console so we are power cycling it, it is booting now. We will need to restart all the virtual servers after the file server is brought back up.

Sat, 28 Mar 2020 16:47:41 +0000 #

We are outside our Sheffield data centre waiting for a member of staff to let us in.

Sat, 28 Mar 2020 16:30:01 +0000 #

Our main file server had gone down and we are on our way to the data centre to investigate the cause.

Sat, 28 Mar 2020 15:57:13 +0000 #

We are having some issues with our infrastructure in Sheffield and are investigating the cause.

Thu, 26 Mar 2020 18:02:54 +0000 #

We are updating GitLab on git.coop so it will be unavailable for a little while.

Sun, 22 Mar 2020 14:36:04 +0000 #

We are upgrading GitLab on git.coop so it will be unavailable for a little while.

Tue, 17 Mar 2020 07:36:01 +0000 #

We are upgrading GitLab on git.coop so it will be down for a little while.

Mon, 16 Mar 2020 15:24:14 +0000 #

We are aware of rate limiting of email from webarch.email to Gmail and Outlook, we believe that this has been triggered by a client who's email account was compromised and then used to send a large volume of spam. Please ensure that all the devices that you use to access your email accounts are secure and that all your email passwords are strong ones. We expect the rate limiting to be lifted in time.

Sun, 15 Mar 2020 13:46:59 +0000 #

We are shutting down webarch.email to add additional disk space, it will be unavailable for a little while.

Wed, 11 Mar 2020 23:21:11 +0000 #

We are upgrading GitLab so git.coop will be unavailable for a little while.

Wed, 11 Mar 2020 20:36:01 +0000 #

We are upgrading Docker on all our servers, this will case a little downtime for servers that use Docker like webarch.email.

Mon, 09 Mar 2020 09:12:17 +0000 #

We are upgrading GitLab on git.coop, it will be unavailable for a little while.

Wed, 04 Mar 2020 06:30:04 +0000 #

We are upgrade Docker on all our servers, this will result in a little downtime on webarch.email.

Tue, 03 Mar 2020 06:51:48 +0000 #

We are updating Docker on all our servers, this will result in a little downtime on servers that use Docker, for example webarch.email.

Fri, 21 Feb 2020 20:03:00 +0000 #

We are upgrading webarch.email so it will be unavailable for a little while.

Mon, 17 Feb 2020 20:37:01 +0000 #

We are rebooting host3.webarch.net and it is currently doing a file system check, it should be back online soon, sorry for any inconvenience caused.

Thu, 13 Feb 2020 19:05:26 +0000 #

Our upstream bandwidth provider had put mitigations in place to prevent the NTP reflection attack from disrupting our services, we hope this this is the end of the matter, however because we still have no idea what the reason for the attack was we have no idea if there is more to come, once again we are very sorry for any inconvenience caused.

Thu, 13 Feb 2020 18:09:31 +0000 #

It looks like it is a NTP reflection attack against webarch4.co.uk, we have no idea of the reason for it, our upstream bandwidth provider is working on mitigating the effect.

Thu, 13 Feb 2020 17:33:29 +0000 #

Our Sheffield network is suffering from a DDOS, we are working with our upstream partners to investigate the cause of this and to see what mitigations can be put in place. We are very sorry for any inconvenience caused.

Wed, 12 Feb 2020 22:24:34 +0000 #

We need to quickly reboot webarch6.co.uk, sorry that it would be available for a few minutes.

Sat, 08 Feb 2020 15:08:00 +0000 #

We believe that all systems are up and running now. There was a power outage for the whole area that our data centre is in and then the UPS for our rack (which is due to have replacement batteries by the end of the month) didn't manage to provide power for quite long enough to allow the backup generator to start up and therefore all our servers lost power. Once power was restored we needed to do a filesystem check on the router before decrypting and bringing up the storage array and once that was up we were able to bring up all the virtual servers, sorry that this all took a while, this is the first time in well over a decade that we have had an issue like this.

Sat, 08 Feb 2020 13:34:00 +0000 #

We hope to have all services restored by 2pm.

Sat, 08 Feb 2020 12:39:45 +0000 #

We are in the process of bringing services back up.

Sat, 08 Feb 2020 12:07:11 +0000 #

There has been a power failure at our Sheffield datacentre and this has caused all our Sheffield services to go down, we are on our way to the datacentre now.

Sat, 01 Feb 2020 22:25:31 +0000 #

The update to webarch.email has completed, sorry for any inconvenience caused.

Sat, 01 Feb 2020 22:08:43 +0000 #

We are updating webarch.email, it will be unavailable for a little while.

Sun, 19 Jan 2020 20:23:58 +0000 #

We are upgrading webarch.email so it will be unavailable for a little while.

Tue, 14 Jan 2020 10:54:04 +0000 #

There are some issues with the git.coop GitLab web interface today, we are unsure of the cause, however we are going to upgrade the server as this has been needed for a while, this wioll result in a little downtime during the day.

Wed, 01 Jan 2020 12:05:23 +0000 #

webarch1.co.uk is up and running again as normal, sorry for any inconvenience caused.

Wed, 01 Jan 2020 11:54:29 +0000 #

We are shutting down webarch1.co.uk for a little while while we add additional disk space to it.

Mon, 16 Dec 2019 10:48:55 +0000 #

We are restarting the webarch.email server, it will be unavailable for a few minutes.

Thu, 12 Dec 2019 18:02:45 +0000 #

We are doing a quick restart of the mail.webarch.email services to enable a configuration change to be enabled.

Mon, 09 Dec 2019 20:30:29 +0000 #

The upgrade of mail.webarch.email has been completed, sorry for any inconvenience caused during the downtime.

Mon, 09 Dec 2019 20:00:19 +0000 #

We are upgading mail.webarch.email, it will be unavailable for a little while.

Mon, 09 Dec 2019 18:52:44 +0000 #

We intend to upgrade mail.webarch.email from 8pm this evening and expect that the server will be unavailable for a little while while this work is undertaken no email should be lost in this process but some might be delayed a little.

Wed, 04 Dec 2019 16:28:13 +0000 #

We have just generated a new TLS certificate for mail1.ecohost.coop so the problems when connecting to the mail server using the old Ecohost settings should now be resolved. However it would be better if people did update their email clients to use the webarch.email settings

Wed, 04 Dec 2019 13:59:41 +0000 #

We are very sorry for the inconvenience caused by the lapsing of the ecohost.coop domain name, this has now been renewed.

Mon, 21 Oct 2019 22:59:24 +0100 #

Sorry for the interruptions in service between 10pm and 10:50pm BST, the new router is now in place and we hope that services will be better than ever as a result.

Mon, 21 Oct 2019 22:01:10 +0100 #

We are installing a new router at our data centre in Sheffield, all Sheffield based services will be unavailable for a little while.

Mon, 21 Oct 2019 13:21:08 +0100 #

We are planning on replacing the temporary router we have in our Sheffield data centre with a new one from 10pm BST tonight, this will result in a little downtime but should solve the limited incoming bandwidth issues we currently have.

Sat, 19 Oct 2019 10:02:50 +0100 #

We believe that the issues with webarch.email this morning have now been fully resolved and that no email was lost (some will have been delayed), once again sorry for the downtime.

Sat, 19 Oct 2019 09:34:00 +0100 #

We are still having some problems with webarch.email, we hope to have them resolved soon.

Sat, 19 Oct 2019 08:46:55 +0100 #

We have resolved the issue with the Mailcow upgrade and webarch.email is up and running again, sorry for the downtime.

Sat, 19 Oct 2019 07:32:17 +0100 #

We are currently upgrading Docker on webarch.email and are afraid that this is taking longer than usual.

Wed, 16 Oct 2019 16:55:09 +0100 #

We are very sorry about the connectivity issues we have had today with all our services hosted in Sheffield, this was due to an inadequate temporary replacement router, we now have a second replacement router one in place and are currently doing some testing on it, we hope it will sufficient as a temporary solution while we source and provision a pair of new replacements.

Wed, 16 Oct 2019 10:29:50 +0100 #

We have just done a very a quick reboot of the replacement router, it was only down for a moment, we do not anticipate any further downtime, once again we are very sorry about the problems this morning.

Wed, 16 Oct 2019 10:23:35 +0100 #

We are doing a quick reboot of the replacement router, it should be back up in a moment.

Wed, 16 Oct 2019 09:56:38 +0100 #

Our sheffield services are restored, we are very sorry for the downtime and the inconvenience caused.

Wed, 16 Oct 2019 09:46:33 +0100 #

We are in the process of configuring a replacement router.

Wed, 16 Oct 2019 09:23:22 +0100 #

We hope to have all services restored by 10am BST.

Wed, 16 Oct 2019 09:01:08 +0100 #

All our Sheffield services are unavailable following a reboot of our main router, which failed to come back up, we are on our way to the data centre to resolve the problem.

Sun, 13 Oct 2019 15:59:59 +0100 #

We are doing some work on our router at 16:00 GMT today and we expect connectivity to all our Sheffield servers to be lost at this time for around one minute.

Wed, 02 Oct 2019 10:03:39 +0100 #

Sorry for the git.coop downtime this morning, we are upgrading it and giving it additional resources (more RAM and CPUs).

Sun, 22 Sep 2019 14:02:37 +0100 #

We are updating GitLab at git.coop and it will be unavailable for a little while.

Fri, 20 Sep 2019 22:29:30 +0100 #

The upgrade for webarch.email has completed.

Fri, 20 Sep 2019 22:00:35 +0100 #

We are upgrading webarch.email and as a result it will be unavailable for a little while.

Tue, 03 Sep 2019 22:13:37 +0100 #

We are upgrading Docker across multiple servers and this will result in some services being unavailable for a while.

Mon, 02 Sep 2019 22:02:53 +0100 #

The problem with the certificate for webarch.email has been fixed.

Mon, 02 Sep 2019 20:50:36 +0100 #

The upgrade for webarch.email is complete.

Mon, 02 Sep 2019 20:24:58 +0100 #

We are upgrading webarch.email so it will be un-available for a little while.

Sun, 25 Aug 2019 02:13:41 +0100 #

We have booted the server and are now bringing up the virtual servers one at a time.

Sun, 25 Aug 2019 01:42:59 +0100 #

We are at the data centre and checking the BIOS on the sever.

Sun, 25 Aug 2019 01:14:01 +0100 #

We are going to the data centre to investigate the problem with the server that is down.

Sat, 24 Aug 2019 23:12:14 +0100 #

One of our front facing servers is currently off-line and this is effecting several of our services, we are looking into the cause of this problem.

Mon, 22 Jul 2019 20:18:12 +0100 #

We are upgrading Docker across all our servers and this will result in some downtime for a variety of applications, the key one being webarch.email.

Wed, 17 Jul 2019 20:28:39 +0100 #

We are upgrading Docker across all our servers and this will result in some downtime for a variety of applications, the key one being webarch.email.

Wed, 03 Jul 2019 11:30:18 +0100 #

We are upgrading GitLab at git.coop so it will be off-line for a little while.

Tue, 18 Jun 2019 07:39:23 +0100 #

We are rebooting services due to last nights Linux kernel update.

Wed, 05 Jun 2019 14:05:55 +0100 #

We are upgrading out GitLab server at git.coop so it will be unavailable for a few minutes.

Mon, 03 Jun 2019 20:46:34 +0100 #

We are about to apply the May updates to webarch.email, it will therefore be unavailable for a few minutes.

Mon, 03 Jun 2019 12:11:16 +0100 #

We are updating the GitLab server at git.coop and it will therefore be unavailable for a few minutes.

Mon, 22 Apr 2019 19:14:00 +0100 #

The update to the webarch.email server has been completed.

Mon, 22 Apr 2019 19:07:48 +0100 #

We are shutting down the webarch.email server to apply some updates, it shouldn't take long.

Tue, 09 Apr 2019 11:03:32 +0100 #

The webarch.email server is up and running again, sorry for any inconvenience caused.

Tue, 09 Apr 2019 10:54:25 +0100 #

We are restarting the webarch.email server to disable the Solr search index (without Solr searching emails using SOGo still works but is slower), the server will be unavailable for a little while.

Sat, 06 Apr 2019 08:59:55 +0100 #

The full re-scan of the email on webarch.email has completed and the search facility in the SOGo interface is now working correctly.

Thu, 04 Apr 2019 06:25:52 +0100 #

We have started a full rescan of email on webarch.email to try to solve the issues there have been with the SOGo search not returning full sets of results, this will result in the server running with a high load for some time and this could have an impact for users.

Mon, 01 Apr 2019 22:24:52 +0100 #

The work on webarch.email has completed.

Mon, 01 Apr 2019 21:55:26 +0100 #

We are adding additional resources to the webarch.email server and it will be down for a little while.

Sat, 26 Jan 2019 16:01:45 +0000 #

The problem with webarch.email has been resolved.

Sat, 26 Jan 2019 15:02:04 +0000 #

We are having problems with the web interface of webarch.email following an update, we hope to have this resolved soon, non-web based email sevices are working normally.

Thu, 24 Jan 2019 15:53:25 +0000 #

We appologise for the issues with our servers in Sheffield today, there has been intermittent slow disk speeds and high iowait times, we haven't yet found the cause of this.

Mon, 14 Jan 2019 20:00:30 +0000 #

The upgrade to webarch.email has completed, sorry for the downtime.

Mon, 14 Jan 2019 19:50:58 +0000 #

We are applying the January updates to webarch.email and it will be unavailable for a little while.

Sat, 12 Jan 2019 12:16:29 +0000 #

webarch.email is up and running again.

Sat, 12 Jan 2019 12:10:17 +0000 #

We are restarting webarch.email as part of the planned migration of email accounts from mail1.ecohost.coop, it should be up again soon.

Sat, 12 Jan 2019 10:56:52 +0000 #

We are very sorry to say that a client's email account on mail1.ecohost.coop has been compromised and used to send a massive volume of spam and this has caused the server to be blacklisted, we have secured the account and are now seeking to have the server de-blacklisted.

Sat, 29 Dec 2018 11:44:54 +0000 #

The disk resizing has completed, webarch2.co.uk is up and running again.

Sat, 29 Dec 2018 11:40:55 +0000 #

We are shutting down webarch2.co.uk to add additional disk space, this shouldn't take long.

Fri, 28 Dec 2018 19:13:13 +0000 #

The upgrade to webarch.email has completed, sorry for any inconvenience caused.

Fri, 28 Dec 2018 19:08:23 +0000 #

We are updating webarch.email and it will be unavailable for a little while.

Sat, 22 Dec 2018 17:35:36 +0000 #

The update to webarch.email has completed, sorry for the downtime.

Sat, 22 Dec 2018 17:20:45 +0000 #

We are updating webarch.email to get the latest Mailcow updates, the server will be unavailable for a little while.

Sat, 08 Dec 2018 18:30:14 +0000 #

The upgrade of webarch.email has been completed.

Sat, 08 Dec 2018 18:19:10 +0000 #

We are upgrading webarch.email and it will therefore be unavailable for a little while.

Fri, 23 Nov 2018 08:23:31 +0000 #

Our GitLab server at git.coop is up and running again, for more detail about the disk space issue see this issue. Sorry for any inconvenience caused.

Fri, 23 Nov 2018 07:19:43 +0000 #

Sorry about the downtime for git.coop, we are currently growing the disk space.

Mon, 19 Nov 2018 19:25:15 +0000 #

We are upgrading webarch.email and it will be unavailable for a little while

Wed, 14 Nov 2018 20:20:37 +0000 #

The upgrade of webarch.email has completed, sorry for any inconvenience caused.

Wed, 14 Nov 2018 20:00:06 +0000 #

We are upgrading webarch.email as as as result it will be unavailable for a little while.

Fri, 12 Oct 2018 15:08:33 +0100 #

We have just restarted the webarch.email server to address some issues that it was having.

Wed, 03 Oct 2018 12:01:55 +0100 #

We have requested the removal of webarch.email from as many anti-spam blacklists as we can, however it might be up to a week before the server is fully delisted, we are very sorry for any inconvenience caused by this.

Wed, 03 Oct 2018 09:35:30 +0100 #

We are sorry about the problems with webarch.email this morning, it was due to an account being compromised and huge volumes of spam being sent.

Sat, 15 Sep 2018 09:43:40 +0100 #

We have upgraded webarch.email and it is available again.

Sat, 15 Sep 2018 09:35:43 +0100 #

We are upgrading webarch.email and it will be unavailable for a little while.

Tue, 04 Sep 2018 20:09:36 +0100 #

The update to webarch.email has been completed and we have also updated the IP address of this status page (see this article for the background) so it soon be available again. We are sorry for any inconvenience caused.

Tue, 04 Sep 2018 19:28:18 +0100 #

We are about to upgrade webarch.email and it will be unavailable for about ten minutes.

Wed, 22 Aug 2018 08:00:00 +0100 #

We have completed the upgrade of webarch.email and are sorry for any inconvenience caused.

Wed, 22 Aug 2018 07:48:02 +0100 #

We are going to upgrade webarch.email and it will be unavailable for a few minutes.

Thu, 16 Aug 2018 07:47:28 +0100 #

We have completed the maintainance on webarch.email, sorry for any inconvenience caused.

Thu, 16 Aug 2018 07:33:02 +0100 #

We are updating webarch.email, it will be unavailable for a few minutes.

Tue, 31 Jul 2018 19:21:50 +0100 #

We believe that webarch.email is fully up and running again, sorry for the downtime.

Tue, 31 Jul 2018 18:53:31 +0100 #

We are updating and restarting the webarch.email server and it will be unavailable for a short while as a result.

Wed, 04 Jul 2018 16:20:06 +0100 #

We are sorry that we are having some issues with our mail server at webarch.email, we hope to have this resolved as soon as possible.

Tue, 03 Jul 2018 19:48:19 +0100 #

We have added additional RAM to webarch.email and updated the server and it is now up and running again, sorry for any inconvenience caused.

Tue, 03 Jul 2018 19:31:14 +0100 #

We have had some issues with webarch.email server this afternoon and we are going to add some additional RAM to the server and reboot it, it will be unavailable for a little while.

Tue, 29 May 2018 19:14:13 +0100 #

The webarch.email server is up and running again.

Tue, 29 May 2018 18:55:06 +0100 #

We are uggrading and restarting the webarch.email server, it will be unavailable for a little while.

Mon, 30 Apr 2018 19:58:58 +0100 #

We are about to upgrade webarch.email it will shortly be unavailable for about 5 minutes.

Sat, 21 Apr 2018 13:06:01 +0100 #

We are rebuilding all our DNS servers, the work on the Sheffield ones, dns0.webarchitects.co.uk and dns1.webarchitects.co.uk has been completed. The European server, dns2.webarch.info has been rebuilt and it is now based in Paris and has a new IP address, 217.70.190.127. Our Icelandic server, dns3.webarch.info, is in the process of being rebuilt and when this is completed it will also have a new IP address, 185.112.146.79.

Wed, 11 Apr 2018 06:50:23 +0100 #

The work on webarch.email has been completed.

Wed, 11 Apr 2018 06:45:10 +0100 #

We are doing some maintenance on the webarch.email server, it will be unavailable for a few minutes.

Mon, 19 Mar 2018 12:13:28 +0000 #

We are restarting webarch.email, it should be back up in a moment.

Wed, 21 Feb 2018 21:51:57 +0000 #

The webarch.email server has been upgraded and is available again.

Wed, 21 Feb 2018 21:32:54 +0000 #

We are upgrading the webarch.email server and expect it to be unavailable for a few minutes.

Wed, 10 Jan 2018 11:22:24 +0000 #

We are seeing very high loads on host3.webarch.net and we have shutdown Apache and MySQL on the server while looking into this issue. host2.webarch.net is running a filesystem check and should be up soon.

Wed, 10 Jan 2018 11:06:38 +0000 #

We are restarting all the virtual servers on xen2.webarch.net.

Wed, 10 Jan 2018 10:46:53 +0000 #

We have resolved the issues with the connection to the IPMI on xen2.webarch.net and are once again shutting down all the virtual servers on the server.

Wed, 10 Jan 2018 10:27:07 +0000 #

We are having issues connecting to the IPMI on xen2.webarch.net, so it hasn't yet been rebooted, so we are restarting the virtual servers on it and will shut them down again when we are ready to reboot the server, sorry for this delay.

Wed, 10 Jan 2018 09:58:34 +0000 #

We are shutting down all the virtual servers on xen2.webarch.net.

Wed, 10 Jan 2018 09:56:02 +0000 #

We are restarting all the virtual servers on xen1.webarch.net.

Wed, 10 Jan 2018 09:38:14 +0000 #

We are rebooting xen1.webarch.net.

Wed, 10 Jan 2018 09:28:13 +0000 #

We are shutting down all the virtual servers on xen1.webarch.net.

Fri, 05 Jan 2018 11:38:44 +0000 #

Due to the Meltdown and Spectre exploit critical vulnerabilities we are planning on rebooting all our Sheffield servers during the course of the morning of Wednesday 10th January 2018, we don't anticipate that there will be a great amount of downtime but all services will be affected for a while.

Thu, 04 Jan 2018 12:59:53 +0000 #

We are expecting to have to reboot all our servers at some point in the next few days, when Debian updates are available for these security issues.

Thu, 28 Dec 2017 16:02:18 +0000 #

The webarch.email server has been upgraded and appear to be working without issues.

Thu, 28 Dec 2017 14:23:58 +0000 #

We are going to upgrade webarch.email this afternoon, an email announcing this has been sent to all mailboxes on the server.

Thu, 16 Nov 2017 21:32:33 +0000 #

Restoration of all the virtual servers in Iceland is now looking hopeful, the lastest from @1984ehf: "24 hours ago we thought that nearly all VPS data was corrupt beyond repair. We have since found a way to restore disks from the VPSs. This means that we will be able to retrieve data from a lot of the VPS disks and we will help every one of our VPS customers to find their data.".

Thu, 16 Nov 2017 12:32:36 +0000 #

We don't yet know if the virtual servers in Iceland can be restored, see @1984ehf for the latest news on this. We are starting to restore sites that were hosted on host-b.ecodis.net, from backups, to shared servers in Sheffield.

Thu, 16 Nov 2017 09:10:24 +0000 #

An update on services in Iceland from @1984efh: "After spending a night with a team of experts in the hardware field, our costly setup was deemed fit for scrap. We have a backup in a distant location and are now working on making it functional from that location.".

Wed, 15 Nov 2017 17:46:34 +0000 #

We are aware that some services in Iceland are down, please see the 1984.is site for updates.

Tue, 14 Nov 2017 18:28:48 +0000 #

We are pleaase to say that mail.webarch.net is back up again and we believe we have solved the cause of the downtime.

Tue, 14 Nov 2017 17:29:04 +0000 #

Sorry about the downtime for mail.webarch.net, it should be up again soon.

Sat, 11 Nov 2017 17:22:54 +0000 #

We are pleased to say that mail.webarch.net is up and running again, sorry for the downtime.

Sat, 11 Nov 2017 15:24:30 +0000 #

We expect the filesystem check being run on mail.webarch.net to take several hours to complete and the server won't be up before this completes. Sorry for the downtime.

Sat, 11 Nov 2017 13:56:16 +0000 #

Veryr sorry about the on-going downtime for mail.webarch.net, we hope to have it back up ASAP.

Sat, 11 Nov 2017 12:59:05 +0000 #

We have had an issue with mail.webarch.net and we are currently restarting the server.

Wed, 08 Nov 2017 21:34:10 +0000 #

We are plased to say that webarch.email is up now and working normally.

Wed, 08 Nov 2017 21:23:48 +0000 #

We are having to reboot webarch.email sorry for the downtime, it should be back up shortly.

Thu, 05 Oct 2017 19:30:06 +0100 #

We believe all our systems are up and running again normally, very sorry for the downtime.

Thu, 05 Oct 2017 19:01:37 +0100 #

We are afraid that we have a problem with our front facing Xen servers in Sheffield and we are having to reboot them, this will result in some downtime for all our Sheffield based services.

Fri, 22 Sep 2017 10:12:02 +0100 #

The Sheffield connectivity issue this morning was caused by a problem with a router in London, we have spoken to our data centre and they have resolved the issue.

Fri, 22 Sep 2017 09:54:47 +0100 #

We are looking into connectivity issues at our Sheffield data centre.

Sat, 16 Sep 2017 15:14:47 +0100 #

We believe that all our servers are now up and running again, please get in contact if you are experiencing any issues, sorry for any inconvenience caused.

Sat, 16 Sep 2017 14:05:50 +0100 #

Our Mailman server is now up and running again.

Sat, 16 Sep 2017 13:16:35 +0100 #

We should have the mailman server up and running very soon. We are afraid it is going to be a little longer for mail.webarch.net and the MKDoc server.

Sat, 16 Sep 2017 12:26:41 +0100 #

We are copying the mail.webarch.net disk, 90G done out of 275G, in one hour, at this rate we are afraid it is probably going to be down until around 3:30pm.

Sat, 16 Sep 2017 11:27:35 +0100 #

We are doing some additional work on mail.webarch.net, it will be down for a little while.

Sat, 16 Sep 2017 11:06:12 +0100 #

mail.webarch.net is back up and we have also fixed the issue with the webserver on webarch1.co.uk. We hope to bring up the Mailman server very soon.

Sat, 16 Sep 2017 04:30:07 +0100 #

We are afraid it is going to be some time before mail.webarch.net is up and running again, we are copying the disk across the network and have copied 41G out of 275G — it is going to take some time to complete.

Sat, 16 Sep 2017 03:35:10 +0100 #

mail1.ecohost.coop is now up again.

Sat, 16 Sep 2017 03:26:27 +0100 #

The file system check on mail1.ecohost.coop is 53% complete and might take another hour or two to complete, then that server will come up again. The server that was running several CentOS servers, cat.webarchitects.co.uk, which had an uptime of 2017 days, had two RAM chips fail after the move, after removing them the server still won't boot, so we moved the disks into another server and used a rescue disk to repair the RAID 0 boot partition on all the disks. The server that the disks from cat are now running in doesn't have enough RAM to run the virtual servers, so we are now copying the partitions over the network to our FreeBDS ZFS file server and when that is completed we will start them up on xen1.webarch.net. This means that it is going to be quite a few hours before these servers are up:

• The mail.webarch.net Webarchitects mail server
• The Mailman server which hosts email-lists.org and several other domains
• Our Gitosis server
• Our MKDoc server

Sorry for the downtime.

Sat, 16 Sep 2017 01:49:04 +0100 #

The Ecohost mail1.ecohost.coop server is running a file system check and should be up soon.

Sat, 16 Sep 2017 01:18:09 +0100 #

We believe all our Sheffield services are up and running again apart from mail.webarch.net, our Mailman server and our MKDoc server, we have had some hardware issues with the physical server these virtual servers run on and are trying to solve the problems.

Fri, 15 Sep 2017 22:00:59 +0100 #

We are starting to shutdown services in our Sheffield data centre for the room move.

Fri, 15 Sep 2017 21:20:54 +0100 #

We will start shutting down servers in our Sheffield data centre at 10pm BST to move them to a new room, will post updates as the work progresses, we are at the data centre now.

Thu, 24 Aug 2017 11:16:35 +0100 #

We believe that all services are now running normally, please contact us if you have any issues.

Thu, 24 Aug 2017 10:45:10 +0100 #

We have some issues with xen1.webarch.net which runs multiple virtual servers and as a result we are in the process of migrating services to xen3.webarch.net, very sorry for any inconvenience caused.

Tue, 25 Jul 2017 21:44:17 +0100 #

The mail1.ecohost.coop server has been removed from the SORBS blacklist and should soon be removed from the Barracuda blacklist.

Tue, 25 Jul 2017 19:24:59 +0100 #

We have requested that mail1.ecohost.coop is delisted from the SORBS and Barracuda email server blacklists, sorry for the inconvenience these listings might be causing.

Tue, 25 Jul 2017 15:52:22 +0100 #

We are aware of issues regarding mail.webarch.net and we are investigating the cause of these problems.

Sun, 23 Jul 2017 21:58:22 +0100 #

One account on mail.ecohost.coop was being used to send spam, we have stopped the abuse, but there is a possibility that this will have a negative impact on other users of the server if the server is blacklisted.

Wed, 05 Jul 2017 17:48:22 +0100 #

Our new mailcow server, webarch.email, is now up and running again, very sorry for the downtime.

Wed, 05 Jul 2017 14:14:09 +0100 #

We are working on restoring webarch.email from a 2am backup from 5th July 2017.

Wed, 05 Jul 2017 11:20:52 +0100 #

All services, apart from webarch.email are now up and running, we are still working on bringing webarch.email back online. Sorry for the inconvenience.

Wed, 05 Jul 2017 09:22:49 +0100 #

We are in the process of restarting all the servers on xen1.webarch.net, very sorry for the downtime.

Wed, 05 Jul 2017 08:57:45 +0100 #

We are having to reboot xen1.webarch.net due to kernel issues across all the virtual servers on this machine, we are afraid this is resulting in some downtime for a large number of services.

Wed, 05 Jul 2017 08:50:54 +0100 #

We have had some problems with webarch.email and are having to reboot the server, it should be up again in a moment.

Fri, 23 Jun 2017 23:03:24 +0100 #

The planned maintainance on mail.webarch.net and webarch.email has now been completed and all our mail services are up and running again, sorry for any inconvenience caused.

Fri, 23 Jun 2017 20:25:32 +0100 #

We are about to conduct some planned maintainance on mail.webarch.net and webarch.email, this will involve an interuption to your email service, but no email should be lost, we will post up update when the work has been completed.

Fri, 19 May 2017 12:23:29 +0100 #

Some of our hosted sites which use Let's Encrypt HTTPS certificates are currently unavailable due to Let's Encrypt OSCP servers being down, see the Let's Encrypt Status Page

Sun, 30 Apr 2017 08:05:36 +0100 #

Very sorry for the downtime this morning, all systems are now up and running again.

Sun, 30 Apr 2017 06:52:27 +0100 #

We are now bringing the virtual servers up one at a time and each file system is being checked, it will be a little while before they are all running again, very sorry for this downtime.

Sun, 30 Apr 2017 05:45:37 +0100 #

We are about to reboot one of our main physical servers in Sheffield which hosts multiple virtual servers so quite a few servers will be unavailable for a while

Sun, 30 Apr 2017 01:54:29 +0100 #

We are currently having some issues with webarch1.co.uk, we hope it will be available again shortly.

Tue, 18 Apr 2017 13:12:43 +0100 #

Sorry about the problems with web2.ecohost.coop, the server hosts a Labour MPs site which is under a very high load today.

Wed, 22 Feb 2017 08:45:40 +0000 #

Very sorry for host3.webarch.net being down for just over an hour this morning, it is now up again.

Mon, 13 Feb 2017 12:33:45 +0000 #

host1.webarch.net and host3.webarch.net are now up and running again, very sorry for the downtime

Mon, 13 Feb 2017 12:26:51 +0000 #

We are rebooting host1.webarch.net and host3.webarch.net for essential maintenance.

Sat, 04 Feb 2017 09:16:03 +0000 #

host1.webarch.net and host3.webarch.net went offline between 1:20am and 1:30am this morning and were not available until 8:30am this morning, we sincererly appologise for any inconvenience and we are investigating the cause of the issue.

Sat, 04 Feb 2017 08:32:41 +0000 #

We appologise for the downtime that host1.webarch.net and host3.webarch.net have just had, the servers are now rebooting and should be up again in a moment.

Tue, 24 Jan 2017 18:31:23 +0000 #

We are sorry that we just had a little downtime across several shared hosting servers in Sheffield due to a large load spike, we are looking into the cause.

Sat, 21 Jan 2017 21:24:52 +0000 #

host3.webarch.net is back up, sorry for any inconvenience caused.

Sat, 21 Jan 2017 21:20:29 +0000 #

We are expanding the disk on host3.webarch.net, it should be up again soon. Sorry for the downtime.

Mon, 09 Jan 2017 15:12:18 +0000 #

We have resolved an issue with payments of invoices via PayPal, invoice payments via Paypal are now working correctly, sorry for any inconvenience caused.

Thu, 05 Jan 2017 11:45:09 +0000 #

We are very sorry to say that we have discovered that our ticketing system was not accepting tickets for 14 days over Christmas and the New Year due to a misconfiguration error. If you tried to contact us during this period you should have recieved a Undelivered Mail Returned to Sender response, please resend your original email as the ticketing system is now up and running again. We appologise for any inconvenience caused by this outage.

Mon, 28 Nov 2016 13:19:06 +0000 #

We are seeing some connectivity issues with servers and services in Iceland this afternoon.

Thu, 03 Nov 2016 13:58:43 +0000 #

The fault with our office phone line has now been resolved and our answerphone is available again when nobody is available.

Thu, 03 Nov 2016 10:29:40 +0000 #

There is currently an issue with our office phone line, The Phone Co-op are investigating the fault, best use email to contact us today.

Mon, 17 Oct 2016 14:06:53 +0100 #

Sorry so the loss of network connectivity for host3.webarch.net between 13:57 and 14:03, we are investigating the cause of this outage.

Tue, 11 Oct 2016 14:32:10 +0100 #

The mail1.ecohost.coop server has now been removed from all but one blacklist and for this one "Automatic removal will occur for IPs that are seen to be clean". We have sent an email to users of the server to explain what happened and to appologise for any inconvenience caused.

Tue, 11 Oct 2016 13:40:59 +0100 #

The mail queue on mail1.ecohost.coop has now cleared but we are sorry to say that the server is now listed on some blacklists due to a compromised account on the server being used to send a large volume of spam — we are now working to have it removed from the blacklists. Sorry for any inconvenience caused by this and please ensure that all the devices you use for email are secure!

Tue, 11 Oct 2016 12:34:51 +0100 #

Please bear with us as the mail queue on mail1.echost.coop clears — until then there will be some delays in outgoing emails.

Tue, 11 Oct 2016 11:16:54 +0100 #

The mail1.ecohost.coop server is back up with additional disk space, once again sorry for any inconvenience caused.

Tue, 11 Oct 2016 09:57:44 +0100 #

Very sorry for the inconvenience but there is going to be a little downtime for mail1.ecohost.coop as we need to add additional disk space to the server.

Thu, 06 Oct 2016 19:51:27 +0100 #

The mailserver mail.webarch.net has been rebooted with an extra 2G of RAM and the MKDoc server is also back up. Sorry for any inconvenience caused by the short downtime.

Thu, 06 Oct 2016 19:09:20 +0100 #

We are about to reboot the Webarchitects mailserver mail.webarch.net in order to add additional RAM to it. There will be a short downtime for the MKDoc server as we are removing some RAM from that machine as it is running fewer sites these days.

Tue, 04 Oct 2016 14:22:30 +0100 #

Sorry to say that there has just been another short outage to services in Iceland, between 14:16:42 and 14:18:33, we will post an update here when we know the cause of the connectivity issues today.

Tue, 04 Oct 2016 12:20:56 +0100 #

There has just need another short connectivity issue with Iceland based services, the network was down between 12:14 and 12:15 BST, it is back up now.

Tue, 04 Oct 2016 10:52:22 +0100 #

Network connectivity to servers and services running in Iceland has been restored.

Tue, 04 Oct 2016 10:49:59 +0100 #

We are experiencing network connectivity issues with our servers in Iceland, we are investigating the cause of this problem.

Fri, 30 Sep 2016 11:25:20 +0100 #

Barracuda Networks have responded to the delisting request for mail1.ecohost.coop as follows:

"We have removed 81.95.52.108 (Please wait 24-48 hours) from our blocklist for 30 days, at which time it will be re-evaluated."

You can check on the delisting progrees a Debouncer Blacklist Check.

Fri, 30 Sep 2016 11:06:57 +0100 #

We believe we have discovered and blocked the abuse of services running on mail1.ecohost.coop and we are now requesting that this server is removed from blacklists. Once again we are sorry for any inconvenience caused by this.

Thu, 29 Sep 2016 20:11:47 +0100 #

One of our email servers, mail1.ecohost.coop is currently listed as having a 'poor' reputation on the Barracuda Reputation System, this is causing some email delivery issues, we are investigating the cause and when we have tracked it down we will request to be delisted. Sorry for any inconvenience caused by this.

Wed, 28 Sep 2016 13:43:28 +0100 #

This site is being launched today, all services and servers are up and running without issues.