Webarchitects server status updates — if there are any issues with our servers or services we will post updates to this page, we also use the #webarch channel on irc.libera.chat, there is a web interface for #webarch chat and our office phone number is 0114 276 9709.
We are upgrading Discourse on all our servers so all hosted Discourse sites will be unavailable for a little while.
We are upgrading Docker on all our servers so services that depend on it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be offline for a little while.
We are updating all servers that use Docker (Mailcow and Discourse) so they will have a little downtime.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are updating our Mailcow servers so webarch.email and others will be unavailable for a little while.
We are shutting down webarch1.co.uk for a little while so we can extend one of it's disks, we don't expect it to be offline for more that a few minutes.
We are very sorry but due to a misconfiguraton error, which we are currently fixing, sites on shared hosting on an account with a MediaWiki site have had a little downtime.
We are updating all our Mailcow servers to apply a fix for a memory leak in Rspamd, this means that services such as webarch.email will be unavailable for a little while.
We are updating all our hosted Discourse servers to version 2.8.0.beta5 so they will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
Maintenance has finished on host xen4.webarch.net and the affected services on webarch.email, runner.git.coop, cloud.leedsbread.coop, holyoake.webarch.coop, sharenergy.webarch.net, host3.webarch.net, host2.webarch.net, webarch4.co.uk, webarch7.co.uk, webarch1.co.uk, webarch2.co.uk, webarch3.co.uk, webarch6.co.uk, webarch5.co.uk, git.coop and animorph.webarch.coop should be functioning again.
Essential network maintenance will be starting at 21:00 GMT (22:00 UK BST). Affected hosts are animorph.webarch.coop, cloud.leedsbread.coop, git.coop, holyoake.webarch.coop, host2.webarch.net, host3.webarch.net, runner.git.coop, sharenergy.webarch.net, webarch1.co.uk, webarch2.co.uk, webarch3.co.uk, webarch4.co.uk, webarch5.co.uk, webarch6.co.uk, webarch7.co.uk and webarch.email
We are upgrading our Discourse servers to Debian Bullseye this afternoon so there will be a little downtime when they are all rebooted.
We are upgrading the Mailcow servers so webarch.email and others will be unavailable for a little while.
The upgrade to the shared hosting servers, webarch1.co.uk, webarch2.co.uk, webarch3.co.uk, webarch4.co.uk, webarch5.co.uk, webarch6.co.uk and webarch7.co.uk are more-or-less complete, we are now going to migrate the Gitlab server at git.coop to a different set of disks and as a result it will be unavailable for a little while.
We are upgrading our main shared hosting servers from Debian Buster to Debian Bullseye, webarch1.co.uk, webarch2.co.uk, webarch3.co.uk, webarch4.co.uk, webarch5.co.uk, webarch6.co.uk and webarch7.co.uk so there will be around an hour of downtime for sites hosted on these servers at some point this evening.
The upgrades for host2.webarch.net and host3.webarch.net have completed but we are still running some tasks to check all the configurations, if you have any issues with sites on either of these servers please email support@webarchitects.coop and we will address any problems tomorrow morning.
The upgrade of host3.webarch.net to Debian Bullseye is mostly completed, hosted sites should be working again, there are still some additional tasks to run and the sites on host2.webarch.net should be up again fairly soon.
We are upgrading two shared hosting servers, host2.webarch.net and host3.webarch.net to Debian Bullseye this evening starting at 8pm BST, sites on these servers will be unavailable for a while during the upgrade process.
We are upgrading all our Mailcow servers so webarch.email will be unavailable for a little while.
we are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading our Mailcow servers so webarch.email and other servers will be unavailable for a little while.
We are upgrading Docker on all our servers and this will result in a little downtime for services that run in Docker containers, specifically Mailcow and Discourse servers.
We are going to shutdown webarch6.co.uk to add some additional disk space to the server, it shouldn't be down for long.
We need to expand the disks on several shared hosting servers and in order to do this several servers will be shutdown for a little while.
We are upgrading Docker on all our servers so services that depend on it, like Mailcow and Discourse, will be restarted.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Discourse on all our servers so they will be unavailable for a little while.
We are upgradng GitLab on git.coop so it will be unavailable for a little while.
Due to some issues with the Execution of Sieve filters on webarch.email we are going to restart the server, this will result ina little downtime.
We are upgrading Docker on all our servers, this will result in a little downtime for servers that use Docker, specifically Discourse and Mailcow servers.
We are upgrading Discourse on all our servers, this means that Discourse sites will be unavailable for a little while.
We are going to shutdown webarch6.co.uk for a little time so we can extend one of the servers disks, so sites hosted on this server will be unavailable for a little while.
We are upgrading our Mailcow servers so webarch.email and other servers will be unavailable for a little while this evening.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab for git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Discourse on all our hosted servers so these forums will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrdaing Docker on all our servers that run it, this will cause a little downtime for servics such as Mailcow on webarch.email and Discorse,
We are upgradeing GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Docker across all our servers and this will result in a little downtime for services that depend on Docker, for example Mailcow on webarch.email and all our Discourse servers.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Discourse on all our servers so there will be a little downtime for servers running Discourse.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
All the services have been restored on older servers while we look into the issue with the new server. We believe that everything is running without issues but there wil be a email backlog for a while, if you find any problems please contact us.
We have brought up most servers effected by the issue with the connection to the disk array on other, older front facing servers, there are three servers still down, these include git.coop and webarch.email, we hope to be able to bring these back up soon.
We appear to have a problem with the mounting of the storage array on our latest front facing server so we are going to restart it, this should solve the issue.
We are upgrading all out hosted Discourse servers so they will be unavailable for a little while.
We are upgrading all the Discourse servers we host so they will be unavailable for a little while.
The webarch.email server is up and running again, very sorry about the downtime this morning.
We believe that all services have been restored.
We are sorry that our new front facing servers connection to the new disk array failed overnight, this resulted in the virtual servers it is running having errors / being out of action, we are restarting everything now.
We are updating adding some additional resources to the webarch.email server and this requires a restart so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be offline for a little while.
We are moving some servers to new hardware, this will result in a little downtime for each server as we power it down on the old computer and then bring it up again on the new one.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
We are upgrading Discourse on all our hosted servers so they will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Docker across all our servers and this will result in a little downtime for services that run via Docker, specifically Mailcow and Discourse.
We are upgrading all the Discourse servers we host so they will be unavailabe for a little while.
We are upgrading GitLab on git.coop, so it will be unavailable for a little while.
We are upgrading Mailcow at webarch.email, this will result in the service being unavailable for a little while.
The upgrade for GitLab at git.coop has been completed.
We are shutting down Gitlab at git.coop in order to extend one of the server disks, this upgrade shouldn't take long.
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
We have had the following response from Cloudmark: 'I have reset the reputation of your IP, so you should see delivery improve shortly. Please note that updates do not occur instantly but should generally happen within an hour of receiving this response.'
We are aware of email delivery issues to servers that use Cloudmark Sender Intelligence, we don't know what the reason for the CSI RBL listing is but we have requested removal.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We need to shutdown git.coop and
We are upgrading Discourse on all our servers so there will be a little downtime for these services.
There is a new containerd.io Docker package available and this means that we will need to restart Docker across all our servers, so Mailcow on webarch.email and all our Discourse servers will be unavailable while this package is upgraded.
We are upgrading a Docker package on all our servers and this will result in a little downtime for all services that run on Docker.
We are upgrading GitLab on git.coop so it will be offline for a little while.
We are updating Docker on all our servers so services that depend on it, Mailcow on webarch.email and all the Discourse servers will be unavailable for a little while.
The webarch4.co.uk server is back up.
We are shutting down the webarch4.co.uk shared hosting server to extend one of it's disks, it shouldn't be offline for long.
We are upgrading Docker on all our servers and this will result in a little downtime for services that run on Docker, specifically Mailcow and Discourse.
We are upgrading GitLab on git.coop so it will be offline for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Discourse on all our servers and this will result in them being offline for a little while.
Migrating git.coop to our new SSD file array is taking a little longer than expected, the biggest 200G disk in 65% done, we hope the transfer will be completed sometime after 6pm.
We are going to shutdown GitLab at git.coop in order to migrate it to the new SSD based storage array, it might be offiline for around an hour or so as a result, but will be faster when it is back up again.
We are shutting down GitLab at git.coop to grow the partition containing the Docker registry, it should only be offline for a little while.
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
The GitLab runner server, runner.git.coop is now up and running again.
We are migrating the GitLab runner server, runner.git.coop from our old DHH storage array to the new SSD storage array so it will be offline for around another half hour.
We are shutting down runner.git.coop to add additional space to the disks.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We need to shutdown host2.webarch.net for a little time to expand a disk, it shouldn't be offline for long.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgradinmg GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Docker on all our servers and this will cause a little downtime for services such as Mailcow on
We are upgrading Discourse on all the managed Discourse servers we host so they will be unavailable for a little while.
We have discovered an issue with our anti-spam gateway, mx.webarch.net this appears to have resulted in some email being delayed or returned, we are now adding some additional capacity to the server and it should be fully functional again shortly.
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgarding all our managed Discourse servers so they will be unavailable for a little while.
The upgrade of GitLab on git.coop has completed and the servers is up and running as normal, sorry for the downtime.
We are extending one of the disks on git.coop prior to the upgrade of GitLab, we expect it will be unavailable for another 10 minutes or so.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
Over the new few hours we are upgrading all Discourse servers to use PostgreSQL 13 and this migration process will result in a little downtime for each server.
We are upgrading Docker on all our servers, this will result in a little downtime for services that depend on it like Discourse and Mailcow on webarch.email
We are shutting down webarch4.co.uk to enable a disk to be extended, this shouldn't take long.
We are upgrading GitLab on git.coop, it will be offline for a little while.
We are upgrading GitLab at git.coop so it will be unavailable for a little while.
We are restarting webarch.email, sorry for te time it took to move the server and the inconvience caused.
50% of the main 1TiB webarch.email disk has now been copied, we anticipate that the mail service will be operational again at some point on Sunday morning, once again we are sorry about this downtime and the fact that it is lasting longer than expected.
We are afraid that copying the webarch.emal server from our old disk array to the new one is taking longer than expected, 33% of the main 1TB disk has been copied to the new server, at this rate we don't anticipate that the server will be back online until Sunday morning, we are very sorry for the inconvience this will cause people today.
The webarch.email file systems have been checked and we have started copying the disks to the new SSD based file server.
The webarch.email server is now shutdown and we are running checks on the disk image prior to copying them to the new file server.
We are shutting down webarch.email at 23:00 UTC in order to migrate it to a new server, due to the size of the server we anticipate that this will take around 7 hours. No email should be lost, other mail servers should queue the email for delivery when the server is up and running again.
We are upgrading Docker on all our servers, this will result in a little downtime for services that run in Docker containers, specifically Mailcow on webarch.email and all Discourse servers.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Docker on a few servers that were not updated this morning.
The issue with the EHLO/HELO on webarch.email has been resolved and we believe that delayed emails will now start to get through.
There is an issue with webarch.email related to this mornings Docker upgrade, it is currently not using the correct EHLO/HELO and this is resulting is some outgoing email being delayed, we are working with the Mailcow developer to resolve this problem ASAP.
The issue with the certificate for mail.webarch.email has been resolved, sorry for any problems this caused.
We are sorry about the certificate issues with mail.webarch.email, we are sorting that out now.
We are upgrading Docker across all our servers and this will cause a little downtime for services that depends on iy like webarch.email and Discourse servers.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Docker, Discourse and Mailcow over the next hour so services that run using these projects will have a little downtime.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are upgrading GitLab on
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
We are upgrading our managed Discourse servers, so they will all be offline for a little while.
The planned upgrade to latest version of mailcow completed successfully. Kate
webarch.email is being upgraded on advice from our support
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
It was the GitLab registry partition that was full on git.coop, it is now being doubled in size from 20GiB to 40GiB — this reflects how much Docker usage is happening!
We are shutting down our GitLab server at git.coop to add additional disk space to it, it should be up again soon.
We are restarting the GitLab service on git.coop to see if that resolves the issues the service has been having today.
All the virtual servers are up and runnning again, very sorry for the downtime.
We have powered the server that went down and are now starting to bring up all the virtual servers it hosts.
One of our front facing virtual server hosts has gone down and we are arranging to visit the data centre now to investigate the problem.
We are restaring webarch2.co.uk so sites hosted on this server will be unavalable for a little while.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading all our hosted Discourse servers so they will be unavailable for a little while.
We are are upgrading GitLab on git.coop so it will be unavalable for a little while.
We are upgrading Mailcow on webarch.email so it will be unavilable for a little while.
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
We are upgrading GitLab on git.coop and all our Discourse servers so these services will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop and also all the Discourse servers we host so there will be a little downtime for these services.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Mailcow on webarch.email to resolve an issue with SOGo autoreply messages, as a result the server will be unavailable for a little while.
The upgrade of webarch.email has completed, sorry for the downtime.
We are sorry about the on-going outage of webarch.email, we hope to have it up and running again soon.
We are upgrading Docker on all our servers, this will results in services that are run in Docker containers restarting and they will be unavailable for a little while, this includes
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgading GitLab on git.coop so it will be unavailable for a little while.
We are uogading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop and also Mailcow on webarch.email so both services will be unavailable for a little while.
We are updating GitLab at git.coop, so it will be unavailable for a little while.
We are currently upgrading all the Discourse servers we host, they will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be offline for a little while.
We are upgrading GitLab on git.coop so it will be offline for a little while.
We are upgrading GitLab on git.coop so it will be offline for a little while.
We have completed the move to the new file server for webarch7.co.uk.
We have completed the move for webarch6.co.uk and it is up and running again. We are now shutting down webarch7.co.uk to move it to the new file server.
We are moving webarch6.co.uk to our new file server so it will be offline for a while from 11pm BST.
We are shutting host3.webarch.net down for a short time to extend a disk.
The upgrade and reconfiguration of webarch4.co.uk has completed.
The upgrade and reconfiguration of webarch5.co.uk has completed.
The upgrade and reconfiguration of webarch2.co.uk has completed.
All four servers we are upgrading this weekend are now running the latest stable version of Debian, Buster, we are now in the process of reconfiguring all the user accounts on the servers and regenerating the Apache and PHP config for the switch from mod_php to php-fpm.
We are starting a planned upgrade on webarch5.co.uk so it will be unavailable for a while.
We are starting a planned upgrade on webarch4.co.uk so it will be unavailable for a while.
We are starting a planned upgrade on webarch3.co.uk so it will be unavailable for a while.
We are starting a planned upgrade on webarch2.co.uk so it will be unavailable for a while.
We are currently having an issue restarting MySQL on webarch4.co.uk, we hope to have this resolved soon.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We have completed the upgrade work on host3.webarch.net and webarch1.co.uk.
We are currently configuring users accounts on host3.webarch.net and upgrading webarch1.co.uk.
The upgrade work on host2.webarch.net has been completed.
We are upgrading GitLab on
We are shutting down webarch1.co.uk in order to copy it to our new file server, once it has been copied we will be upgrading it, so it will probably be unavailable for some time.
We are shutting down host3.webarch.net in order to copy it to our new file server, once it has been copied we will be upgrading it, so it will probably be unavailable for some time.
We are shutting down host2.webarch.net in order to copy it to our new file server, once it has been copied we will be upgrading it, so it will probably be unavailable for some time.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading Mailcow on webarch.email so email services be unavailable for a little while.
We are upgrading GitLab on git.coop and as a result it will be unavailable for a little while.
The issues with our old disk array this morning have been resolved. We have a new SSD based disk array in place and shall be migrating service to it out of office hours over the next few weeks.
We are upgrading Mailcow on webarch.email as it won't take much longer than a restart and there have been a few users who have has issues that appear to be related to the issues with the disk array earlire. It will be offline for a little while.
There is currently degraded performance on our old disk array and this is effecting most of our Sheffield based systems, we shall be migrating services to our new disk array as quickly as possible but we are afraid that this situation is probably going to take some time to fully resolve.
The members discourse forum and nextcloud are available again
The members discourse forum and office nextcloud is offline for maintenance
We are upgradeing GitLab on git.coop, it will be unavailable for a little while.
Scheduled maintenance this evening (20:00-24:00) is cancelled
The migration of git.coop back to the old disk array has completed.
We have having to migrate the GitLab server at git.coop back to our old storage array as we have found that the new array needs rebuilding before it can be used in production and we are sorry to say that we anticipate that this is going to result in the git.coop server being offline for a little over two hours.
git.coop will be offline between 20:00 - 24:00 06/07/2020 for maintenance
The migration and upgrade of git.coop has completed, sorry it took so long.
The git.coop GitLab server will be unavailable for a little while as we are upgrading it and migrating it to the new storage array.
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
We are upgrading Mailcow on webarch.email so it will be unavailable for a little while.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are in the process of upgrading Docker on all our server and this will result in services that depend on Docker being upavailable for a little while, sorry for any problems caused by this.
We will be stopping and restarting webarch.email to allow Docker to be upgraded, this will result in a little downtime.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
Sorry for the disruption to the webarch.email service this morning, it has now been updadet and allocated additional resources.
We are having some issues with webarch.email and as a result we are going to upgrade and restart it, it will be unavailable for a little while.
We are going to be shutting down webarch1.co.uk, webarch2.co.uk, webarch4.co.uk and webarch5.co.uk for short periods this evening to extend the the size of their disks.
We are upgrading all our hosted Discourse servers, they will be unavailble for a little while.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are upgrading Mailcow on webarch.email and extending a disk on webarch3.co.uk so these services will be unavilable for a little while.
We are upgrading GitLab on git.coop.
We are upgrading GitLab on git.coop.
We are upgrading Docker on our servers, this will result in services such as webarch.email being unavailable for a little while.
We are upgrading GitLab on git.coop so it will be down for a little while.
We are upgrading Docker on all our servers, this is going to result in a little downtime for everything that depends on it including webarch.email.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are manually rebuilding the Discourse servers we host to upgrade PostgreSQL, so there will be a little downtime this evening.
We are uugrading GitLab on git.coop so it will be unavailable for a little while.
A new version of Docker has been released so we are going to upgrade it across all our servers, this will cause a little downtime for services that run on Docker, including webarch.email.
We are upgrading GitLab on git.coop to version 12.10.6.
This mornings upgrades have completed, in a little while webarch.email should have fully restarted, all other services should already be up and running without issues.
We are about to update the Docker containerd.io package on all our servers, this will cause all the services running via Docker to be restarted so there will be some downtime for some things like webarch.email for a little while.
We are about to upgrade git.coop so it will be down for a little while.
We are upgrading GitLab on git.coop.
We are upgrading GitLab on git.coop.
We are sorry that the Apache web server stopped running on webarch3.co.uk just before 1:20am BST this morning, we haven't found the cause of this but the service was restored at 9:10am.
We are rebooting webarch6.co.uk to add additional CPUs to the server, it will be unavailable for a little while.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are upgrading webarch.email, it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be offline to a little time.
We are going to be upgrading webarch.email shortly and this will result in it being unavailable for a little while.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are going to upgrade webarch.email so it will be unavailable for a little while from around 11:00 UTC.
The webarch6.co.uk server is up now, we are very sorry for the downtime of services today.
We believe that all our servers are now back up apart from webarch6.co.uk, we are still working on this one. Please contact us at support@webarchitects.coop if you are having an issue with a service not on webarch6.co.uk.
We are having issues with MariaDB InnoDB recovery on several shared hosting servers.
We are now leaving the data centre, we will bring up the remaining servers remotly.
We have started bringing up the virtual servers, it will be a little while before they are all running again.
There appears to be no hardware problem with the file server, we are now decrypting the disks and bringing it back up, once this has been done we will be able to mount the filesystem from the front facing servers and then we will be able to start bringing all the virtual servers back up, this is going to probably take a couple of hours in total, very sorry for the inconvienience caused.
Our main file server is still booting.
Our main file server was not responding on the console so we are power cycling it, it is booting now. We will need to restart all the virtual servers after the file server is brought back up.
We are outside our Sheffield data centre waiting for a member of staff to let us in.
Our main file server had gone down and we are on our way to the data centre to investigate the cause.
We are having some issues with our infrastructure in Sheffield and are investigating the cause.
We are updating GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be unavailable for a little while.
We are upgrading GitLab on git.coop so it will be down for a little while.
We are aware of rate limiting of email from webarch.email to Gmail and Outlook, we believe that this has been triggered by a client who's email account was compromised and then used to send a large volume of spam. Please ensure that all the devices that you use to access your email accounts are secure and that all your email passwords are strong ones. We expect the rate limiting to be lifted in time.
We are shutting down webarch.email to add additional disk space, it will be unavailable for a little while.
We are upgrading GitLab so git.coop will be unavailable for a little while.
We are upgrading Docker on all our servers, this will case a little downtime for servers that use Docker like webarch.email.
We are upgrading GitLab on git.coop, it will be unavailable for a little while.
We are upgrade Docker on all our servers, this will result in a little downtime on webarch.email.
We are updating Docker on all our servers, this will result in a little downtime on servers that use Docker, for example webarch.email.
We are upgrading webarch.email so it will be unavailable for a little while.
We are rebooting host3.webarch.net and it is currently doing a file system check, it should be back online soon, sorry for any inconvenience caused.
Our upstream bandwidth provider had put mitigations in place to prevent the NTP reflection attack from disrupting our services, we hope this this is the end of the matter, however because we still have no idea what the reason for the attack was we have no idea if there is more to come, once again we are very sorry for any inconvenience caused.
It looks like it is a NTP reflection attack against webarch4.co.uk, we have no idea of the reason for it, our upstream bandwidth provider is working on mitigating the effect.
Our Sheffield network is suffering from a DDOS, we are working with our upstream partners to investigate the cause of this and to see what mitigations can be put in place. We are very sorry for any inconvenience caused.
We need to quickly reboot webarch6.co.uk, sorry that it would be available for a few minutes.
We believe that all systems are up and running now. There was a power outage for the whole area that our data centre is in and then the UPS for our rack (which is due to have replacement batteries by the end of the month) didn't manage to provide power for quite long enough to allow the backup generator to start up and therefore all our servers lost power. Once power was restored we needed to do a filesystem check on the router before decrypting and bringing up the storage array and once that was up we were able to bring up all the virtual servers, sorry that this all took a while, this is the first time in well over a decade that we have had an issue like this.
We hope to have all services restored by 2pm.
We are in the process of bringing services back up.
There has been a power failure at our Sheffield datacentre and this has caused all our Sheffield services to go down, we are on our way to the datacentre now.
The update to webarch.email has completed, sorry for any inconvenience caused.
We are updating webarch.email, it will be unavailable for a little while.
We are upgrading webarch.email so it will be unavailable for a little while.
There are some issues with the git.coop GitLab web interface today, we are unsure of the cause, however we are going to upgrade the server as this has been needed for a while, this wioll result in a little downtime during the day.
webarch1.co.uk is up and running again as normal, sorry for any inconvenience caused.
We are shutting down webarch1.co.uk for a little while while we add additional disk space to it.
We are restarting the webarch.email server, it will be unavailable for a few minutes.
We are doing a quick restart of the mail.webarch.email services to enable a configuration change to be enabled.
The upgrade of mail.webarch.email has been completed, sorry for any inconvenience caused during the downtime.
We are upgading mail.webarch.email, it will be unavailable for a little while.
We intend to upgrade mail.webarch.email from 8pm this evening and expect that the server will be unavailable for a little while while this work is undertaken no email should be lost in this process but some might be delayed a little.
We have just generated a new TLS certificate for mail1.ecohost.coop so the problems when connecting to the mail server using the old Ecohost settings should now be resolved. However it would be better if people did update their email clients to use the webarch.email settings
We are very sorry for the inconvenience caused by the lapsing of the ecohost.coop domain name, this has now been renewed.
Sorry for the interruptions in service between 10pm and 10:50pm BST, the new router is now in place and we hope that services will be better than ever as a result.
We are installing a new router at our data centre in Sheffield, all Sheffield based services will be unavailable for a little while.
We are planning on replacing the temporary router we have in our Sheffield data centre with a new one from 10pm BST tonight, this will result in a little downtime but should solve the limited incoming bandwidth issues we currently have.
We believe that the issues with webarch.email this morning have now been fully resolved and that no email was lost (some will have been delayed), once again sorry for the downtime.
We are still having some problems with webarch.email, we hope to have them resolved soon.
We have resolved the issue with the Mailcow upgrade and webarch.email is up and running again, sorry for the downtime.
We are currently upgrading Docker on webarch.email and are afraid that this is taking longer than usual.
We are very sorry about the connectivity issues we have had today with all our services hosted in Sheffield, this was due to an inadequate temporary replacement router, we now have a second replacement router one in place and are currently doing some testing on it, we hope it will sufficient as a temporary solution while we source and provision a pair of new replacements.
We have just done a very a quick reboot of the replacement router, it was only down for a moment, we do not anticipate any further downtime, once again we are very sorry about the problems this morning.
We are doing a quick reboot of the replacement router, it should be back up in a moment.
Our sheffield services are restored, we are very sorry for the downtime and the inconvenience caused.
We are in the process of configuring a replacement router.
We hope to have all services restored by 10am BST.
All our Sheffield services are unavailable following a reboot of our main router, which failed to come back up, we are on our way to the data centre to resolve the problem.
We are doing some work on our router at 16:00 GMT today and we expect connectivity to all our Sheffield servers to be lost at this time for around one minute.
Sorry for the git.coop downtime this morning, we are upgrading it and giving it additional resources (more RAM and CPUs).
We are updating GitLab at git.coop and it will be unavailable for a little while.
The upgrade for webarch.email has completed.
We are upgrading webarch.email and as a result it will be unavailable for a little while.
We are upgrading Docker across multiple servers and this will result in some services being unavailable for a while.
The problem with the certificate for webarch.email has been fixed.
The upgrade for webarch.email is complete.
We are upgrading webarch.email so it will be un-available for a little while.
We have booted the server and are now bringing up the virtual servers one at a time.
We are at the data centre and checking the BIOS on the sever.
We are going to the data centre to investigate the problem with the server that is down.
One of our front facing servers is currently off-line and this is effecting several of our services, we are looking into the cause of this problem.
We are upgrading Docker across all our servers and this will result in some downtime for a variety of applications, the key one being webarch.email.
We are upgrading Docker across all our servers and this will result in some downtime for a variety of applications, the key one being webarch.email.
We are upgrading GitLab at git.coop so it will be off-line for a little while.
We are rebooting services due to last nights Linux kernel update.
We are upgrading out GitLab server at git.coop so it will be unavailable for a few minutes.
We are about to apply the May updates to webarch.email, it will therefore be unavailable for a few minutes.
We are updating the GitLab server at git.coop and it will therefore be unavailable for a few minutes.
The update to the webarch.email server has been completed.
We are shutting down the webarch.email server to apply some updates, it shouldn't take long.
The webarch.email server is up and running again, sorry for any inconvenience caused.
We are restarting the webarch.email server to disable the Solr search index (without Solr searching emails using SOGo still works but is slower), the server will be unavailable for a little while.
The full re-scan of the email on webarch.email has completed and the search facility in the SOGo interface is now working correctly.
We have started a full rescan of email on webarch.email to try to solve the issues there have been with the SOGo search not returning full sets of results, this will result in the server running with a high load for some time and this could have an impact for users.
The work on webarch.email has completed.
We are adding additional resources to the webarch.email server and it will be down for a little while.
The problem with webarch.email has been resolved.
We are having problems with the web interface of webarch.email following an update, we hope to have this resolved soon, non-web based email sevices are working normally.
We appologise for the issues with our servers in Sheffield today, there has been intermittent slow disk speeds and high iowait times, we haven't yet found the cause of this.
The upgrade to webarch.email has completed, sorry for the downtime.
We are applying the January updates to webarch.email and it will be unavailable for a little while.
webarch.email is up and running again.
We are restarting webarch.email as part of the planned migration of email accounts from mail1.ecohost.coop, it should be up again soon.
We are very sorry to say that a client's email account on mail1.ecohost.coop has been compromised and used to send a massive volume of spam and this has caused the server to be blacklisted, we have secured the account and are now seeking to have the server de-blacklisted.
The disk resizing has completed, webarch2.co.uk is up and running again.
We are shutting down webarch2.co.uk to add additional disk space, this shouldn't take long.
The upgrade to webarch.email has completed, sorry for any inconvenience caused.
We are updating webarch.email and it will be unavailable for a little while.
The update to webarch.email has completed, sorry for the downtime.
We are updating webarch.email to get the latest Mailcow updates, the server will be unavailable for a little while.
The upgrade of webarch.email has been completed.
We are upgrading webarch.email and it will therefore be unavailable for a little while.
Our GitLab server at git.coop is up and running again, for more detail about the disk space issue see this issue. Sorry for any inconvenience caused.
Sorry about the downtime for git.coop, we are currently growing the disk space.
We are upgrading webarch.email and it will be unavailable for a little while
The upgrade of webarch.email has completed, sorry for any inconvenience caused.
We are upgrading webarch.email as as as result it will be unavailable for a little while.
We have just restarted the webarch.email server to address some issues that it was having.
We have requested the removal of webarch.email from as many anti-spam blacklists as we can, however it might be up to a week before the server is fully delisted, we are very sorry for any inconvenience caused by this.
We are sorry about the problems with webarch.email this morning, it was due to an account being compromised and huge volumes of spam being sent.
We have upgraded webarch.email and it is available again.
We are upgrading webarch.email and it will be unavailable for a little while.
The update to webarch.email has been completed and we have also updated the IP address of this status page (see this article for the background) so it soon be available again. We are sorry for any inconvenience caused.
We are about to upgrade webarch.email and it will be unavailable for about ten minutes.
We have completed the upgrade of webarch.email and are sorry for any inconvenience caused.
We are going to upgrade webarch.email and it will be unavailable for a few minutes.
We have completed the maintainance on webarch.email, sorry for any inconvenience caused.
We are updating webarch.email, it will be unavailable for a few minutes.
We believe that webarch.email is fully up and running again, sorry for the downtime.
We are updating and restarting the webarch.email server and it will be unavailable for a short while as a result.
We are sorry that we are having some issues with our mail server at webarch.email, we hope to have this resolved as soon as possible.
We have added additional RAM to webarch.email and updated the server and it is now up and running again, sorry for any inconvenience caused.
We have had some issues with webarch.email server this afternoon and we are going to add some additional RAM to the server and reboot it, it will be unavailable for a little while.
The webarch.email server is up and running again.
We are uggrading and restarting the webarch.email server, it will be unavailable for a little while.
We are about to upgrade webarch.email it will shortly be unavailable for about 5 minutes.
We are rebuilding all our DNS servers, the work on the Sheffield ones, dns0.webarchitects.co.uk and dns1.webarchitects.co.uk has been completed. The European server, dns2.webarch.info has been rebuilt and it is now based in Paris and has a new IP address, 217.70.190.127. Our Icelandic server, dns3.webarch.info, is in the process of being rebuilt and when this is completed it will also have a new IP address, 185.112.146.79.
The work on webarch.email has been completed.
We are doing some maintenance on the webarch.email server, it will be unavailable for a few minutes.
We are restarting webarch.email, it should be back up in a moment.
The webarch.email server has been upgraded and is available again.
We are upgrading the webarch.email server and expect it to be unavailable for a few minutes.
We are seeing very high loads on host3.webarch.net and we have shutdown Apache and MySQL on the server while looking into this issue. host2.webarch.net is running a filesystem check and should be up soon.
We are restarting all the virtual servers on xen2.webarch.net.
We have resolved the issues with the connection to the IPMI on xen2.webarch.net and are once again shutting down all the virtual servers on the server.
We are having issues connecting to the IPMI on xen2.webarch.net, so it hasn't yet been rebooted, so we are restarting the virtual servers on it and will shut them down again when we are ready to reboot the server, sorry for this delay.
We are shutting down all the virtual servers on xen2.webarch.net.
We are restarting all the virtual servers on xen1.webarch.net.
We are rebooting xen1.webarch.net.
We are shutting down all the virtual servers on xen1.webarch.net.
Due to the Meltdown and Spectre exploit critical vulnerabilities we are planning on rebooting all our Sheffield servers during the course of the morning of Wednesday 10th January 2018, we don't anticipate that there will be a great amount of downtime but all services will be affected for a while.
We are expecting to have to reboot all our servers at some point in the next few days, when Debian updates are available for these security issues.
The webarch.email server has been upgraded and appear to be working without issues.
We are going to upgrade webarch.email this afternoon, an email announcing this has been sent to all mailboxes on the server.
Restoration of all the virtual servers in Iceland is now looking hopeful, the lastest from @1984ehf: "24 hours ago we thought that nearly all VPS data was corrupt beyond repair. We have since found a way to restore disks from the VPSs. This means that we will be able to retrieve data from a lot of the VPS disks and we will help every one of our VPS customers to find their data.".
We are pleaase to say that mail.webarch.net is back up again and we believe we have solved the cause of the downtime.
Sorry about the downtime for mail.webarch.net, it should be up again soon.
We are pleased to say that mail.webarch.net is up and running again, sorry for the downtime.
We expect the filesystem check being run on mail.webarch.net to take several hours to complete and the server won't be up before this completes. Sorry for the downtime.
Veryr sorry about the on-going downtime for mail.webarch.net, we hope to have it back up ASAP.
We have had an issue with mail.webarch.net and we are currently restarting the server.
We are plased to say that webarch.email is up now and working normally.
We are having to reboot webarch.email sorry for the downtime, it should be back up shortly.
We believe all our systems are up and running again normally, very sorry for the downtime.
We are afraid that we have a problem with our front facing Xen servers in Sheffield and we are having to reboot them, this will result in some downtime for all our Sheffield based services.
The Sheffield connectivity issue this morning was caused by a problem with a router in London, we have spoken to our data centre and they have resolved the issue.
We are looking into connectivity issues at our Sheffield data centre.
Our Mailman server is now up and running again.
We should have the mailman server up and running very soon. We are afraid it is going to be a little longer for mail.webarch.net and the MKDoc server.
We are copying the mail.webarch.net disk, 90G done out of 275G, in one hour, at this rate we are afraid it is probably going to be down until around 3:30pm.
We are doing some additional work on mail.webarch.net, it will be down for a little while.
mail.webarch.net is back up and we have also fixed the issue with the webserver on webarch1.co.uk. We hope to bring up the Mailman server very soon.
We are afraid it is going to be some time before mail.webarch.net is up and running again, we are copying the disk across the network and have copied 41G out of 275G — it is going to take some time to complete.
mail1.ecohost.coop is now up again.
The file system check on mail1.ecohost.coop is 53% complete and might take another hour or two to complete, then that server will come up again. The server that was running several CentOS servers, cat.webarchitects.co.uk, which had an uptime of 2017 days, had two RAM chips fail after the move, after removing them the server still won't boot, so we moved the disks into another server and used a rescue disk to repair the RAID 0 boot partition on all the disks. The server that the disks from cat are now running in doesn't have enough RAM to run the virtual servers, so we are now copying the partitions over the network to our FreeBDS ZFS file server and when that is completed we will start them up on xen1.webarch.net. This means that it is going to be quite a few hours before these servers are up:
mail.webarch.net Webarchitects mail serveremail-lists.org and several other domainsSorry for the downtime.
The Ecohost mail1.ecohost.coop server is running a file system check and should be up soon.
We believe all our Sheffield services are up and running again apart from mail.webarch.net, our Mailman server and our MKDoc server, we have had some hardware issues with the physical server these virtual servers run on and are trying to solve the problems.
We are starting to shutdown services in our Sheffield data centre for the room move.
We will start shutting down servers in our Sheffield data centre at 10pm BST to move them to a new room, will post updates as the work progresses, we are at the data centre now.
We believe that all services are now running normally, please contact us if you have any issues.
We have some issues with xen1.webarch.net which runs multiple virtual servers and as a result we are in the process of migrating services to xen3.webarch.net, very sorry for any inconvenience caused.
The mail1.ecohost.coop server has been removed from the SORBS blacklist and should soon be removed from the Barracuda blacklist.
We have requested that mail1.ecohost.coop is delisted from the SORBS and Barracuda email server blacklists, sorry for the inconvenience these listings might be causing.
We are aware of issues regarding mail.webarch.net and we are investigating the cause of these problems.
One account on mail.ecohost.coop was being used to send spam, we have stopped the abuse, but there is a possibility that this will have a negative impact on other users of the server if the server is blacklisted.
Our new mailcow server, webarch.email, is now up and running again, very sorry for the downtime.
We are working on restoring webarch.email from a 2am backup from 5th July 2017.
All services, apart from webarch.email are now up and running, we are still working on bringing webarch.email back online. Sorry for the inconvenience.
We are in the process of restarting all the servers on xen1.webarch.net, very sorry for the downtime.
We are having to reboot xen1.webarch.net due to kernel issues across all the virtual servers on this machine, we are afraid this is resulting in some downtime for a large number of services.
We have had some problems with webarch.email and are having to reboot the server, it should be up again in a moment.
The planned maintainance on mail.webarch.net and webarch.email has now been completed and all our mail services are up and running again, sorry for any inconvenience caused.
We are about to conduct some planned maintainance on mail.webarch.net and webarch.email, this will involve an interuption to your email service, but no email should be lost, we will post up update when the work has been completed.
Some of our hosted sites which use Let's Encrypt HTTPS certificates are currently unavailable due to Let's Encrypt OSCP servers being down, see the Let's Encrypt Status Page
Very sorry for the downtime this morning, all systems are now up and running again.
We are now bringing the virtual servers up one at a time and each file system is being checked, it will be a little while before they are all running again, very sorry for this downtime.
We are about to reboot one of our main physical servers in Sheffield which hosts multiple virtual servers so quite a few servers will be unavailable for a while
We are currently having some issues with webarch1.co.uk, we hope it will be available again shortly.
Sorry about the problems with web2.ecohost.coop, the server hosts a Labour MPs site which is under a very high load today.
Very sorry for host3.webarch.net being down for just over an hour this morning, it is now up again.
host1.webarch.net and host3.webarch.net are now up and running again, very sorry for the downtime
We are rebooting host1.webarch.net and host3.webarch.net for essential maintenance.
host1.webarch.net and host3.webarch.net went offline between 1:20am and 1:30am this morning and were not available until 8:30am this morning, we sincererly appologise for any inconvenience and we are investigating the cause of the issue.
We appologise for the downtime that host1.webarch.net and host3.webarch.net have just had, the servers are now rebooting and should be up again in a moment.
We are sorry that we just had a little downtime across several shared hosting servers in Sheffield due to a large load spike, we are looking into the cause.
host3.webarch.net is back up, sorry for any inconvenience caused.
We are expanding the disk on host3.webarch.net, it should be up again soon. Sorry for the downtime.
We have resolved an issue with payments of invoices via PayPal, invoice payments via Paypal are now working correctly, sorry for any inconvenience caused.
We are very sorry to say that we have discovered that our ticketing system was not accepting tickets for 14 days over Christmas and the New Year due to a misconfiguration error. If you tried to contact us during this period you should have recieved a Undelivered Mail Returned to Sender response, please resend your original email as the ticketing system is now up and running again. We appologise for any inconvenience caused by this outage.
We are seeing some connectivity issues with servers and services in Iceland this afternoon.
The fault with our office phone line has now been resolved and our answerphone is available again when nobody is available.
There is currently an issue with our office phone line, The Phone Co-op are investigating the fault, best use email to contact us today.
Sorry so the loss of network connectivity for host3.webarch.net between 13:57 and 14:03, we are investigating the cause of this outage.
The mail1.ecohost.coop server has now been removed from all but one blacklist and for this one "Automatic removal will occur for IPs that are seen to be clean". We have sent an email to users of the server to explain what happened and to appologise for any inconvenience caused.
The mail queue on mail1.ecohost.coop has now cleared but we are sorry to say that the server is now listed on some blacklists due to a compromised account on the server being used to send a large volume of spam — we are now working to have it removed from the blacklists. Sorry for any inconvenience caused by this and please ensure that all the devices you use for email are secure!
Please bear with us as the mail queue on mail1.echost.coop clears — until then there will be some delays in outgoing emails.
The mail1.ecohost.coop server is back up with additional disk space, once again sorry for any inconvenience caused.
Very sorry for the inconvenience but there is going to be a little downtime for mail1.ecohost.coop as we need to add additional disk space to the server.
The mailserver mail.webarch.net has been rebooted with an extra 2G of RAM and the MKDoc server is also back up. Sorry for any inconvenience caused by the short downtime.
We are about to reboot the Webarchitects mailserver mail.webarch.net in order to add additional RAM to it. There will be a short downtime for the MKDoc server as we are removing some RAM from that machine as it is running fewer sites these days.
Sorry to say that there has just been another short outage to services in Iceland, between 14:16:42 and 14:18:33, we will post an update here when we know the cause of the connectivity issues today.
There has just need another short connectivity issue with Iceland based services, the network was down between 12:14 and 12:15 BST, it is back up now.
Network connectivity to servers and services running in Iceland has been restored.
We are experiencing network connectivity issues with our servers in Iceland, we are investigating the cause of this problem.
Barracuda Networks have responded to the delisting request for mail1.ecohost.coop as follows:
"We have removed 81.95.52.108 (Please wait 24-48 hours) from our blocklist for 30 days, at which time it will be re-evaluated."
You can check on the delisting progrees a Debouncer Blacklist Check.
We believe we have discovered and blocked the abuse of services running on mail1.ecohost.coop and we are now requesting that this server is removed from blacklists. Once again we are sorry for any inconvenience caused by this.
One of our email servers, mail1.ecohost.coop is currently listed as having a 'poor' reputation on the Barracuda Reputation System, this is causing some email delivery issues, we are investigating the cause and when we have tracked it down we will request to be delisted. Sorry for any inconvenience caused by this.
This site is being launched today, all services and servers are up and running without issues.
We can be found in #webarch on irc.freenode.net, the Freenode Internet Relay Chat (IRC) network has a web interface at webchat.freenode.net.
Our announcement emails lists have public archives and you can browse these at lists.webarch.co.uk.
If there is a problem with the main Webarchitects site you should be able to view a recent snapshot at the Internet Archive.
This site and the DNS servers forwebarch.info are hosted totally independently from all
other Webarchitects server infrastructure to ensure that any outages elsewhere don't affect this site. The
DNS servers are provided by Gandi and the web hosting by
GitLab.
The code for this site and a script for updating it, can be found at GitLab. This site takes a few minutes to be regenerated after an update is committed to the status.html file, you can check if there are any outstanding updates by looking at the jobs page.
