News & Information
Affecting Other - West Midlands Data Centre 2
Reference: WM-DC2-2022-11-03
Maintenance Type: Essential
Current Status: Scheduled
Priority: Urgent
Affected Data Centres: West Midlands DC2
Affected Functions/Services: DC2 Core Network
Date: 2022-11-04 to 2022-11-04
Time of Commencement: 19:00
Time of Completion: 19:30
Downtime Window: 15 minutes (minimum expected service interruption)
Expected Future Outage: N/A
Description of works: We are going to be replacing one of our core network switches located in DC2. This is to implement additional levels of resilience and redundancy for our network in this particular data centre. This will require a complete device power off to occur whilst we conduct these upgrades. This will result in a small amount of service-impacting disruption for a short period of time during the above dates and times, for any customers directly connected to this device.
Affecting Other - Data Centre
Wednesday 20th July
We are aware of an issue with power at one of our key data centres in the Midlands. FyfeWeb Data Centre Engineers are now on-site and are investigating the issue.
19:45: A large number of systems and services have now come back online. We have declared a P1 incident and have now invoked our Business Continuity Plan and are taking all necessary actions to ensure we return back to 100% operations as quickly as possible. All company staff and stakeholders are working hard towards a swift service restoration. Operations team are working through our continuity procedures and measures as we speak. We'll provide an update as and when we can.
20:20: We are continuing to work on service restoration and incident resolution. More services and systems are returning to an operable state and the majority of systems are now back online. Operations team continue to work through continuity procedures and measures. We'll provide another update as and when we can.
20:35: All systems and services now appear to be back online and operating at full capacity. This incident has now been downgraded to a P2 Incident, however services are still classed as "at-risk". We, alongside our data centre partners, will be investigating the root cause of this issue and will be issuing an RFO when we can. We sincerely apologise for the inconvenience and disruption caused. Operations team are continuing to action contingency and continuity procedures and measures. We are continuing to monitor all systems and services in the affected data centre and will provide updates should anything change.
23:10: We are conducting some emergency maintenance on our Cloud04 Cluster following today's outage. We are working as quick as we can to get this completed and get the cluster back online. Once completed, Customers should see their instances come back online.
23:28: This has now been completed, and all servers hosted on Cloud04 are back online.
Thursday 21st July
13:47: All services have remained stable overnight, so we're closing this incident. A full RFO (Reason For Outage) report will be published in the coming days. If you have any questions, please don't hesitate to contact us.
Affecting Other - FyfeWeb Core Network
18:07 | We have detected issues with one of our upstream providers. At the moment, we have lost connectivity with them. We believe that this is something their end. We will update you when we can.
18:11 | We have worked with our upstream provider and they are aware of an issue. We should now start seeing services become accessible once again.
18:20 | We are continuing to work on the issue. Service connectivity may continue to be interrupted at times. Connectivity is, however, more stable.
18:33 | We have disabled our connectivity with the upstream provider in question and services seem to be returning to normal.
19:07 | Whilst working with our upstream provider, it would now appear that they resolved the issues. We are continuing to montior this incident and will provide any other updates should they arise.
21:48 | We have observed stabilty on our DDoS mitigation platform and have re-enabled connectivity with the third party upstream. We're still monitoring this, and will be over the next 24 hours.
11:40, 01/06/2022 | We're now closing this incident.
Affecting System - Hosted Private Cloud
During the listed date and time periods, we will be conducting essential maintenance on hosts situtated in our Midlands Data Centre. This will result in affected services being unavailable for an approximate period of thirty (30) minutes.
Timeframe:
Begins - 10:30 AM
Ends - 11:00 AM
Date: Saturday 17th April 2021
Services affected by this maintenance window include:
- Private Cloud Infrastructure
- Hosted Firewall and Network Address Translation Services
- Mail Protection and Filtering Services
Please note that we advise affected customers to ensure their data is saved prior to the scheduled start time to prevent data loss, customers on our public cloud (VPS) clusters are unaffected by this maintenance and emails sent to our mail filtering services during this period will not be lost - only delayed.
Thank you for your patience during this time.
Affecting System - Virtual Private Cloud Servers
Update at 14:14: This work has been completed.
Update at 13:50: We have started this work early.
We have identified a small issue relating to storage array cache on the physical RAID Controller used within one of our Virtual Private Cloud nodes.
We have tried to remediate the issue tonight (04/12/2020) but the server requires a small hardware replacement, since the following error was parsed:
"Please replace Cache Module Super-Cap. Caching will be enabled once Super-Cap has been replaced and charged."
This work will be completed within a timely manner on the Friday 11th December at 2PM. The server will need powering off, so please ensure all work has been saved to prevent data loss.
Affecting System - FyfeWeb DNS Resolver 01
Update at 10:19AM: This work has been completed.
During the stated time period, we will be performing maintenance on DNS01, one of our data centre DNS resolvers.
During this time, clients will fallback onto DNS02.
Thank you for your patience.
Affecting System - FyfeWeb Core Network
MAINTENANCE COMPLETED
Location: FyfeWeb Telehouse East Network PoP
Window Commences: 13-11-2020 23:00:00 GMT
Window Concludes: 13-11-2020 23:30:00 GMT
---------------------
Dear Customer,
FyfeWeb will be performing essential core network maintenance at our Telehouse East, London PoP on the date and time listed above. This maintenance is essential to ensure reliable and continuous service.
Following the service impacting network event on 10-11-2020 between 14:08 and 14:12 GMT, we have identified the root cause to be a vendor specific bug.
During the above maintenance window, we will be applying a Juniper recommended fix/work around to our iBGP configuration to ensure this doesn't re-occur. This will cause a single session reset (BGP flap) to all iBGP peers configured on the router in Telehouse East.
We expect a service impacting outage / intermittent connectivity during this window for 3 to 4 minutes whilst BGP sessions flap, re-establish and routes re-converge in the core network.
If you have any queries regarding this maintenance please get in touch.
Kind Regards,
FyfeWeb Network Operations
Affecting Other - FyfeWeb Core Network
[Update 14:22]
Service should have been restored within a few minutes of the brief blip. Initial investigations show that BGP flapped between two of our core internet routers.
Sometimes, BGP Flapping can occur when you have an unstable peer. This occurs when the BGP route disappears and reappears in the routing table. Investigations are continuing in order to determine the root cause.
All services should be restored.
------------------
We're investigating a network flap between two Juniper core devices on our network. The issue was related to BGP connectivity between two core routers.
This issue should now be resolved, and we will be keeping this network status open for about 1 hour.
Thank you
FyfeWeb Network Operations
Affecting System - fyfeweb.com
Dear Client,
We will be performing some necessary maintenance and updates to our company website on Saturday 22nd August, starting at 11 PM, and finishing at 00:30 on the 23nd.
During this time, the website will be periodically inaccessible. No customer-facing hosting infrastructure, or the FyfeWeb Portal (https://portal.fyfeweb.com) will be affected.
Kind regards,
Jaden Fyfe
Managing Director
FyfeWeb Hosting
Affecting Other - FyfeWeb Network Infrastructure
Dear Client,
We've extended our maintenance period to 16:00 BST. Please keep in mind that you should now see no connectivity issues on services, we're just making some final touches.
We will be performing essential network and server hardware maintenance during the time periods of 10:00 AM and 13:00 PM (British Standard Time) on Saturday 18th July. This is so we can introduce necessary extra capacity to our Core Network connectivity between our Wolverhampton and London Data Centres, with the additional capacity exceeding 10Gbps+. There may be brief dropouts or packet loss experienced on the following select services for up to 10 minutes at a time. This only applies to services that are not hosted in our Newcastle or Manchester data centres:
We’re doing this work to bring some amazing new expansion to your services. Please do not hesitate to get in touch with us, if you have any questions.
For more information on what is happening when we’re performing this work, please visit https://status.fyfeweb.com
Kind regards
FyfeWeb Support
Affecting Server - fw-plesk01
We're currently performing Plesk Server maintenance. This includes moving the Virtual Infrastructure to some new equipment within our Wolverhampton Data Centre.
Apologies for any issues this may cause you.
We're currently performing cPanel Server maintenance. This includes moving the Virtual Infrastructure to some new equipment within our Wolverhampton Data Centre.
Apologies for any issues this may cause you.
Affecting System - FyfeWeb Core Network (WLV)
During the maintenance window below we will be performing some routing changes to a section of our core network in our Wolverhampton, UK data centre.
A number of very brief outages of approximately 3 minutes may be observed.
Start Date: 10/04/2020
Start Time: 10:30pm
Start Date: 10/04/2020
End Time: 11:00pm
We're doing this to bring new improvements to sections of our UK core network. Thanks for your understanding during this time.
Update: 22:49 - This was completed without issue. An "all clear" has been issued after the routing change was made on our WLV Core Router.
We are aware of on-going issues with our cPanel Services. The team are online and investigating.
Update: We have identified the root cause and are actively working on a remedy.
Update: We are seeing service returning to normal. We will continue to monitor the situation over the coming hours
Resolved: This incident is now resolved.
Affecting Server - fw-plesk01
We're currently updating our firewall services on FW-PLESK01.
This may cause issues with the service.
Thank you for your patience.
Affecting System - Corporate Website
We're currently working on our main website. During this time, it may be inaccessible.
In the meantime, you can still open a support ticket via our Client Area - FyfeWeb Advanced.
Affecting System - Shared Web Hosting Servers
Hello,
We're just letting you know that we've sucessfully completed major software updates on two of our main Shared Web Hosting Servers.
This would have caused a few minutes downtime for hosted websites. Please ignore any uptime monitors alerting you to issues during this period.
Kind regards,
FyfeWeb IT Support
We're currently updating FW-CP01.
Apologies to any inconvenience this may cause you.
This may take up to 30 minutes.
We're currently investigating and updating our cPanel Hosting Server FW-CP01.
We apologise for any inconvenience caused.
We're installing updates on some of our Servers. The following services may be unavailable:
We'll have this maintenance completed as soon as possible.
Quick update regarding new cPanel Server:
We've finally sorted our cPanel Web Hosting Server with our new fibre internet connection! Thanks to Joshua Cutting (Director of Data Resilience) for coming to our DataCentre today to help restore all the encrypted backups to the new server, and to set up his own server in our racks. Also, thanks to Connor McDonald (Director of Hosting Services) for sorting out all client DNS records - and ensuring that they pointed to the new IP.
Info for clients:
New IP: 159.242.64.37
New hostname: FW-CP01.fyfeweb.com
Other critical information:
If your website appears to still be offline, the nameservers still may be propagating, which may take up to two days. We haven't yet seen any websites with this issue, but if it does occur, use our new cPanel NameServers:
ns1.fyfeweb.com
ns2.fyfeweb.com
Please let us know if you have any more issues after the migration. It's clear that everything on the cPanel Hosting is now a lot faster!
Affecting System - FyfeWeb Microsoft Exchange Email and FyfeWeb Cloud
We're currently migrating users to our new Microsoft Exchange Server, which will provide a better experience to clients.
This may take a while - and we apologise for any issues this may cause you.
On this date and time, we'll be moving all our hardware to a new server rack. We're doing this at 1 AM (British time) to avoid issues of downtime during the day. This will bring the following services offline for up to 1 hour and 30 minutes:
Machines such as VPS Servers and Remote Desktop Services will be shut down to avoid data loss.
We thank you for your co-operation during this planned event!
Affecting System - Software Hosting Services
Please check https://status.discordapp.com/ to keep up to date with the issue.
If your bot continues to not work after the issue is fixed, feel free to submit a ticket with support and we will double check for you.
Affecting System - Core Networking Infrastructure
We will be upgrading our core network connection at the given time. This could affect the following services:
This will not affect any websites hosted on FyfeWeb Advanced via our Plesk Onyx panel.
Downtime of these services could occur, but it's unlikely. We're sorry for any inconvenience this may cause you.
Affecting Server - fw-plesk01
We have identified why sending emails to some external email accounts was having issues with classifying mail as spam. We are still mitigating the issue with the final blacklist and the effect of this issue is now minimal, and many popular email services such as gmail and outlook aren't highly affected anymore.
However, it seems that the situation is recovering and emails aren't being classified as "spam", and emails sent to Gmail and Outlook are returning back to normal
We again apologise for any inconveniences caused during this time.
Affecting Server - fw-plesk01
We have reset everyones Plesk Hosting Control Panel password, so please ask us what your new one is if you didn't get the email.
Please contact our IT support team immediately if you encounter any problems using our new Linux based server enviroment.
We have finally installed our new amazing piece of hardware. Everything seems to be online at the server end now, since we had to reboot most of our infrastructure.
You may have small issues connecting to the network at times. Don't worry, this is becuase things may take up to 12 hours to patch through to the new system.
We apologise for any inconvenience this may cause you.
We have recently updated our SSL certificate for our network services at FyfeWeb Advanced. If you face errors regarding SSL errors on your mail client software, please just "continue" or "accept".
Alternatively, if you still are facing issues, please re-setup your email account on your mail client software by following the instructions in our knowledgebase.
We are updating and re-licencing our Plesk web hosting control panel servers. This may cause interruption with our hosting control panel.
Websites will not go offline.
We apologise for any inconvenience this may cause for you.
