Our Engineering teams have validated the changes and at this point the incident has been resolved.
Posted Jul 06, 2022 - 22:28 EDT
Update
On Wednesday, July 6, 2022, between 9:00 PM ET - 12:00 AM ET, our Engineering teams will be performing another round of maintenance in our US-MIDWEST-2 (DC3) region. This maintenance is being performed to exercise prior changes made and validate that we've fully resolved the incident.
Posted Jul 06, 2022 - 14:22 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 06, 2022 - 05:09 EDT
Update
During the maintenance, our teams have identified further issues which will sadly extend the duration of the maintenance. We are sorry for the inconvenience and appreciate your continued patience as we work to resolve this.
Posted Jul 06, 2022 - 00:33 EDT
Update
On Tuesday, July 5th, 2022, between 9:00 PM ET - 12:00 AM ET, our Engineering teams will be performing another round of maintenance in our US-MIDWEST-2 (DC3) region. A service interruption of up to 20 minutes is expected during the window as Engineers perform the maintenance. Engineering teams will be engaged throughout window, and will treat any impact to service with the highest priority. We appreciate your continued patience as we work to fully resolve the stability issues in this region.
Posted Jul 05, 2022 - 16:57 EDT
Update
While our Engineering team was able to implement changes to the underlying software infrastructure during the maintenance window, they have confirmed that the underlying root cause is still not durably addressed As the steps to replicate the issue have been isolated and measures have been take to prevent these circumstances from emerging, we can confirm that apart from the scheduled maintenance window that was implemented in order to make further adjustments, there have been no other systemic interruptions of service. We will continue to monitor and provide updates as further progress is made in this matter.
For questions, please contact the support team via support@nexcess.net, client portal ticket, live chat, or toll-free at 866-639-2377 or +1.517.322.0434 (international).
Posted Jul 03, 2022 - 08:38 EDT
Update
On Thursday, June 30, 2022, between 9:00 PM - 12:00 AM ET, our Network Engineers will be performing another round of maintenance in US-MIDWEST-2 (DC3). While we hope to avoid further disruption to services, there is potential for up to 20 mins of downtime as a result of the maintenance. This is our highest priority and we appreciate your continued patience as we work to fully resolve the performance issues we have recently experienced.
Posted Jun 30, 2022 - 11:15 EDT
Update
On Tuesday, June 28, 2022, between 9:00 PM - 12:00 AM ET, our Network Engineers will be performing emergency maintenance in US-MIDWEST-2 (DC3) to improve recent issues with stability. We anticipate a brief disruption of services throughout the maintenance window as rolling reboots are initiated on networking devices.
For questions, please contact the support team via support@nexcess.net, client portal ticket, live chat, or toll-free at 866-639-2377 or +1.517.322.0434 (international).
Posted Jun 28, 2022 - 12:17 EDT
Update
The work of our Engineering team continues to make progress towards implementing a durable solution to the identified sources of the internal communications issues within our US-MIDWEST-2 OpenStack environment in collaboration with engineers from our networking equipment vendor. This remains our team's highest priority and there have been no further disruptions related to this event since June 21st.
We are continuing to monitor performance closely and will post additional updates as soon as new information becomes available.
Posted Jun 27, 2022 - 15:08 EDT
Update
Our Engineering team continues to make progress in implementing a durable solution to the identified sources of the internal communications issues within our US-MIDWEST-2 OpenStack environment in collaboration with engineers from our networking equipment vendor. This remains our team's highest priority and there have been no further disruptions related to this event since June 21st.
We are continuing to monitor performance closely and will post additional updates as soon as new information becomes available.
Posted Jun 25, 2022 - 20:58 EDT
Update
Our Engineering team is still working on the identified sources of the internal communications issues within our US-MIDWEST-2 OpenStack environment in collaboration with engineers from our networking equipment vendor. This remains our team's highest priority and there have been no further disruptions related to this event since June 21st.
We are continuing to monitor performance closely and will post additional updates as soon as new information becomes available.
Posted Jun 24, 2022 - 21:05 EDT
Update
Our Engineering team has continued to work through the identified the sources of the internal communications issues within our US-MIDWEST-2 OpenStack environment in collaboration with engineers from our networking equipment vendor. This is our team's highest priority and there have been no further disruptions related to this event since June 21st.
We are continuing to monitor performance closely and will post additional updates as soon as new information becomes available.
Posted Jun 23, 2022 - 19:04 EDT
Update
Our Engineering team has identified the sources of the internal communications issues within our US-MIDWEST-2 OpenStack environment and continues to work through them systematically to eliminate potential problems in the future.
Between 9:00-10:00 PM ET on June 21, 2022, our engineers performed emergency maintenance to correct one of the outstanding issues and contributing factors impacting the environment. There was no downtime associated with the maintenance.
We anticipate no further unscheduled downtime. We will provide additional information and full details once the issue is fully resolved. In the meantime, we appreciate your continued patience while we work as quickly as possible to resolve the issue.
Posted Jun 22, 2022 - 07:35 EDT
Update
All services have been restored at this time and additional investigation is underway.
Posted Jun 21, 2022 - 14:58 EDT
Identified
Networking confirms that we have experienced a disruption in US-MIDWEST-2 and the team is actively working to bring all services back online at this time.
Posted Jun 21, 2022 - 14:46 EDT
Update
Our Engineering team continues to work methodically through the DC to ensure all contributing factors have been resolved. There have been no further disruptions related to this event since June 15, 2022. However, we continue to monitor performance closely and will post additional updates as new information becomes available.
Posted Jun 21, 2022 - 09:07 EDT
Update
We continue to monitor performance closely as our Engineering team progresses towards full resolution. Additional updates will be provided as new information becomes available.
Posted Jun 20, 2022 - 09:26 EDT
Monitoring
Active monitoring is still in place and our Engineering teams continue to make progress towards a full and durable resolution. Services remain stable at this time. Additional updates will be provided as new information becomes available.
Posted Jun 19, 2022 - 04:18 EDT
Update
Services have been stable again throughout the day on Friday June 17th. Engineering teams continued to make progress in identifying the root cause of the issue. Active monitoring remains in place as well. We appreciate your patience as our teams work to ensure this issue is fully resolved. We will provide additional updates as new information becomes available.
Posted Jun 17, 2022 - 19:19 EDT
Update
Performance remained stable throughout the night. Our Engineering team continues to monitor closely as they methodically move the investigation forward to ensure this issue is fully resolved.
Additional updates will be provided as new information becomes available.
Posted Jun 17, 2022 - 09:11 EDT
Update
Services have been stable throughout the day and Engineering teams have made progress isolating the issue. Our teams are still working to ensure this issue is fully resolved and active monitoring is still in place. We will provide additional updates as new information becomes available.
Posted Jun 16, 2022 - 19:13 EDT
Update
We are continuing to investigate this issue.
Posted Jun 16, 2022 - 19:09 EDT
Update
Performance has been stable throughout the night and our engineers continue to work as quickly as possible to isolate and fully resolve the root cause of the issue. Additional updates will be provided as new information becomes available.
Posted Jun 16, 2022 - 10:06 EDT
Update
Our Engineering team is continuing with their investigation of this issue. Progress has been made with improving stability, but we are still monitoring the situation, in order to completely mitigate this interruption of service.
If you have any questions, please contact a member of our support team via ticket or by phone at +1-866-639-2377 (US) or +1-517-322-0434 (international).
Posted Jun 16, 2022 - 05:13 EDT
Update
We are continuing to investigate this issue but have seen better stability as we've worked on possible causes. Our engineering team is still currently investigating the root cause behind the issue to fully stabilize the environment. This investigation continues to be high priority and our teams are actively monitoring and available to act quickly, if needed. We will update this status page again as further information becomes available.
If you have any questions, please contact a member of our support team via ticket or by phone at +1-866-639-2377 (US) or +1-517-322-0434 (international).
Posted Jun 15, 2022 - 19:52 EDT
Investigating
Our Engineers are still working to resolve an interruption in service affecting a subset of cloudhost services in our us-midwest-2 data center. We'll provide further updates as soon as they become available. This is a continuation of https://status.nexcess.net/incidents/7nfrbnv9vjx1 which was inadvertently resolved.
If you have any questions, please contact a member of our support team via ticket or by phone at +1-866-639-2377 (US) or +1-517-322-0434 (international).
Posted Jun 15, 2022 - 14:38 EDT
This incident affected: Nexcess Cloud (us-midwest-2).