The problem on Microsoft Azure's end seems to have been mitigated. Our workloads have returned to normal.
Posted Sep 07, 2022 - 16:32 EDT
Update
Microsoft Azure's team hasn't reported any additional updates, but based on our monitoring, traffic seems to have returned to normal in the last 30 minutes (3:30 PM EDT).
We are standing by for a full resolution confirmation to be confirmed by the Azure team.
Thank you for your patience and understanding.
Posted Sep 07, 2022 - 16:01 EDT
Monitoring
Microsoft Azure's team has confirmed that they are working on a full system recovery following a large spike in traffic that disrupted their network infrastructure:
"We are recovering intermittent connectivity issues. Traffic managed by Azure Front Door service is being recovered by systematically going through the regions where we are observing resource impact and enforcing traffic management on the same. Once the recovery process is completed, the service should be able to resume handling traffic normally."
We are monitoring the health of our Front Door and are seeing a steady recovery.
Posted Sep 07, 2022 - 15:07 EDT
Update
The Microsoft Azure team has confirmed that they identified the potential cause as "a spike in traffic".
They have further clarified the following: "While we are not currently observing any traffic spikes currently, we are working on remediating the residual impact. We are recovering a number of nodes that are showing intermittent connectivity issues. For customers who are experiencing connectivity issues, retries are likely to be successful. Most customers should be seeing recovery at this stage."
From our monitoring systems, our DNS health seems to be recovering, but there is still a 5% fluctuation that may impact some customers in some regions.
Posted Sep 07, 2022 - 14:23 EDT
Update
New update from Azure. They are still investigating the issue and we remain on hold:
Starting at 16:10 (UTC) on 07 Sep 2022, customers using Azure Front Door could be experiencing connectivity issues. This could also be impacting customers’ ability to access the Azure Management Portal. We are investigating a spike in traffic as a potential cause. Retries are likely to be successful.
Posted Sep 07, 2022 - 13:22 EDT
Identified
Our platform is currently experiencing a partial outage. A portion of our users are fully impacted and are unable to access the platform, for other response times are slow, and the rest of our users are unaffected.
The report from the Microsoft Azure team is as follows (as of 2022-09-07 1:05 PM EDT):
Connectivity issues We are aware of connectivity issues to the Azure Portal and customers using Azure Front Door. We will provide more information as it is known. This message was last updated at 17:00 UTC on 07 September 2022
Posted Sep 07, 2022 - 13:07 EDT
This incident affected: Dispatch Web Platform, Customer Tracker, Public API, Operator API, iOS, Android, Routing and Itinerary Optimization, Maps, Notifications, Outbound Email Service, and Outbound SMS Service.