Network Degradation Impacting Cigo Services

📣 Incident Summary – March 18–19, 2025

Service Impact on Cigo Tracker due to Azure Regional Outage

On March 18, 2025, Cigo Tracker experienced intermittent service disruption due to a regional outage within Microsoft Azure’s East US data center region. Below is a summary of the root cause, impact, and the steps being taken to prevent future occurrences.

🕒 What Happened?

Azure’s East US region suffered two separate impact windows:

• March 18, 13:37 to 16:52 UTC
• March 18, 23:20 to March 19, 00:30 UTC

The incident was triggered by a third-party fiber cut during external drilling work, which caused reduced network capacity in one of Azure’s Availability Zones. A tooling failure during Azure's recovery efforts later reintroduced traffic prematurely, leading to congestion and a second round of intermittent connectivity issues.

🔍 Root Cause

1. Fiber Cut: A construction-related accident physically damaged fiber cabling serving the East US datacenter, degrading network capacity.
2. Router Maintenance: A key router in the same zone was already under repair, limiting redundancy.
3. Tooling Error: Azure’s automated recovery system failed to fully isolate damaged infrastructure, inadvertently reintroducing traffic too early.
4. Congestion Spillover: The unexpected traffic load caused congestion to spread beyond AZ03 into neighboring zones.

🎯 Impact on Cigo Tracker

While the Azure issue only affected a subset of inter-zone traffic in East US, this included infrastructure we rely on, resulting in intermittent connectivity issues for some customers during the incident windows. Core services were restored once Azure manually completed isolation and fiber recovery work.

🛠 Resolution Timeline

• 13:37 UTC, Mar 18 – Outage begins due to fiber cut
• 13:55 UTC – Initial mitigation starts; traffic rerouted
• 16:52 UTC – First impact window ends
• 23:20 UTC – Second outage begins due to tooling error during recovery
• 00:30 UTC, Mar 19 – Final mitigation complete
• 06:50 UTC – Full restoration of all infrastructure

✅ What Azure is Doing to Prevent Recurrence

• Fixing tooling failures that allowed reintroduction of unready capacity (by May 2025)
• Accelerating a capacity upgrade for the East US datacenter (by July 2025)
• Architecting better safeguards to prevent impact from spreading across zones (by February 2026)

We apologize for the inconvenience caused. Please rest assured that our team is working closely with Azure and continuing to invest in the resiliency of our platform.

If you have any questions or would like help designing a more resilient setup, feel free to reach out to our support team.

Thank you for your continued trust