SSL Certification Issue Affecting Import Tool and Confirmation Module Notifications

Incident Report for Cigo

Postmortem

Summary

On September 15th, 2025, we experienced service disruptions affecting the Import Tool and Confirmation module notifications. The root cause was a misidentified SSL certificate dependency during the scheduled Azure Database for MySQL maintenance, which included the rotation of the MySQL server’s SSL certificates.

Impact

  • The Import Tool was unavailable for a period of time.
  • Confirmation module notifications (via SMS and email) were disrupted.
  • Some other automated functions experienced degraded performance.

Root Cause

While we were aware of the scheduled SSL certificate rotation as part of Azure’s maintenance process, our evaluation process incorrectly identified which certificates were being used by certain subsystems. As a result, several services lost connectivity when the certificates were rotated.

Mitigation and Resolution

  • Our engineering team corrected the certificate configuration for the affected subsystems.
  • The Import Tool and Confirmation module notifications were restored to full functionality.
  • Safeguards are being added to ensure that future SSL rotations do not cause similar disruptions.

Contributing Factors

At the onset of the incident, we initially believed the outages were linked to updates and fixes deployed on Thursday and Friday of the previous week. This led to rollbacks that did not address the issue. Although unrelated to the outage, some of those updates remain rolled back and will be re-published in the coming days.

Next Steps

  • We are reworking our dependency system analysis to more accurately map SSL and package dependencies across all services.
  • We are improving our monitoring and review processes for certificate rotations to ensure full coverage across all subsystems.
  • We will re-publish rolled-back updates after additional validation.

Conclusion

We sincerely apologize for the operational problems this incident caused. We take these issues seriously and are committed to ensuring that similar outages do not reoccur in the future.

Posted Sep 15, 2025 - 16:28 EDT

Resolved

The SSL certification issue that impacted several internal services, including the Import Tool and the Confirmation module notifications, has been fully resolved. All affected services are now fully operational.

We have implemented safeguards to prevent similar issues during future SSL certification rotations.

Thank you for your patience and understanding throughout this incident.
Posted Sep 15, 2025 - 16:24 EDT

Update

We have resolved the issues affecting the Import Tool and the Confirmation module notifications. Both are now fully operational.

Our team is currently working on finalizing the recovery of a few remaining automated functions that may still be degraded.

We will provide another update once all affected services are fully restored.
Posted Sep 15, 2025 - 14:52 EDT

Identified

We recently experienced unexpected issues with SSL certification that disrupted database connections for several of our internal services.

The Import Tool was initially affected but has since been fully restored. During our investigation, we also identified that the issue impacted the notifications subsystem for the Confirmation module, which relies on SMS and email sending.

While the Import Tool is now operating normally, we are still working to restore full connectivity for Confirmation module notifications, along with a few other automated functions that may currently be degraded.

Our engineering team has identified the root cause and is implementing safeguards to prevent similar issues during future SSL certification rotations.

We sincerely apologize for the disruption and will continue to provide updates until all affected services are fully restored.
Posted Sep 15, 2025 - 14:12 EDT
This incident affected: Web applications (Dispatch Web Platform) and Services (Outbound Email Service, Outbound SMS Service).