Friday Jun 27, 2025

Escalation notifications sent out of order for some policies

  Operational

Escalation notifications sent out of order for some policies
Postmortem

On Friday, after resolving the issue where escalation steps fired out of order, our team sat down to conduct a thorough review. We documented exactly what went wrong, identified the root cause, and drafted an internal post-incident analysis.


In the spirit of honesty and transparency, we’re now sharing it publicly so you can see every step we took to restore normal service.


We apologize for any confusion this may have caused and thank you for your ongoing patience and support. Please consider reading the full post-incident analysis to learn more about the incident and our improvements:


Read our post-incident analysis →

Jun 30, 2025 - 06:44AM (UTC)
Resolved

We’ve deployed a fix that resolves this incident. The correct order of escalation notifications has been restored. All steps now execute in the configured sequence including the ones that are currently triggering.


Thanks so much for your support and special thanks to Marko, Jouko, Tarif, and team for first bringing this to our attention.

Jun 27, 2025 - 11:12AM (UTC)
Update

We have identified the root cause (it's timestamps) and pushed a fix.


Currently monitoring the incident.

Jun 27, 2025 - 10:30AM (UTC)
Identified

We’ve received reports that, in certain escalation policies, notification steps are not following the configured sequence. Some steps are delivered mostly earlier than usual.


Our team has confirmed this behavior and is actively investigating to pinpoint the root cause and implement a fix.


Updates coming your way as we make progress.

Jun 27, 2025 - 09:31AM (UTC)

Reported on

Fri, Jun 27, 2025 9:31 AM (UTC)

Resolved at

Fri, Jun 27, 2025 11:12 AM (UTC)