Degraded performance on telephony
Incident Report for Dixa
Postmortem

On Wednesday the 16th of February, 2022, around 9.10 am CET we received notification from some customers that they were experiencing issues with callbacks. A few minutes later, reports came in about problems with accepting and terminating calls.

By 9.20am CET, we realised the incident was related to a fix we made after Monday’s incident (which in itself was an attempt to fix a bug we discovered the week before), and we rolled back the change made the previous day.

By 9.30am CET, the changes were rolled back and our internal monitoring saw a decline in platform errors. A few minutes later we also received confirmation from some of the affected customers.

The issue only manifests itself when there’s significant load on the telephony infrastructure and is hard to replicate in development environments. Because of this, regretfully, the problem recurred, despite extensive testing. We sincerely apologise for the inconvenience caused.

We’re working on a permanent solution to fix the root cause of the issue which manifested both on Monday and today and will implement it as soon as possible.

Posted Feb 17, 2022 - 09:14 CET

Resolved
This incident has been resolved.
Posted Feb 16, 2022 - 09:42 CET
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 16, 2022 - 09:34 CET
Identified
The issue has been identified and a fix is being implemented.
Posted Feb 16, 2022 - 09:29 CET
Investigating
We are experiencing degraded performance on telephony. We're currently investigating the issue.
Posted Feb 16, 2022 - 09:24 CET
This incident affected: Telephony & SMS (Inbound, Outbound).