Degraded performance on telephony

Incident Report for Dixa

Postmortem

At 12:32 CEST Friday 20th of September, we received reports of problems with the telephony channel, indicating customers were unable to make outbound calls, and a subset of incoming calls failing.

We immediately initiated an investigation and found the issue to be caused once again by an upstream provider providing services for validating phone number ownership, used by Dixa for outbound and inbound calls.. These delays caused internal API timeouts which had the effect that numbers became unavailable to use for outbound calls and, in some cases, caused inbound calls to take longer to connect than usual or to fail entirely.

At around 14:20 CEST, error rates were going down rapidly and the incident was changed to monitoring.

Given this was the second time in as many days our upstream provider had this issue, we discovered a potential method to mitigate these problems, and we planned on implementing these as soon as possible

We apologize for the inconvenience this has caused.

Posted Sep 26, 2024 - 12:02 CEST

Resolved

Problems with telephony have been resolved.

Additionally, we've identified a measure we can take to potentially prevent this issue from occurring again in the future, and we will implement this as soon as possible.

If you continue to encounter issues, please reach out to Dixa Support by sending an email to friends@dixa.com, or via live chat inside Dixa's agent interface.

We sincerely apologize for the inconvenience this incident has caused. Postmortem for this incident will be posted within 5 business days.

Posted Sep 20, 2024 - 14:39 CEST

Monitoring

Our upstream provider has implemented a fix, and we see stability improvement on the Dixa side. We will continue to monitor telephony services.

Next update at 2.45pm CEST.

Posted Sep 20, 2024 - 14:20 CEST

Update

We continue to work with our provider, but there isn't any update to share at this time.

We'll update you as soon as we know more, or at 2.45pm, whichever comes first.

Posted Sep 20, 2024 - 14:16 CEST

Update

We continue to work with our provider, but there isn't any update to share at this time.

We'll update you as soon as we know more, or at 2.15pm, whichever comes first.

Posted Sep 20, 2024 - 13:46 CEST

Update

We're still working on resolving this issue with our upstream provider.

Our apologies for the inconvenience.

Next update 1.45pm CEST, or as soon as we have more information to share.

Posted Sep 20, 2024 - 13:30 CEST

Update

The issues originate from an upstream provider providing telephony related services to Dixa.

We're currently in contact with said upstream provider to resolve these issues as quickly as possible.

Next update 1.30pm CEST, or sooner when there is more information to share.

Posted Sep 20, 2024 - 13:00 CEST

Identified

The issue has been identified as potentially being caused by an upstream provider. We're currently working on a solution and we'll update you when we know more.

Next update 1pm CEST, or earlier if there is more information to share.

Posted Sep 20, 2024 - 12:45 CEST

Investigating

We've received reports that there may be problems with in- and outbound calls similar to yesterday.

We're investigating the matter.

Next update 12.45 pm CEST

Posted Sep 20, 2024 - 12:32 CEST

This incident affected: Telephony & SMS (Inbound, Outbound).