Dixa Status - Degraded performance

Degraded performance - telephony

Incident Report for Dixa

Postmortem

At 11:00 AM CEST Saturday, September 21st, we received reports of problems with the telephony channel, indicating customers were unable to make outbound calls, and a subset of incoming calls failing.

We immediately initiated an investigation and found the issue to be caused once again by an upstream provider providing services for validating phone number ownership, used by Dixa for outbound and inbound calls.. These delays caused internal API timeouts which had the effect that numbers became unavailable to use for outbound calls and, to lesser extend, caused inbound calls to take longer to connect than usual or to fail entirely.

At around 12:00 PM CEST, error rates were going down rapidly and the incident was changed to monitoring.

We want to assure you we’ve taken steps to prevent this from happening again. Given this was the third time in as many days our upstream provider had this issue, we expedited implementing a caching mechanism, hoping this will mitigate any short outages higher up the chain. The hotfix was deployed later that Saturday.

Our sincere apologies for the inconvenience this may have caused.

Posted Sep 26, 2024 - 12:09 CEST

Resolved

We haven't seen new reports of telephony problems related to our upstream provider and are confident the problem is resolved for now.

We're still actively working on expediting the implementation of a fix, so we can prevent this issue from causing further problems.

A postmortem will be shared within 5 business days.

Posted Sep 21, 2024 - 12:46 CEST

Update

Telephony should be fully operational again for all Dixa customers. We will continue to actively monitor for now.

If you continue to experience issues with phone calls, please reach out to friends@dixa.com via email, or via the chat inside your Dixa account and let us know.

Posted Sep 21, 2024 - 12:15 CEST

Update

We're seeing improvements across customers and are monitoring the situation actively.

We're also attempting to expedite implementation of a solution that can potentially prevent this issue from occurring again the next day(s) in case our upstream provider has issues again.

Next update in half an hour, or earlier if there is more information to share.

Posted Sep 21, 2024 - 12:00 CEST

Monitoring

We're still working with our upstream provider to resolve the issue.

Meanwhile, we're seeing improvements across customers and are monitoring the situation actively.

We're also attempting to expedite implementation of a solution that can potentially prevent this issue from occurring again the next day(s) in case our upstream provider has issues again.

Next update in half an hour, or earlier if there is more information to share.

Posted Sep 21, 2024 - 11:50 CEST

Identified

We've identified the issue as being caused by an upstream provider. We're in contact with them to resolve the issue as soon as possible.

Next update 12pm CEST or as soon as there is more information to share, whichever comes first.

Posted Sep 21, 2024 - 11:06 CEST

Investigating

We're receiving reports again about failing phone calls. We're investigating.

Next update 11.15 am CEST

Posted Sep 21, 2024 - 11:00 CEST

This incident affected: Telephony & SMS (Inbound, Outbound).