At 11:00 AM CEST Saturday, September 21st, we received reports of problems with the telephony channel, indicating customers were unable to make outbound calls, and a subset of incoming calls failing.
We immediately initiated an investigation and found the issue to be caused once again by an upstream provider providing services for validating phone number ownership, used by Dixa for outbound and inbound calls.. These delays caused internal API timeouts which had the effect that numbers became unavailable to use for outbound calls and, to lesser extend, caused inbound calls to take longer to connect than usual or to fail entirely.
At around 12:00 PM CEST, error rates were going down rapidly and the incident was changed to monitoring.
We want to assure you we’ve taken steps to prevent this from happening again. Given this was the third time in as many days our upstream provider had this issue, we expedited implementing a caching mechanism, hoping this will mitigate any short outages higher up the chain. The hotfix was deployed later that Saturday.
Our sincere apologies for the inconvenience this may have caused.