On Wednesday the 16th of February, 2022, around 9.10 am CET we received notification from some customers that they were experiencing issues with callbacks. A few minutes later, reports came in about problems with accepting and terminating calls.
By 9.20am CET, we realised the incident was related to a fix we made after Monday’s incident (which in itself was an attempt to fix a bug we discovered the week before), and we rolled back the change made the previous day.
By 9.30am CET, the changes were rolled back and our internal monitoring saw a decline in platform errors. A few minutes later we also received confirmation from some of the affected customers.
The issue only manifests itself when there’s significant load on the telephony infrastructure and is hard to replicate in development environments. Because of this, regretfully, the problem recurred, despite extensive testing. We sincerely apologise for the inconvenience caused.
We’re working on a permanent solution to fix the root cause of the issue which manifested both on Monday and today and will implement it as soon as possible.