On Wednesday, May 15, at 6 AM CEST, Dixa engineers started their planned maintenance by upgrading some backend services. It was expected that existing agent connections would be closed and agents would have to reload the agent interface. This was sadly not communicated on the status page due to a miscommunication between teams.

Around 6.37 AM CEST we received the first report of a Dixa customer having trouble accessing Dixa. Due to the ongoing maintenance, and intermittent problems with connecting, it was assumed the issues were related to the maintenance which was still ongoing.

After an increasing number of reports from customers, the incident process was triggered just after 7 AM CEST.

We soon realized some agents had issues with accessing Dixa, while others did not, and seemingly after a few tries agents were able to get in. We did not see increased errors in our internal monitoring tools, and agent connections kept increasing as the day started in Europe, which is normal.

Around 7.15 AM CEST, we also experienced intermittent issues with internal tooling returning an error we did not expect, and which later pointed us in the right direction as to where to look.

20 Minutes later, around 7.40 AM CEST, Dixa engineers and customer success teams found what was causing the issues and started implementing a fix and 7 minutes later, at 7.47 AM CEST the issue was resolved. We communicated the fix 7.51 AM CEST after internal verification everything was back to normal.

We sincerely apologize for any inconvenience this may has caused and want to assure you we’ve taken action to prevent this issue from surfacing again in the future, and have added additional logging and error notifications to be able to detect issues earlier on. Furthermore, we’re re-evaluating the planned maintenance process to make sure maintenance is always communicated via this page.