Partial outage in Agent interface
Incident Report for Dixa

On Friday 25th of August at 11:00 CEST, we had a partial outage that affected the conversation offers, the agent presence and the performance of Dixa with reports of messages not loading, offers not being accepted and general slowness of the platform. The incident was caused due to an exhaustion of capacity on one of our systems tied to the above functions.

The incident was resolved by restarting the services and rolling back a previous update of the agent interface which brought Dixa back to normal.

We thank you for your patience and apologize for the inconvenience this caused you. We have taken action to prevent this from happening again.

Timeline (CEST)

August 25, 2023 10:52 AM: A number of services connected to the agent interface ran out of memory

August 25, 2023 11:00 AM: We noticed a sudden drop in incoming agent connections followed by a rapid recovery.

August 25, 2023 11:18 AM: Reports are coming in to Dixa support team about messages not loading, offers not being accepted and general slowness of the platform.

August 25, 2023 11:26 AM: Incident process is triggered and status page is updated.

August 25, 2023 11:30 AM: Issue identified as one of our services running out of memory without automatically restarting. The services are restarted manually.

August 25, 2023 11:35 AM: Rolled back a product update that relied on that process to an older version.

August 25, 2023 12:00 PM: Service recovered.

Posted Sep 01, 2023 - 09:18 CEST

No further issues have been detected and the system is back to normal. We thank you for your patience and cooperation.

Post mortem about this incident will be posted within 5 business days.
Posted Aug 25, 2023 - 14:09 CEST
The issues have been resolved. We are continuing to see stability across our platform. We will continuing monitoring.
Posted Aug 25, 2023 - 12:12 CEST
We have identified the issue and our team have deployed a fix. We are seeing significant improvements for Messenger, conversation offers and stability of the platform. We will continue to monitor the results.

Next update at 12:20 CEST
Posted Aug 25, 2023 - 11:51 CEST
We are continuing to investigate this issue.
Posted Aug 25, 2023 - 11:36 CEST
We are receiving reports of slowness in the agent interface and reports of Messenger messages being blank. Reports of problems with conversations being offered and the agent status. We are investigating the issue.

Next update at 12:00 CEST
Posted Aug 25, 2023 - 11:35 CEST
We have received reports of instability in the platform. We are investigating the issue. Updates will follow
Posted Aug 25, 2023 - 11:26 CEST
This incident affected: Agent Interface (Agent Interface, Conversation Offers (WSS connection)) and Other Channels (Chat & Dixa Messenger).