Partial outage on APIs

Incident Report for Dixa

Postmortem

Between 06:54 to 08:55 CET on 24/03/2022, Dixa experienced a partial outage of the Integrations and Dixa APIs.

The APIs sent an unusual amount of error codes. This triggered internal alerts. And was identified as an issue

Our incident manager involved the relevant engineering teams and management at 07:48, and an active investigation began.

After further investigation, it was concluded that one of the components handling the API. Was not updated correctly. And a fix was deployed.

We continued to monitor until full recovery and system stability were confirmed.

Timeline (all times are in CET):

06:59 First alert of increased error messages.
07:48 Incident management process triggered.
08:25 Issue was identified and solution to the issues was started
08:55 Solution deployed and confirmed working.
09:01 Monitoring phase. The system was operating normally.
09:32 incident resolved and closed

Posted Apr 04, 2022 - 17:31 CEST

Resolved

All known issues to this incident have been resolved. We thank you for your patience and cooperation.

Post mortem about this incident will be posted within 5 business days.

Posted Mar 24, 2022 - 09:32 CET

Monitoring

We are happy to inform you that our teams have deployed a fix for the issue. We are seeing significant improvements in API availability. We will continue to monitor the results.

Next update at 09:30

Posted Mar 24, 2022 - 09:01 CET

Identified

Our teams has now identified the issue. We are working on a fix, that will resolve the issue. We will provide more information soon.

Exports APIs (for ETL pipelines) and Webform Endpoints are not affected by this outage.

Next update at 9:00

Posted Mar 24, 2022 - 08:40 CET

Update

We are still experiencing some difficulties with the Integrations API and Dixa API. We are investigating the matter.

Next update at 09:00

Posted Mar 24, 2022 - 08:31 CET

Update

We are continuing to investigate this issue.

Posted Mar 24, 2022 - 07:57 CET

Update

We are continuing to investigate this issue.

Posted Mar 24, 2022 - 07:56 CET

Investigating

We are experiencing some difficulties with the Dixa API and Integrations API.

We are investigating the matter. Next update at 08.30

Posted Mar 24, 2022 - 07:56 CET

This incident affected: Integrations (Dixa API, Exports API and Integrations API).