On February 17th, 2021 at 11.36 am CET our engineering deployed a new version of the Dixa Webhook service. Approximately 1 hour later, we received alerts of a large number of webhook events being queued for processing. In the hour that passed, events from the past two weeks as well as new events had been (and were still) queueing up. This was caused by misconfiguration on a service - something we were not aware of at that time.
After considering our options and finding the cause of the re-queueing, we decided to process all events once more as to not lose new events mixed inside the queue of old events and at 4.01 pm CET all events, new and old, had been processed again.
We have performed a Root Cause Analysis of this incident and are implementing solutions in order to prevent such instances from happening in the future.
We sincerely apologise for any inconvenience that this may have caused.