Summary Parts of Dixa’s search functionality malfunctioned during a 30-minute window, following a deployment including some deficient code, affecting, amongst others, the search feature within the Email and Phone composer features, respectively.
Root Cause Starting at 2.28 PM (CET) on 8 Dec, the faulty application started serving requests, some of which it was unable to handle. This was due to an insufficiently tested code path that got invoked under certain feature flag configurations, resulting in the search request not being handled correctly.
Action Items On-call engineers were alerted about elevated error levels within 15 minutes of the first failing requests, and shortly after, were able to identify the impaired deployment, rolling it back to a previous version while investigations continued. At 3.07 PM (CET), error rates related to failing search requests had completely levelled off.
A fix to the root issue has been released since, further improving the overall safety and handling of code dictated by feature flag configurations. This includes more thorough runtime parameter checks as well as adding several regression tests to ensure the issue won’t be reintroduced.