Between 15:09 and 16:02 CEST on 27/09/2023, users with Agent roles were affected sporadically with the message: "Error message: Permission denied. Your agent does not have sufficient roles to log in".
When agents tried to use certain features, they got kicked out of the platform with the screen, as mentioned above.
Our support team triggered the Incident management process following the client's reports.
Our incident manager involved the relevant engineering teams and management at 15:25, and an active investigation began.
After further examination, we deployed a fix at 15:48hs, after which we started to see improvements.
We continued to monitor until full recovery and system stability were confirmed.
Timeline (all times are in CEST):
15:09 We've become aware of a potential issue with sporadic reports of agents.
15:25 We triggered the incident process.
15:28 We reported on the Status page.
15:40 We identified that this only affected users with the role of Agent.
15:48 We deployed a fix. We started to see improvements in the services.
15:52 Moved to Monitoring.