An experimental change targeted at a test environment was accidentally sent to the production environment.
This change had the effect of redirecting all traffic of a central part of Dixa to a single local developer machine.
The single machine was unable to cope with this load.
At Dixa we pride ourselves on delivering stable service and we are truly sorry for such an unnecessary interruption in service.
We will review our internal processes to make them more resilient to human errors.
We use Kubernetes to deploy and manage our services.
Kubernetes is a great platform that has served us well and continue to serve us well as we scale Dixa.
However kubernetes can also introduce some overhead in terms of development deployments.
We are in the process of investigating options reducing this overhead to speed up development.
In the process of testing one of those options (telepresence.io) a production development was inadvertently swapped out.