Updates and upgrades are normal yet critical occurrences in the lifecycle of any software. They are deployed to introduce new features, improve performance, and fix bugs and security vulnerabilities. However, when a bug surfaces because of such an update, it becomes challenging.
That’s exactly what happened when the latest version of Kubernetes was rolled out last month. Mirantis engineers were working on improving customer experience for their Kubernetes distribution k0s when they found a bug in iptables.
Mirantis Software Engineer Jussi Nummelin explains that due to an incompatibility issue between iptables 1.8.8 and older versions, the network policy and rules get corrupted and block all traffic on the host, making those nodes inaccessible. Shaun O’Meara, Field CTO at Mirantis, points out that their multi-tier testing process helped find and create a workaround for this issue.
If this particular bug remained undiscovered, “You would basically bork every customer’s clusters. It’s like ‘game over’ for all clusters,” says Nummelin.
In this video interview conducted during KubeCon, O’Meara and Nummelin dive deep into the internal processes at Mirantis to ensure the integrity of software. They also demonstrated their commitment to the open source community so it can benefit from their work.