K8S Certificate Rotation, or How I learned to start worrying and never stop
2019-11-17, 14:45–15:15, The Theater

This talk will explore the role the client, server, and cluster CA certificates play in a cluster and the ramifications of their expiration. We'll look at a cluster whose certificates have expired and what lead to that scenario. Finally, we'll look at techniques to avoid certificate expiration and how to recover an inoperable cluster.

Expiration and rotation of the internal certificates of a Kubernetes cluster has not been a topic widely discussed in the Kubernetes community. Rather, it's been swept under the rug waiting to trip up operations teams. The purpose of this talk is to bring this topic out into the open. To make our community aware of the pitfalls of certificate expiration and how to restore a cluster that’s lived just a little too long.

This talk will include a real life example of a Kubernetes cluster whose internal certificates have expired. We'll discuss the work it took to restore cluster functionality as well as the business decisions around keeping a cluster over recreating it. This talk will include the non-technical effects on personal and business relationships an outage like this can have.

Certificate expiration can be avoided using the right techniques. This talk will touch on some of these techniques, including: upgrading Kubernetes clusters regularly, using an intermediary certificates, and methods to handle rotation. We'll also touch on monitoring and alerting best practices using Prometheus.

Attendees should expect to walk away from this presentation with an understanding of the role certificates play inside a Kubernetes cluster. They should feel confident in their future decisions on how to care for the whole life of their clusters. Finally, they should know how to avoid certificate expiration and be able to save themselves if they go over the cliff.

See also: None