K8S Certificate Rotation, or How I learned to start worrying and never stop
11-17, 14:45–15:15 (UTC), The Theater

This talk will explore the role the client, server, and cluster CA certificates play in a cluster and the ramifications of their expiration. We'll look at a cluster whose certificates have expired and what lead to that scenario. Finally, we'll look at techniques to avoid certificate expiration and how to recover an inoperable cluster.


Expiration and rotation of the internal certificates of a Kubernetes cluster has not been a topic widely discussed in the Kubernetes community. Rather, it's been swept under the rug waiting to trip up operations teams. The purpose of this talk is to bring this topic out into the open. To make our community aware of the pitfalls of certificate expiration and how to restore a cluster that’s lived just a little too long.

This talk will include a real life example of a Kubernetes cluster whose internal certificates have expired. We'll discuss the work it took to restore cluster functionality as well as the business decisions around keeping a cluster over recreating it. This talk will include the non-technical effects on personal and business relationships an outage like this can have.

Certificate expiration can be avoided using the right techniques. This talk will touch on some of these techniques, including: upgrading Kubernetes clusters regularly, using an intermediary certificates, and methods to handle rotation. We'll also touch on monitoring and alerting best practices using Prometheus.

Attendees should expect to walk away from this presentation with an understanding of the role certificates play inside a Kubernetes cluster. They should feel confident in their future decisions on how to care for the whole life of their clusters. Finally, they should know how to avoid certificate expiration and be able to save themselves if they go over the cliff.

See also: None (477.0 KB)

Duffie is a Staff Cloud Native Architect at VMware focused on helping enterprises find success with technologies like Kubernetes. Duffie has been working with all things virtualization and networking for 20 years and remembers most of it. He likes to present on topics ranging from How do I solve this problem with Kubernetes to What even is a CNI implementation and which one should I choose? A student of perspective, Duffie is always interested in working through problems and design choices from more than one perspective.

Nicholas Lane is a Kubernetes Architect at VMware and formerly of Heptio. He’s been using Kubernetes since 2015 when he was a consultant for Red Hat working with OpenShift. Since then Nicholas has become a Kubernetes Org member, became involved in the Azure cluster-api project, and joined the Kubernetes release team. His previous speaking engagements include Kubernetes meetups across North America,at Red Hat Summit, and hosting the regular web series “The Cloud Native Social Hour”.