As a company, we manage hundreds of clusters across multiple users. Achieving consistent, repeatable, and verifiable configurations, all while maintaining a single source of truth, is paramount. That's why we are big fans of the GitOps approach, with a special fondness for Flux. However, the journey doesn’t end once you select a GitOps tool; in fact, it has just started.

The “Day 2” problems of GitOps force you to figure out answers to multiple questions: How to structure your git repositories? How to reuse your configuration? And how do you ensure customer separation? In real life, you also have to handle aspects such as security, configuration layout migrations or providing safe extension points for your users. Oh, and even if you answer all the questions, it’s still possible to accidentally delete a cluster in a really fancy way.

In this talk, I want to share our real-life experience resulting from running, structuring, migrating, and, yes, making errors with our GitOps setup.