Clay Baenziger
Clay Baenziger is a Senior Engineer in Bloomberg’s Cloud Native Compute Services organization, where he focuses on building common tooling used by our Kubernetes platforms, including KServe inference, Jupyter notebooks, Argo workflows, Spark computation, and more. Clay has been active in the development of open source software from computer operating systems to data infrastructure, such as Apache Hadoop and Apache HBase. He is passionate about resolving all kinds of infrastructure challenges.
Senior Software Engineer, Cloud Native Compute Services, Bloomberg
Session
In this case-study, we cover a pattern used at Bloomberg to describe and orchestrate our many in-house infrastructure platforms run on top of Kubernetes. Specifically, we use node-less, Kine-backed API Servers to host CRDs describing our:
* Kubernetes platforms
* Their deployment tiers
* Many, many clusters
* Even more numerous application environments
We describe how data synchronizes from these highly-available, multi-site metadata clusters to the operational platform clusters that operate like any normal Kubernetes cluster.
Issues we've overcome range from referential consistency across CRDs, safety systems which prevent amplification of configuration errors, data transformations from CRDs to propagate Kubernetes resources to operational clusters, and ways to replicate OPA-backed policy stores in etcd for cluster autonomy.
This case study brings lessons learned from using CRDs and Kine-backed API servers to manage multiple Kubernetes clusters. Our work has been similar to that proposed in SIG-Multicluster regarding ClusterSet, ClusterProfile API, and Cluster Inventory API.
By sharing this information, we aim to provide patterns to those who are building similar infrastructure on top of Kubernetes.