04-01, 13:55–14:25 (UTC), Track 1 (UTC)
In container land, image tags are mutants. Are you using “latest” tag, or per-environment tags like “dev”, “staging”, “prod”, etc.? Then, you might not be aware, but you are already suffering their attack!
In this talk, we will analyze some use cases where mutability of tags could be troublesome, like:
* Race conditions when deploying an image in different cluster nodes.
* Time-of-Check vs Time-of-Use (TOCTOU) security issues that allows an attacker to trick image scanners with admission controllers or OPA Gatekeeper and run unverified images in Kubernetes.
* Garbage collection not reclaiming space in the registry storage.
* Accidental deletion of images using the registry API.
Should tags be always mutable? Immutable? Should we use regular expressions? How can we prevent these security incidents and accidents from happening? Which approach is the best? Join this session to find out!
In this talk we will start explaining the meaning of tag mutation, and how you can end up running the same tag but different images in different environments if you don’t take care.
Then we will focus and make a demo on some race-condition problems with Kubernetes that can lead to security problems, allowing an attacker to bypass an image scan triggered by admission controller.
We will share our experience with Harbor registry, and how lack of knowledge on the internals of the Docker registry and how the API works can lead to misassumptions leading to destruction or corruption of data. For example, in Harbor UI a user can remove an image by tag, but the underlying Docker registry API delete manifest endpoint (https://docs.docker.com/registry/spec/api/#deleting-an-image deletes by digest). So, deleting a tag will result in the deletion of all the tags pointing to the same manifest. Also, users might wonder why the garbage collection is releasing very little space and ignore the fact that due to CA IDs, manifests are preserved in the registry when a tag is mutated. This prevents the old tag layers from being reclaimed by the GC.
Finally, we will see how registries usually allow enabling immutability as a feature and open a debate about tag mutability: should tags be always mutable? Always immutable? Should we use regular expressions? Which approach is the best? Join this session to find out.
Álvaro is a solutions engineer at Sysdig. Before that, he worked in an IoT and financial micro-transactions company for 11 years as a Ninja developer. Then, he discovered he had been doing the fuzzy concept of DevOps when Adidas hired him for CI/CD support and platform engineering. He loves dissecting things to discover the internals, but only for tech stuff, not living beings. Alvaro was also the founder and leader of AMSN (an open-source clone of MSN messenger) project several years ago.