DNS plays a pivotal role in a Kubernetes environment. It is the centerpiece that enables applications to locate each other dynamically. In a production environment, where many pods extensively communicate with each other using DNS, the importance of observability becomes increasingly paramount.

Gaining telemetry insights throughout the lifecycle of a DNS request is challenging due to the numerous hidden systems involved. Components such as the application pod, system resolver, nodelocaldns, CoreDNS, and upstream DNS servers add layers of complexity, making debugging ever more difficult.

In this talk, we will explore the journey of DNS requests across various components with Kuberentes/OS context. We will then move on to tools, starting with the CoreDNS log plugin, before delving into advanced tools like Hubble and Inspektor Gadget’s DNS gadget. By leveraging the power of eBPF, these tools provide deep insights, enabling efficient tracing and resolution of complex DNS problems.