working note · confidence: high · 2026-09-02 · [01]

eBPF on the packet and process path

I do not run eBPF because kernel code sounds clever. I run it because the network and the process tree should leave useful evidence.

eBPF is a way to run small, verified programs at selected Linux kernel hooks. The kernel checks each program before it loads. That makes eBPF useful near events that are difficult to see from a normal application.

In this cluster, eBPF has two jobs. Cilium handles the packet path. Tetragon watches the process path.

The packet path

Cilium is the cluster network interface. It also replaces kube-proxy. Service routing, policy decisions, load balancing, and masquerading can happen in the eBPF data path instead of a long chain of packet-filter rules.

The current setup uses native routing. Service traffic uses Maglev load balancing and direct server return. Socket load balancing stays limited to the host namespace. L2 announcements put service addresses on the local network.

That is the fast part. The useful part is that the same data path reports what it did.

Hubble records bounded flow metrics for DNS, TCP, drops, workload pairs, and traffic that leaves the cluster. The labels stop at namespace and workload identity. They do not include every pod name. Authorization headers, cookies, URL queries, and user information are redacted.

I keep Hubble Relay and its user interface off. Prometheus gets the low-cost signals that I use. The raw stream does not become a second observability product by accident.

The process path

Tetragon attaches to kernel events that describe process execution and network connections. Three cluster policies audit shells, sensitive tools, and connections made by the companion process.

Audit is the default. A shell start creates an event. A package tool or network utility creates an event. The event is rate-limited so routine startup does not turn into noise.

One workload has a narrower rule. The platform-tools pod opts into an enforcement policy with two labels. If that exact workload tries to start a shell, package manager, or network utility, Tetragon records the event and kills the process.

The labels matter. I do not want a broad policy that can stop every shell in the cluster. I want a small boundary around a tool container that should not need those child processes in production.

Tetragon also keeps compressed JSON events on each node. The archive is bounded. Prometheus carries the summary, while the node file keeps enough detail for an incident review.

The part that can still fail

Kernel visibility has its own capacity limits. eBPF maps fill. Event rings lose records. A policy can fail to load. A dashboard can look calm because the observer dropped the interesting event.

The cluster alerts on map pressure before it reaches capacity. It also alerts on Cilium event loss, Hubble loss, denied flows, Tetragon ring loss, and unhealthy policy load state. These alerts make the observation path part of the system, not decoration around it.

I keep read-only baseline and canary scripts in the repository. They check every Cilium agent, the active configuration, the metrics endpoints, the dashboards, and the alert rules. A change rolls through Flux, and rollback stays a Git revert.

This post is the probe

This note is also an end-to-end test. Adding this file should rebuild the site and its RSS feed. The workflow should then create one Listmonk campaign for confirmed subscribers.

Stalwart relays that message. Cloudflare reports the final recipient-server result. ntfy reports the terminal CI state.

That chain is much larger than eBPF. The point is the same: a production path should leave evidence at every boundary.