Default-deny, namespace isolation, egress control — the patterns we use, the gotchas around DNS, and where Cilium changed our calculus.
Kubernetes NetworkPolicies in Practice
By default, every pod in a Kubernetes cluster can talk to every other pod. Across namespaces, across services, regardless of intent. From a defense-in-depth perspective that's terrible — a compromised pod can reach anything. NetworkPolicies fix this; they're the kubectl-native way to lock pod-to-pod traffic down.
We rolled out NetworkPolicies cluster-wide ~18 months ago. The benefits are real (compromised-pod blast radius dropped meaningfully); the operational cost is real too. This is what we've learned.
NetworkPolicies are namespaced rules that select pods (by labels) and specify which traffic is allowed in/out. The semantics:
- If no NetworkPolicy selects a pod, all traffic is allowed (the default).
- If at least one policy selects a pod, only what's explicitly allowed is permitted.
- Policies are additive — if two policies select the same pod, the pod allows traffic that either policy permits.
The implication: the moment you write your first NetworkPolicy for a pod, you've moved that pod into deny-by-default. Easy to break things if you forget allowed traffic.
The Kubernetes API for NetworkPolicy is a contract; the CNI plugin enforces it. Not all CNIs do.
- Calico — supports NetworkPolicy (and an extended
GlobalNetworkPolicy). Mature.
- Cilium — supports NetworkPolicy and a much richer extended
CiliumNetworkPolicy (L7 rules, DNS-based egress, identity-based).
- Weave Net, kube-router — supported, less feature-rich.
- Flannel — does NOT enforce NetworkPolicy. If your cluster uses Flannel, the resources exist but do nothing.
We use Cilium. The L7 + identity features pay back the operational complexity.
The first policy we deploy in every namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: payments
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
podSelector: {} matches all pods in the namespace. No ingress or egress rules = deny everything. This makes the namespace deny-by-default for both ingress and egress.
Now you add specific allow rules.
The first thing that breaks under default-deny: DNS. Pods can't resolve hostnames; everything that uses Service names dies. The fix:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: payments
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Allow egress to CoreDNS pods on port 53. We bake this into every namespace alongside the default-deny.
If you're using NodeLocal DNSCache, you also need to allow traffic to the local DNS cache (usually a DaemonSet with a specific IP).
For each Service that other pods talk to, add an ingress rule allowing the expected callers:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: payments-api-ingress
namespace: payments
spec:
podSelector:
matchLabels:
app: payments-api
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
team: web
podSelector:
matchLabels:
app: api-gateway
ports:
- protocol: TCP
port: 8080
This selects pods labeled app: payments-api and allows ingress on port 8080 only from pods labeled app: api-gateway in namespaces labeled team: web.
The matrix of "who calls what" becomes a set of these policies. Painful to bootstrap; easy to maintain once written.
The harder one. Egress from your cluster to external endpoints (databases, third-party APIs, cloud services). Two patterns:
By IP/CIDR. Allow egress to specific IPs:
egress:
- to:
- ipBlock:
cidr: 10.0.0.0/8
ports:
- port: 5432
protocol: TCP
Works for stable IPs (internal services). Doesn't work for cloud services with dynamic IPs (AWS endpoints, third-party SaaS).
By DNS (Cilium-specific). With CiliumNetworkPolicy:
egress:
- toFQDNs:
- matchName: "api.stripe.com"
toPorts:
- ports:
- port: "443"
protocol: TCP
Cilium resolves the FQDN periodically and updates the allowlist. Hugely more usable for external services. The trade-off is Cilium-specific (not portable).
For multi-tenant clusters or just defense-in-depth, deny cross-namespace traffic by default:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-from-other-namespaces
namespace: payments
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {} # any pod IN this namespace
Now ingress to payments pods is only from payments pods. Cross-namespace callers need explicit allow rules.
This catches the "someone deployed a misconfigured service in a dev namespace and it accidentally hit production DBs" class of mistakes.
The Kubernetes-native API is L3/L4. It can't:
- L7 rules. "Allow GET /healthz but not POST /payments." Need Cilium / service mesh.
- DNS-based egress. (See above; Cilium-specific.)
- Pod-identity-based rules. "Allow service A's spiffe identity to call service B." Need service mesh or Cilium identity.
- Logging of denied traffic. Standard NetworkPolicy enforcement is silent. Cilium's Hubble + flow logs add observability.
If you need any of these, you're using Cilium or Istio or similar. We use Cilium for the network-policy layer and Istio for the service mesh on top.
Updating labels breaks policies silently. If a pod's labels change and no longer match the policy's podSelector, the policy stops applying. The pod might become accidentally permissive or accidentally locked-out. Test label changes.
Init containers run under the same policies. An init container that needs to pull artifacts from an unusual location is constrained by the policy. Either add allow rules or use a sidecar pattern.
Egress to the Kubernetes API. Some pods (controllers, operators) need to call the Kubernetes API. The API server lives outside pod CIDRs; you need explicit allow rules.
Egress through hostNetwork. Pods with hostNetwork: true bypass the pod CIDR entirely. NetworkPolicies don't apply to them (or at least, not in the obvious way). Avoid hostNetwork for application pods.
Order of policies. No precedence; policies are additive. If you accidentally write a permissive policy that overlaps a strict one, the permissive wins. Audit policies that grant broad access.
- Denied connections. Hubble (Cilium) gives flow logs with deny reasons. Spike in denies = recent change broke something. We alert on > 10/minute sustained.
- Policy count per namespace. Drift indicator. Some namespaces should have ~5 policies; some have 30. Outliers warrant review.
- Pods without policies. Should be near zero on a fully-locked-down cluster. We have a periodic audit script.
Trying to write policies pod-by-pod. Doesn't scale. We grouped pods by team / role labels and wrote policies against those labels.
Default-deny without dry-run. First rollout, we enabled default-deny in production. DNS broke. Multiple things broke. Now we use "audit mode" first — log what would be denied without actually denying — for a week before flipping to enforce.
Forgetting kube-system traffic. Many infrastructure pods (kube-proxy, CSI drivers, etc.) need to talk to system services. We had to add allow rules for them after a few mysterious failures.
Policies without label discipline. If pods are inconsistently labeled, policies miss them. Pair NetworkPolicy rollout with a label audit.
NetworkPolicies are one of those Kubernetes features that's optional on day one and load-bearing forever after you turn it on. Build them in incrementally, default-deny one namespace at a time, monitor what breaks, fix it. The destination is a cluster where compromise of one pod doesn't mean compromise of everything; the journey is six months of iterative policy tightening. Worth it.
