Kubernetes NetworkPolicies in Practice

By default, every pod in a Kubernetes cluster can talk to every other pod. Across namespaces, across services, regardless of intent. From a defense-in-depth perspective that's terrible — a compromised pod can reach anything. NetworkPolicies fix this; they're the kubectl-native way to lock pod-to-pod traffic down.

We rolled out NetworkPolicies cluster-wide ~18 months ago. The benefits are real (compromised-pod blast radius dropped meaningfully); the operational cost is real too. This is what we've learned.

The mental model #

NetworkPolicies are namespaced rules that select pods (by labels) and specify which traffic is allowed in/out. The semantics:

If no NetworkPolicy selects a pod, all traffic is allowed (the default).
If at least one policy selects a pod, only what's explicitly allowed is permitted.
Policies are additive — if two policies select the same pod, the pod allows traffic that either policy permits.

The implication: the moment you write your first NetworkPolicy for a pod, you've moved that pod into deny-by-default. Easy to break things if you forget allowed traffic.

Step 1: pick a CNI that supports it #

The Kubernetes API for NetworkPolicy is a contract; the CNI plugin enforces it. Not all CNIs do.

Calico — supports NetworkPolicy (and an extended GlobalNetworkPolicy). Mature.
Cilium — supports NetworkPolicy and a much richer extended CiliumNetworkPolicy (L7 rules, DNS-based egress, identity-based).
Weave Net, kube-router — supported, less feature-rich.
Flannel — does NOT enforce NetworkPolicy. If your cluster uses Flannel, the resources exist but do nothing.

We use Cilium. The L7 + identity features pay back the operational complexity.

Step 2: default-deny per namespace #

The first policy we deploy in every namespace:

yaml.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

podSelector: {} matches all pods in the namespace. No ingress or egress rules = deny everything. This makes the namespace deny-by-default for both ingress and egress.

Now you add specific allow rules.

Step 3: allow DNS (or break everything)#

The first thing that breaks under default-deny: DNS. Pods can't resolve hostnames; everything that uses Service names dies. The fix:

yaml.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Allow egress to CoreDNS pods on port 53. We bake this into every namespace alongside the default-deny.

If you're using NodeLocal DNSCache, you also need to allow traffic to the local DNS cache (usually a DaemonSet with a specific IP).

Step 4: ingress rules per service #

For each Service that other pods talk to, add an ingress rule allowing the expected callers:

yaml.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-ingress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          team: web
      podSelector:
        matchLabels:
          app: api-gateway
    ports:
    - protocol: TCP
      port: 8080

This selects pods labeled app: payments-api and allows ingress on port 8080 only from pods labeled app: api-gateway in namespaces labeled team: web.

The matrix of "who calls what" becomes a set of these policies. Painful to bootstrap; easy to maintain once written.

Step 5: egress rules to external services #

The harder one. Egress from your cluster to external endpoints (databases, third-party APIs, cloud services). Two patterns:

By IP/CIDR. Allow egress to specific IPs:

yaml.yaml

egress:
- to:
  - ipBlock:
      cidr: 10.0.0.0/8
  ports:
  - port: 5432
    protocol: TCP

Works for stable IPs (internal services). Doesn't work for cloud services with dynamic IPs (AWS endpoints, third-party SaaS).

By DNS (Cilium-specific). With CiliumNetworkPolicy:

yaml.yaml

egress:
- toFQDNs:
  - matchName: "api.stripe.com"
  toPorts:
  - ports:
    - port: "443"
      protocol: TCP

Cilium resolves the FQDN periodically and updates the allowlist. Hugely more usable for external services. The trade-off is Cilium-specific (not portable).

Step 6: namespace isolation #

For multi-tenant clusters or just defense-in-depth, deny cross-namespace traffic by default:

yaml.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-from-other-namespaces
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}  # any pod IN this namespace

Now ingress to payments pods is only from payments pods. Cross-namespace callers need explicit allow rules.

This catches the "someone deployed a misconfigured service in a dev namespace and it accidentally hit production DBs" class of mistakes.

What you can't do with standard NetworkPolicy #

The Kubernetes-native API is L3/L4. It can't:

L7 rules. "Allow GET /healthz but not POST /payments." Need Cilium / service mesh.
DNS-based egress. (See above; Cilium-specific.)
Pod-identity-based rules. "Allow service A's spiffe identity to call service B." Need service mesh or Cilium identity.
Logging of denied traffic. Standard NetworkPolicy enforcement is silent. Cilium's Hubble + flow logs add observability.

If you need any of these, you're using Cilium or Istio or similar. We use Cilium for the network-policy layer and Istio for the service mesh on top.

Gotchas we've hit #

Updating labels breaks policies silently. If a pod's labels change and no longer match the policy's podSelector, the policy stops applying. The pod might become accidentally permissive or accidentally locked-out. Test label changes.

Init containers run under the same policies. An init container that needs to pull artifacts from an unusual location is constrained by the policy. Either add allow rules or use a sidecar pattern.

Egress to the Kubernetes API. Some pods (controllers, operators) need to call the Kubernetes API. The API server lives outside pod CIDRs; you need explicit allow rules.

Egress through hostNetwork. Pods with hostNetwork: true bypass the pod CIDR entirely. NetworkPolicies don't apply to them (or at least, not in the obvious way). Avoid hostNetwork for application pods.

Order of policies. No precedence; policies are additive. If you accidentally write a permissive policy that overlaps a strict one, the permissive wins. Audit policies that grant broad access.

What we monitor #

Denied connections. Hubble (Cilium) gives flow logs with deny reasons. Spike in denies = recent change broke something. We alert on > 10/minute sustained.
Policy count per namespace. Drift indicator. Some namespaces should have ~5 policies; some have 30. Outliers warrant review.
Pods without policies. Should be near zero on a fully-locked-down cluster. We have a periodic audit script.

Things we got wrong #

Trying to write policies pod-by-pod. Doesn't scale. We grouped pods by team / role labels and wrote policies against those labels.

Default-deny without dry-run. First rollout, we enabled default-deny in production. DNS broke. Multiple things broke. Now we use "audit mode" first — log what would be denied without actually denying — for a week before flipping to enforce.

Forgetting kube-system traffic. Many infrastructure pods (kube-proxy, CSI drivers, etc.) need to talk to system services. We had to add allow rules for them after a few mysterious failures.

Policies without label discipline. If pods are inconsistently labeled, policies miss them. Pair NetworkPolicy rollout with a label audit.

What to read next #

Container resource limits — what they actually do — adjacent defense-in-depth
Kubernetes HPA and VPA — tuning from production pain — operational discipline on Kubernetes
Cloud security best practices — securing AWS infrastructure — broader security context
HashiCorp Vault as a secrets backend for Kubernetes — paired secrets discipline

NetworkPolicies are one of those Kubernetes features that's optional on day one and load-bearing forever after you turn it on. Build them in incrementally, default-deny one namespace at a time, monitor what breaks, fix it. The destination is a cluster where compromise of one pod doesn't mean compromise of everything; the journey is six months of iterative policy tightening. Worth it.

#Cilium

Kubernetes NetworkPolicies in Practice