Kubernetes Ingress vs Gateway API: Migrating Without Downtime

Kubernetes Ingress vs Gateway API: Migrating Without Downtime#

The thing that finally pushed us off Ingress wasn't a feature. It was a 200-line annotation block on a single Ingress object that nobody on the team could fully explain. Half of it was nginx-specific rewrite rules, the rest was rate-limiting glued on through nginx.ingress.kubernetes.io/limit-rps, and every change meant reloading the whole controller and praying the config passed the lint step. When a junior engineer added a trailing slash rewrite and knocked out checkout for four minutes, we booked the Gateway API migration for real.

Why the annotation soup exists#

Ingress was designed in 2015 as a lowest-common-denominator spec. It gives you host and path routing and TLS, and nothing else. Every vendor bolted their real features on through annotations, so an Ingress written for nginx is useless on Traefik or Contour. You aren't writing portable config, you're writing nginx config in a Kubernetes costume.

Gateway API splits that mess into typed resources. Cluster operators own GatewayClass and Gateway (the listeners, ports, TLS certs). App teams own HTTPRoute objects that attach to those gateways. Header matching, traffic weighting, and request mirroring are first-class fields, not string annotations a typo can silently break.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: edge
  namespace: gateway-system
spec:
  gatewayClassName: envoy
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.devopsness.io"
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-devopsness-tls
      allowedRoutes:
        namespaces:
          from: All

The part that keeps you online#

The migration only works if both stacks answer traffic during the cutover. We ran the existing nginx Ingress controller and an Envoy Gateway side by side, each on its own LoadBalancer service with its own external IP. DNS still pointed at nginx.

For each service we wrote an HTTPRoute that mirrored the existing Ingress rules, attached it to the new Gateway, and tested it by hitting the Envoy IP directly with a Host header override:

curl -sk -H "Host: api.devopsness.io" \
  --resolve api.devopsness.io:443:203.0.113.44 \
  https://api.devopsness.io/healthz

That --resolve flag is the whole game. You exercise the real route, TLS and all, without touching production DNS. We caught two broken path prefixes and one missing Host rewrite this way before any user saw the new path.

Weighted cutover at the DNS layer#

Once a route passed checks, we shifted traffic with a weighted DNS record in Route 53 rather than flipping it all at once. Start at 5 percent to Envoy, watch p99 latency and 5xx rate for 20 minutes, then ramp.

resource "aws_route53_record" "api_envoy" {
  zone_id = var.zone_id
  name    = "api.devopsness.io"
  type    = "A"
  set_identifier = "envoy"
  weighted_routing_policy { weight = 5 }
  alias {
    name    = aws_lb.envoy.dns_name
    zone_id = aws_lb.envoy.zone_id
    evaluate_target_health = true
  }
}

Our old nginx record stayed at weight 95 and dropped as Envoy climbed. Because both stacks routed to the same pods, a request landing on either edge behaved identically. The one gotcha: sticky sessions. If you rely on nginx.ingress.kubernetes.io/affinity: cookie, Gateway API's equivalent lives in the BackendLBPolicy on some implementations and is missing on others. We checked ours supported it before ramping anything with sessions.

What actually broke#

Two things. First, request body size. nginx defaults to 1MB and we'd raised it to 25MB via annotation; Envoy Gateway defaults higher, so uploads that nginx rejected suddenly went through and hit an app bug nobody knew about. Fix your app, don't just match the limit.

Second, header casing. A legacy internal client sent X-Request-Id and read back x-request-id. nginx normalized casing in a way the client tolerated; Envoy preserved it differently. Cost us an hour of confused debugging. Grep your clients for case-sensitive header reads before you migrate anything old.

The call we'd make#

If you're on a single cluster with five services and nginx is quiet, leave it. The migration is real work and Ingress isn't dying tomorrow. But the moment you have multiple teams fighting over one Ingress object, or you're copy-pasting annotations you don't understand, move to Gateway API. Run both edges in parallel, cut over with weighted DNS at 5 percent, and keep the --resolve trick in your pocket. We finished 40 services in three weeks with zero dropped connections, and deleting that 200-line annotation block felt better than any feature ever could.