AWS Graviton Migration: What Broke and What We Saved

AWS Graviton Migration: What Broke and What We Saved#

The finance review flagged our EC2 spend at $48,000 a month and someone asked the obvious question: why are we still on x86 when Graviton is cheaper for the same work? The pitch is roughly 20% lower on-demand price and better price-performance. We ran the migration over six weeks. The savings were real, better than advertised in our case, but the path there had three sharp edges that would have bitten anyone.

The easy 60% of services just worked#

Anything pure-interpreted moved with a one-line change. Our Go services recompiled for ARM64 without a code change. Node and Python services that only used pure-JS or pure-Python packages ran as soon as the base image was ARM. For those, the migration was literally changing the instance type in Terraform.

# before
instance_type = "m6i.xlarge"
# after
instance_type = "m7g.xlarge"   # Graviton3

The m7g.xlarge ran about 19% cheaper than the m6i.xlarge for the same 4 vCPU / 16GB, and in our benchmarks handled slightly more requests per second on the Go services. If your whole stack is like this, stop reading and go do it.

Native dependencies were the wall#

The other 40% hit a wall the moment a package tried to load a compiled binary built for x86. The errors are ugly and not always obvious. A Python service died on import with:

ImportError: .../grpc/_cython/cygrpc.so: cannot open shared object file

The root cause was a wheel compiled for x86_64 baked into our image. The fix is to build on the target architecture. We moved to multi-arch Docker builds with buildx so the image actually contains ARM64 binaries.

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t registry/order-service:1.8.0 \
  --push .

The trap underneath this: our Dockerfile pinned FROM node:20-bullseye but a transitive sharp (image processing) dependency shipped prebuilt x86 binaries and silently fell back to them. We had to force a source rebuild:

RUN npm rebuild --build-from-source sharp

Audit every dependency that has "native," "binding," "compiled," or crypto/image/ML work in its description. Those are where ARM breaks. We found seven across the fleet: sharp, grpcio, some pandas/numpy pins, a Postgres driver, and a Sentry native extension.

The performance regression nobody warned us about#

One service got slower on Graviton, our JSON-heavy API gateway saw p99 climb about 12%. That made no sense until we profiled it. The Node build was using a generic ARM build of an underlying library instead of one tuned for Graviton's specific chip. Newer runtime versions ship better ARM-optimized code paths, so the fix was moving from Node 20.9 to Node 20.12 and updating the JSON library. p99 came back and then went slightly below x86.

The lesson: don't assume ARM equals slower or faster. Profile the actual service. Most got faster, one got slower for a fixable reason, and only measurement told them apart.

CI had to learn ARM too#

Our runners were x86, so images built and tested there passed CI and then failed on ARM in production. We added an ARM runner and a smoke test that runs the actual container on ARM64 before promotion.

build-arm:
  runs-on: ubuntu-24.04-arm
  steps:
    - run: docker buildx build --platform linux/arm64 -t test:arm .
    - run: docker run --rm test:arm /app/healthcheck --strict

Skipping this is how you turn a migration into a series of production incidents. The --strict healthcheck that actually imports every native module caught two problems before users did.

What we actually saved#

Compute dropped from $48,000 to about $33,000 a month, a 31% cut, better than the headline 20% because Graviton3 also let us right-size a few over-provisioned services that now had headroom. Add the RDS instances we moved to Graviton and the total was closer to $16,000/month saved. The migration cost roughly three engineer-weeks spread across the six calendar weeks.

The call we'd make#

Do it, but stage it. Move the pure-interpreted services first for a fast, safe win that funds the rest. Budget real time for the native-dependency tail, that's where the effort concentrates and where a rushed migration causes outages. And put an ARM runner in CI on day one, before you migrate anything, so the whole project has a safety net. The savings are large enough that the payback is measured in weeks, not quarters.