Cutting Cloud Costs: What Actually Moved Our Bill

A while back our AWS bill was creeping up faster than our usage. We did a quarter of focused cost work and brought it down 38%. Most of the savings came from a small number of specific changes; the rest was cumulative small wins. This is the order we'd do them in if we had to start over, with the actual dollar impact for our profile.

The starting point#

Pre-optimization monthly bill:

• EC2 (compute): ~52% of total
• RDS: ~14%
• S3 + data transfer: ~12%
• Other AWS services: ~22%

Total: meaningful 6-figure annual bill. 38% cut → significant annual savings.

Where cost optimization usually finds money#

In rough order of how often a given change pays off:

1. Right-sizing instances (most underutilized resources)
2. Spot/preemptible/savings plans for steady-state workloads
3. NAT gateway data transfer (often surprisingly large)
4. Storage tier optimization (S3 lifecycle, EBS volume types)
5. Reservations / commitments (RIs, Savings Plans)
6. Killing zombie resources (forgotten instances, old snapshots)
7. Network architecture (cross-region/AZ data flows)
8. Right-sizing managed services (RDS classes, ElastiCache)

We worked roughly in this order. Each change had its own ROI.

Change 1: Right-sizing instances (largest impact)#

We had 200+ EC2 instances. Most were sized for "what we thought peak might be." Real utilization was much lower.

The work:

1. Pulled CloudWatch metrics for all instances (CPU, memory, network) for 30 days.
2. Flagged instances where p95 utilization was < 30% on the bottlenecked resource.
3. Right-sized: smaller instance class, often a generation up (newer = better price/perf).

Specific moves:

• Many m5.xlarge → m6i.large (newer gen, 2x smaller, similar cost-per-perf)
• A handful of c5.4xlarge → c6i.2xlarge (smaller; CPU was the bottleneck and they had headroom)
• Some r5.2xlarge → r6i.xlarge (RAM bottleneck, smaller fit fine)

Saving: ~$8,000/month. Largest single change.

The trick: right-sizing isn't always "go smaller." Sometimes a smaller-but-newer-generation instance is cheaper AND faster. AWS Compute Optimizer (free tool) suggests these moves automatically.

Change 2: Spot for non-critical workloads#

EKS nodes were on-demand. Most workloads were idempotent (could survive a node going away). We migrated:

• Worker pools that handle batch / async work → 100% spot
• Web service pools → mixed (50% spot, 50% on-demand)
• Stateful pools (databases, etc.) → on-demand only

Spot instances are 60-90% cheaper than on-demand depending on the instance type and zone availability.

For Kubernetes specifically, we use Karpenter to manage spot:

• Provisions spot when workloads tolerate it
• Handles graceful node drain on spot interruption (2-minute notice from AWS)
• Falls back to on-demand if spot capacity isn't available

Saving: ~$4,500/month. The interruption rate is real (~2 spot interruptions per week on average) but our workloads handle it cleanly.

Change 3: Reserved capacity / Savings Plans#

For the workloads that stay on-demand (stateful services, baseline web capacity), reserve them:

• 1-year compute Savings Plan covering ~70% of our baseline EC2 spend → ~30% discount
• 1-year RDS Reserved Instances for prod databases → ~40% discount

We don't go 3-year because our workload changes. 1-year is a reasonable balance between commitment risk and savings.

Savings Plans are flexible across instance types, so they're forgiving if we right-size or migrate workloads during the term. Reserved Instances are tied to specific instance classes.

Saving: ~$3,200/month.

Change 4: NAT gateway data transfer#

This one was a surprise. NAT gateway data transfer was costing us ~$2,800/month.

Investigation: a service was downloading a 1.5GB ML model from S3 once per pod startup. The pods scaled up frequently. Multiplied by traffic volume, the NAT egress was massive.

Fix: S3 Gateway VPC Endpoint. Free. Routes S3 traffic directly without going through NAT.

We then audited other AWS service traffic patterns:

• Added Interface VPC Endpoints for ECR (image pulls were going through NAT)
• Added Interface VPC Endpoints for Secrets Manager
• Added Interface VPC Endpoints for SSM (used by all instances for systems manager)

Saving: ~$2,000/month from these endpoints alone.

This is the change we'd do first next time. The ROI is great and the work is mechanical.

Change 5: S3 lifecycle policies#

We had ~40TB in S3. Most of it was old.

Lifecycle policies we added:

• Logs older than 30 days → Glacier Instant Retrieval (~50% cheaper)
• Logs older than 1 year → Glacier Deep Archive (~80% cheaper than Standard)
• Backups older than 90 days → Glacier
• Old artifact builds → Delete after 180 days

Saving: ~$600/month. Smaller in absolute terms but the per-month-of-retention cost is large.

Change 6: Zombie resources#

Things sitting around costing money:

• Old EBS snapshots from terminated instances
• Unattached EBS volumes
• Old RDS manual snapshots
• ELBs with no targets
• NAT gateways in test VPCs that nobody used

We wrote a script that lists likely-zombies; reviewed manually; deleted what wasn't needed.

Saving: ~$400/month. Small but trivial to find and free to fix.

Change 7: Right-sizing RDS#

Same exercise as EC2 but for managed databases:

• Some RDS instances were on r5.4xlarge for "headroom"; actual usage suggested r5.xlarge was fine. Downsized.
• Old db.t2 instances → db.t3 or db.t4g (Graviton; cheaper).
• Storage type review: a couple of databases on io1 (provisioned IOPS) didn't actually need that many IOPS. Switched to gp3 (general purpose, configurable IOPS but cheaper baseline).

Saving: ~$1,200/month.

Change 8: ElastiCache right-sizing and cluster mode#

ElastiCache (Redis) was over-provisioned. We had cache.r5.xlarge × 6 nodes for a workload that fit comfortably on cache.r6g.large × 3 nodes.

Saving: ~$700/month.

Change 9: Data transfer audit#

Cross-region data transfer adds up. We audited:

• An analytics pipeline replicating data from us-east-1 to us-west-2 had unnecessary copies. Consolidated.
• A logging pipeline was sending logs across regions. Moved to in-region.
• Some VPC peering traffic could have been via Transit Gateway with shared services.

Saving: ~$500/month.

Change 10: Misc small wins#

A handful of smaller wins:

• CloudWatch Logs retention policies (most logs default to "forever"; we set 30-90 days). ~$200/month.
• Old EBS snapshots in regions we don't actively use. ~$100/month.
• Reduced CloudWatch alarms to only meaningful ones. ~$80/month.
• Old AMI cleanup. ~$50/month.

Total: ~$430/month from cleanup work.

What didn't work#

A few things we tried that didn't pay off:

Switching to Graviton (ARM) instances broadly. For our specific workloads (mostly Java and Python), some saw 10-20% performance improvement on Graviton; others saw no difference or slight regressions. Net cost savings were modest, and the migration toil (fixing per-arch container images, dependencies) was real. We did it for new workloads, not as a forced migration.

Aggressive scale-to-zero with autoscaling. For very bursty workloads, scale-to-zero saves money. For workloads with steady-state baseline load, scale-to-zero just causes constant scale-up/down churn with no benefit. We're selective.

Multi-cloud price arbitrage. Tried routing some workloads to GCP for specific services where pricing seemed better. The savings on the workloads were real but small; the operational overhead of multi-cloud (which we already have for other reasons) wasn't worth chasing for cost alone.

Self-hosting things to save SaaS costs. Replacing Datadog with self-hosted Grafana + Prometheus saved meaningful $/month on the SaaS bill, but the engineer-time added up to ~the same. Net-positive but the gain is the engineer-time-when-it-matters, not the $.

The discipline of keeping costs down#

After the initial cleanup, the discipline:

• Monthly cost review. A 30-minute meeting per month looking at the trends. Catches drift early.
• Tagging enforcement. Every resource has tags (team, environment, project). The bill is filterable by team. Untagged resources get flagged.
• Cost-per-feature tracking. For specific features (e.g., the customer-support assistant), we know the monthly cost. Anomalies get attention.
• Pre-deployment cost estimates. Major new infrastructure changes include a cost estimate in the PR. Lots of "we don't need that bigger instance class" caught in review.
• Quarterly Compute Optimizer review. AWS's free tool surfaces right-sizing opportunities. We act on the high-confidence ones.

Without ongoing discipline, costs creep back up. The 38% cut was the easy part; staying lean is the harder ongoing work.

What I'd tell a team starting#

Look at NAT data transfer first. Often the biggest surprise. Free fix (VPC endpoints).

Right-sizing tools are free. AWS Compute Optimizer, GCP Recommender, Azure Advisor. Use them.

Spot for stateless workloads. The biggest single lever after right-sizing.

Reserved capacity once you know your baseline. Don't over-commit; 1-year is usually right.

Lifecycle policies on S3 from day one. Costs compound silently without them.

Tag everything. Without tags, attribution is impossible and accountability evaporates.

Monthly cost review. 30 minutes/month catches drift before it becomes a problem.

Cost optimization isn't sexy work but the ROI is high. The 38% cut took ~2 months of focused effort and saved 5-figure-monthly-recurring. That's better ROI than most engineering work. The discipline is in keeping the wins; without ongoing attention, the costs come back.