The bill arrives three weeks late. By then the runaway logging pipeline has already burned $9,000. Here is how we catch spikes in hours, not weeks.
Catching AWS Cost Spikes Before the Invoice
The worst thing about an AWS bill is that it is a report of history. You find out you overspent long after the money is gone. Our finance close runs on the 5th, which means a spike that starts on the 8th sits invisible for nearly a month before anyone with authority sees a number. That is roughly 27 days of a misconfigured NAT gateway, a debug log level left on in production, or a forgotten load test hammering an endpoint. We learned this the hard way when a single service started shipping DEBUG logs to CloudWatch and quietly cost us about $9,000 before the monthly review caught it.
The fix is not a better spreadsheet. It is near-real-time visibility, wired to alerts that reach a human within hours. Here is the setup we actually run.
AWS Cost Anomaly Detection is free and it uses a model to learn your normal spend pattern per service, then flags deviations. You do not tune statistical thresholds by hand. You create monitors, attach subscriptions, and it emails or SNS-pings you when something looks off.
We run two monitors. One watches every service (AWS::CostExplorer::Service dimension), and one watches spend by linked account so a single misbehaving team account shows up on its own. The account-level monitor is the one that has paid for itself most often, because it isolates blame fast.
Creating a monitor and subscription over the API looks like this:
aws ce create-anomaly-monitor --anomaly-monitor '{
"MonitorName": "per-service-monitor",
"MonitorType": "DIMENSIONAL",
"MonitorDimension": "SERVICE"
}'
aws ce create-anomaly-subscription --anomaly-subscription '{
"SubscriptionName": "cost-spikes-slack",
"MonitorArnList": ["arn:aws:ce::123456789012:anomalymonitor/abc123"],
"Frequency": "IMMEDIATE",
"ThresholdExpression": {
"Dimensions": {
"Key": "ANOMALY_TOTAL_IMPACT_ABSOLUTE",
"Values": ["100"]
}
},
"Subscribers": [
{"Type": "SNS", "Address": "arn:aws:sns:us-east-1:123456789012:cost-alerts"}
]
}'
Use IMMEDIATE frequency with an absolute-impact threshold rather than daily digests. A digest that lands at 6am is a digest nobody reads before lunch. Route the SNS topic to Slack through a Lambda or a Chatbot integration so the alert lands where the on-call engineer already lives. The $100 threshold filters out noise; a $12 blip on a $400k account is not worth a ping.
Anomaly detection tells you something changed. Budgets tell you that you crossed a line you drew on purpose, and Budgets can pull a trigger. We set monthly budgets per environment and per major service, at 80% and 100% of forecast. The 80% alert is a warning. The 100% one, for non-production accounts, can attach an action that applies a restrictive IAM policy or stops instances.
A budget with a forecasted-spend alert:
{
"BudgetName": "staging-monthly",
"BudgetLimit": { "Amount": "5000", "Unit": "USD" },
"TimeUnit": "MONTHLY",
"BudgetType": "COST",
"CostFilters": { "TagKeyValue": ["user:env$staging"] },
"NotificationsWithSubscribers": [
{
"Notification": {
"NotificationType": "FORECASTED",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 100,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [
{"SubscriptionType": "SNS", "Address": "arn:aws:sns:us-east-1:123456789012:cost-alerts"}
]
}
]
}
We do not wire automated shutdown actions to production. Killing a prod service to save money is a worse outage than the overspend. In staging and sandbox accounts, though, an action that stops idle instances at the budget ceiling has saved us more than once from a load test that never turned itself off.
An anomaly alert that says "EC2 spend is up $2,000" is a starting point, not an answer. What you want is "the payments team's staging fleet is up $2,000." That only works if every resource carries the tags to attribute it, and if those tags are actually enforced.
We standardized on four allocation tags: team, env, service, and cost-center. Then we made them non-optional with a tag policy in the organization and an SCP that denies resource creation for the expensive services when the tags are missing:
{
"Effect": "Deny",
"Action": ["ec2:RunInstances", "rds:CreateDBInstance"],
"Resource": "*",
"Condition": {
"Null": {
"aws:RequestTag/team": "true"
}
}
}
Enforcement is the part everyone skips and then regrets. Untagged resources drift into an "unallocated" bucket that grows until it is the biggest line item and nobody can explain it. Activate the tags as cost allocation tags in the billing console (they take about a day to start populating), and audit the unallocated bucket weekly until it is close to zero.
Once tags flow, Cost Explorer is enough for most investigations. Group by service, filter by the tag the alert named, set the granularity to daily, and the spike shows up as a step change on a specific day. Nine times out of ten that is all you need.
For the tenth, the Cost and Usage Report is the ground truth. It is the line-item, hourly dataset behind every chart, delivered to S3 and queryable with Athena. When a data transfer charge appears with no obvious owner, CUR's line_item_usage_type column tells you it was cross-AZ traffic on a specific ENI, which points at a chatty service crossing availability zones for no reason.
Almost every spike we have chased falls into one of four buckets. Runaway logging, where a log level or a retry storm floods CloudWatch or ships terabytes to a SIEM. A forgotten environment, usually a load-test or demo account left running over a weekend. Data transfer, the sneakiest one, from cross-AZ or NAT gateway traffic that never shows up in a compute dashboard. And a brand-new service whose defaults are expensive, like provisioned concurrency or a large OpenSearch cluster someone spun up "just to try."
Knowing the four buckets means the triage runbook is short. Check logging volume, check for accounts that should be idle, check data transfer by usage type, check what launched in the last 48 hours.
The tempting move when finance asks about cloud spend is to jump straight to Reserved Instances and rightsizing. That is backwards. You cannot rightsize what you cannot attribute, and a Savings Plan on top of a runaway workload just locks in the waste. Get the monitors, budgets, and tags in place first so every dollar has an owner and every spike has an alarm. Then the deeper cloud cost optimization work has real numbers to act on instead of guesses.
Turn on Cost Anomaly Detection today. It is free, it takes twenty minutes, and it is the single highest-leverage thing you can do about a cost anomaly before the invoice ever arrives. Add per-environment Budgets with SNS alerts this week, and spend the next sprint making allocation tags non-optional. Skip the automated shutdown actions on production; the outage costs more than the overspend. Everything after that is optimization, and optimization only works once you can name where the money goes.
