A p99 that jumped to 3.4 seconds during traffic ramps turned out to be cold starts. Here's how we measured them properly and cut the tail, with real init timings.
Serverless Cold Starts: Measuring and Fixing Them on Lambda
A product manager pinged me at 9am: "the app feels slow first thing in the morning." Our average latency looked fine. But the p99 on our order API told a different story, it spiked to 3.4 seconds every day right as European traffic woke up, then settled back to 180ms by mid-morning. Classic cold start signature: pain concentrated at the start of a traffic ramp, invisible in the average.
Total invocation time hides the split between "cold" and "warm." Lambda already logs the init duration, you just have to go pull it. The REPORT line in CloudWatch has an Init Duration field, but only on cold invocations. Query it directly with Logs Insights:
filter @type = "REPORT"
| fields @initDuration
| stats count() as coldStarts,
avg(@initDuration) as avgInit,
pct(@initDuration, 99) as p99Init
Our numbers: about 2,100 cold starts a day, average init 1,850ms, p99 init 2,600ms. That init time is pure overhead the user pays before your code runs a single line. Ours was mostly the Node.js runtime loading a 240MB deployment bundle and initializing an AWS SDK client tree plus a database connection.
We ran du on the unzipped bundle and found the whole AWS SDK v2 shipped in, plus dev dependencies that slipped past the build. Two fixes:
Move to the modular AWS SDK v3 and import only what you call.
// before: pulls the whole SDK
const AWS = require('aws-sdk');
// after: just the DynamoDB client
const { DynamoDBClient } = require('@aws-sdk/client-dynamodb');
Then bundle with esbuild and tree-shake, so only reachable code ships:
esbuild src/handler.js --bundle --minify --platform=node \
--target=node20 --external:@aws-sdk/* --outfile=dist/handler.js
The bundle went from 240MB to 11MB. Init dropped to about 900ms. Package size is the cheapest lever and most teams ignore it.
Package cuts help every invocation but won't get you to zero cold starts. For the morning ramp, we knew roughly how many concurrent executions we'd need, so we bought provisioned concurrency and scheduled it instead of leaving it on 24/7 (that's the expensive mistake).
resource "aws_lambda_provisioned_concurrency_config" "order_api" {
function_name = aws_lambda_function.order_api.function_name
qualifier = aws_lambda_alias.live.name
provisioned_concurrent_executions = 20
}
resource "aws_appautoscaling_scheduled_action" "morning_ramp" {
name = "warm-before-eu-traffic"
resource_id = "function:order_api:live"
scalable_dimension = "lambda:function:ProvisionedConcurrency"
service_namespace = "lambda"
schedule = "cron(45 5 * * ? *)" # 05:45 UTC, before the ramp
scalable_target_action {
min_capacity = 20
max_capacity = 20
}
}
We scale it back to 2 overnight. Running 20 units flat around the clock would have cost roughly $260/month; the scheduled version runs about $70. Provisioned concurrency bills for the time it's allocated whether or not it's used, so the schedule is where the savings live.
For our one Java Lambda, provisioned concurrency felt wasteful for spiky traffic. SnapStart was the better fit, it snapshots the initialized execution environment and restores from it, so the JVM warmup cost is paid once at deploy, not per cold start.
SnapStart:
ApplyOn: PublishedVersions
It took a Java cold start from around 4.2 seconds to under 400ms for us, at no extra runtime charge. The catch: anything you cache during init (a random seed, a connection with a fixed source port) gets frozen into the snapshot and reused across environments. We had to move connection setup out of init and into the handler, and add a runtime hook to reseed randomness.
One thing we tried and abandoned: a "warmer" CloudWatch rule pinging the function every 5 minutes to keep it alive. It kept exactly one environment warm, which does nothing for a ramp that needs twenty at once, and it muddied our invocation metrics. Provisioned concurrency solves the same problem correctly. Ditch the ping trick.
Init duration from 1,850ms to 900ms via bundle cuts, morning p99 from 3.4s to 210ms via scheduled provisioned concurrency, and the Java service fixed with SnapStart. The PM stopped pinging me at 9am.
Fix package size first, it's free and helps everything. Only then reach for provisioned concurrency, and always put it behind a schedule tied to your actual traffic shape. Leaving provisioned concurrency running flat is how serverless quietly gets as expensive as the servers you left behind.
