GKE Pod Snapshots: Cold Starts Drop 89%, If Your Nodes Match
Google's benchmarks show a 70B model restoring in 37 seconds instead of minutes. The catch is a hash and a hardware match that sends the pod back to a cold start when either is off.
Key takeaways
- GKE Pod snapshots checkpoint a running GKE Sandbox pod, including memory and optionally GPU state, to Cloud Storage and restore new pods from it.
- Google reports startup cuts of up to 89%, with a 70B model loading in 37 seconds.
- Restores need an identical Pod spec hash, machine series, sandbox kernel and GPU driver, or the pod cold starts.
On this page
GKE Pod snapshots restore a GPU inference pod from a saved checkpoint instead of reloading weights. Google reports startup cuts of up to 89%: a 70B model in 37 seconds, an 8B model in 15. Use it for scale-to-zero inference on a pinned node pool, because any hardware or driver drift sends you back to a cold start.
Verified against
GKE 1.35.3-gke.1234000 (the documented minimum), against the docs.cloud.google.com Pod snapshots concept, prepare, trigger and restore pages and Google Cloud's Pod snapshots announcement. Checked October 2026.
That fixes the worst part of GPU autoscaling: the minutes a scaled-to-zero pod spends reloading weights. What will bite teams is the restore fence, which mostly shows up as "the pod started slowly again" rather than as an alertable error.
What actually gets checkpointed
Pod snapshots are checkpoint and restore, not caching. GKE captures process memory, execution threads, CPU registers and open file descriptors, plus the container's root filesystem, emptyDir volumes, tmpfs mounts, loopback connections, listening sockets and Unix domain sockets. For GPU workloads the snapshot can optionally include the pod's GPU state, so model weights already sitting in accelerator memory ride along. The whole thing is written to Cloud Storage (the only supported backend) through gVisor, which is why the pod has to run under GKE Sandbox in the first place. PersistentVolumeClaims, active external connections, custom routes and iptables/nftables rules are left out. On restore, external connections are closed, and the pod gets a new IP address and hostname, so anything unique per pod (random seeds, experiment IDs) has to be re-initialized by your app.
That gVisor dependency is the architectural choice to internalize before you plan around this: you are opting into the sandboxed runtime everywhere you want snapshot support, not layering snapshots onto an existing unsandboxed deployment. A snapshot can be triggered by a workload policy or manually, as its own resource against a running pod, not a flag on the Deployment:
apiVersion: podsnapshot.gke.io/v1
kind: PodSnapshotManualTrigger
metadata:
name: inference-pod-snapshot
namespace: model-serving
spec:
targetPod: inference-sandbox-0
kubectl apply -f pod-snapshot-trigger.yaml
# status.snapshotCreated names the PodSnapshot once the checkpoint completes
kubectl get podsnapshotmanualtriggers.podsnapshot.gke.io inference-pod-snapshot -n model-serving -o yaml
kubectl get podsnapshots.podsnapshot.gke.io -n model-serving
The target pod must be selected by a snapshot policy whose triggerConfig.type is manual. There is no separate restore command: a new pod with an identical spec is matched to the most recent compatible, Ready snapshot in its namespace automatically. Pin a specific one with the podsnapshot.gke.io/ps-name annotation.
The number that matters is 37 seconds, not 89%
Headline percentages are easy to misread. The 89% figure is Google's best-case reduction in startup latency across its own benchmarks (llama3-70b is the example it names), but the absolute numbers are what you should plan capacity around: a 70B model back in 37 seconds, an 8B model in 15. Compare that with a cold load of a 70B model from object storage into GPU memory, which routinely runs to several minutes. For a scale-to-zero inference service, that gap is the difference between scale to zero being viable and scale to zero meaning a multi-minute queue on every burst.
Measure it yourself the way the docs suggest: at the point the model server can serve (time to first token or a readiness probe), not when the pod reports Running. A restored pod can show Running before its GPU memory is fully repopulated.
If your workload never scales past a steady floor of warm replicas, none of this changes your bill. It matters specifically for bursty or idle-heavy GPU inference, where the autoscaler that removes a replica during a lull is the one creating the cold start this feature erases. If your replicas are not scaling at all, start with why the Kubernetes HPA is not scaling before adding snapshots on top.
The restore fence: a hash, a machine series, and two versions
This is where adoption gets expensive if you skip the fine print. In the default whole-pod scope, GKE builds what the docs call a distilled Pod spec, a hash of runtime fields such as image, command, args, volumes, volumeMounts, securityContext, runtimeClassName and hostNetwork, and embeds it in the snapshot. A restoring pod has to produce an identical hash. The target node must also run the same machine series, CPU vendor and CPU feature set as the node that made the snapshot, and the GKE Sandbox kernel version and GPU driver version have to match what was captured. E2, and any series that mixes Intel and AMD or CPU generations, is not supported for whole-pod at all.
Miss one of those and you lose the speedup without a clear signal. When no compatible snapshot exists, the docs say the workload performs a cold start; a CPU feature mismatch makes the restore fail. Either way, nothing pages you with "your driver drifted." If your node pools auto-upgrade GPU drivers on a different cadence than you cut new snapshots, you can lose the speedup for weeks without a dashboard pointing at why. The fix is operational: pin the node image and driver version for any pool serving snapshot-restoring workloads, log a marker on restore (the sample app logs GKE Pod Snapshot: restore), and treat a driver bump as an event that invalidates existing snapshots.
There is a second, looser mode: the rootfs-only scope skips the hash check and allows restores across machine families, including E2, because it does not restore process memory. It drops most of the hardware fence, and with it the part that makes GPU cold starts disappear.
Where the GPU cost math actually changes
The honest case for Pod snapshots is narrow: GPU-backed inference that scales to zero or near zero between bursts, on a stable node pool with pinned driver and image versions. That combination lets you stop paying for idle accelerator time without eating a multi-minute cold start on every scale-up. With GPU prices rising in 2026, idle accelerator hours are the line item worth attacking. The prepare guide requires GKE 1.35.3-gke.1234000 or later on both Autopilot and Standard.
Outside that lane, the fence cancels the benefit. Mixed-instance node pools that spread accelerator families across zones for availability will restore some pods cold. Clusters on a release channel that bumps GPU drivers on its own schedule need an explicit process to re-snapshot after every driver change, or the feature quietly degrades to doing nothing. MIG GPU sharing, TPUs and the Cloud Storage FUSE CSI sidecar are not supported, which rules out a lot of existing serving setups.
The decision, concretely
- Do GPU pods scale to zero or near zero between bursts? Pod snapshots directly address this; the 37-second restore for a 70B model is the number to budget against.
- Is your GPU node pool a single pinned machine series and driver version? If yes, adopt it. If you run mixed pools or auto-upgrading images, fix that first or compatibility mismatches will erase the win.
- Do you need cross-machine-family flexibility more than GPU-state restore? Use the
rootfs-onlyscope, and budget for a normal cold GPU load on every restore. - Can your platform team own re-snapshotting after driver bumps? If nobody owns that step, the feature degrades on its own within one driver upgrade cycle.
Questions people ask
Do GKE Pod snapshots work without GKE Sandbox?
No. Pod snapshots depend on the gVisor runtime that GKE Sandbox provides, so every pod you want to checkpoint or restore must run sandboxed. That is the main adoption cost: you are moving the workload onto gVisor, not toggling a feature on an existing runc deployment. Snapshots are stored only in Cloud Storage buckets, and the cluster needs GKE 1.35.3-gke.1234000 or later.
What happens if a Pod snapshot does not match the new node?
If no compatible, ready snapshot exists for the pod's spec hash, machine series, sandbox kernel and GPU driver, the workload simply cold starts as it would without snapshots. The docs say a CPU feature mismatch in whole-pod scope causes the restore to fail. Because there is no "drift" alert, log a restore marker from your app and watch startup latency per node pool.
Can I restore a GKE Pod snapshot on a different machine type?
Only with the rootfs-only scope. The default whole-pod scope requires the same machine series, CPU vendor and CPU feature set, and does not support E2. rootfs-only skips the distilled spec hash and works across families, but it restores only the filesystem, not process memory or GPU state, so model weights still load cold.
The call we'd make
Adopt GKE Pod snapshots for GPU inference that genuinely scales to zero, on a node pool you keep pinned to one machine series and one driver version on purpose. That is a real fix for the worst part of GPU autoscaling economics, and the 37-second restore for a 70B model is worth planning around. Do not adopt it on a mixed-instance or auto-upgrading pool and expect the speedup to hold: the compatibility fence will drop you back to a cold start the moment the hardware or driver drifts, and nothing in the restore path will page you about it.
Sources
Cite this article
Kiril Urbonas (2026). GKE Pod Snapshots: Cold Starts Drop 89%, If Your Nodes Match. DevOpsNess. Updated 2026-11-15. https://www.devopsness.com/blog/gke-pod-snapshots-cut-cold-starts
[GKE Pod Snapshots: Cold Starts Drop 89%, If Your Nodes Match](https://www.devopsness.com/blog/gke-pod-snapshots-cut-cold-starts) (Kiril Urbonas, DevOpsNess, updated 2026-11-15)
More from Cloud
Explore more articles in this category
AWS Cost Allocation Tags for Shared Platforms: What Finally Worked
A hands-on guide to AWS cost allocation tags for shared environments, built from a real platform-team problem: everyone used the cluster, but nobody trusted the bill.
AWS vs Azure vs GCP Pricing in 2026: Real Cost Comparison
The three big clouds are closer on list price than the sales decks suggest. Where they actually diverge is discounts, egress, and Windows licensing.
RDS Restore Drills for Busy Teams: The Recovery Workflow That Surfaced Real Gaps
A hands-on RDS restore drill guide for small cloud teams that thought backups were covered until a timed restore test exposed missing steps, DNS confusion, and stale credentials.
You might have missed
Evergreen posts worth revisiting.