Google has released benchmarks showing that GKE Pod snapshots can reduce startup latency by as much as 89%. In its tests, a 70B-parameter model loaded in 37 seconds and an 8B model in 15 seconds. The feature, which became generally available in May, requires clusters running version 1.35.3-gke.1234000 or later.
Rather than caching model files, Pod snapshots checkpoint a running workload and restore its execution state, including CPU and GPU memory. Captured state includes file descriptors, threads, CPU registers, the container root filesystem, EmptyDir volumes and tmpfs mounts. A restored replica resumes without repeating the initialization process that ordinarily loads the model.
Runtime and storage requirements
The capability depends on gVisor, so participating Pods must run in GKE Sandbox. Autopilot clusters already provide that environment; Standard clusters require a node pool with gVisor enabled. A per-node agent manages snapshot lifecycle operations, a control-plane controller removes obsolete snapshots, and Cloud Storage stores the captured data.
Configuration uses two custom resources. PodSnapshotStorageConfig identifies the storage bucket. PodSnapshotPolicy selects Pods through labels, specifies a workload-driven or manual trigger, and controls retention through lastAccessTimeout and a maximum number of snapshots per group.
Google cites Codeway's Retake platform as a customer example. Its existing cache for compiled artifacts had reduced startup to one minute. Lead DevOps engineer Ahmet Furkan Çomak reported that Pod snapshots brought startup down to 8 seconds, allowing the team to launch H100 instances for individual jobs and shut them down afterward.
Compatibility becomes an operational concern
Practitioner discussion has focused on maintaining usable snapshots after capture. In response to a LinkedIn analysis by Suresh Rajashekaraiah, senior DevOps and MLOps engineer Mohana Narasimha G. raised concerns about snapshot invalidation. He identified model digests, CUDA and driver versions, GPU hardware and topology, and runtime configuration as potential compatibility inputs. He also questioned whether snapshots should be immutable artifacts checked before scheduling, with secrets, DNS and downstream connections explicitly refreshed after restoration.
Google documents several enforced compatibility checks. GKE hashes essential runtime fields, known as the distilled Pod spec, and stores that hash in the snapshot. A restoring Pod must generate the same hash. The destination node must also match the original machine series and CPU architecture—for example, N2 to N2 or G2 to G2—and use matching gVisor kernel and GPU driver versions. If no compatible snapshot is available, the Pod follows its normal startup path.
A rootfs-only snapshot has fewer restrictions. Because it does not restore process memory, GKE omits the Pod-spec hash check and permits restoration across machine families, including E2.
Applications must refresh restored state
Restoration does not eliminate application-level recovery work. Google specifies that encryption keys and certificates generated before capture must be recreated afterward. A resumed process otherwise retains the values present when it was frozen.
Environment variables also remain in application memory, where gVisor cannot reliably locate and replace them. Workloads requiring updated values must read them from /proc/gvisor/spec_environ. External connections are terminated during restoration, persistent volumes are excluded from checkpoints, and user-configured iptables or nftables rules and routes are not restored.
These constraints make snapshot maintenance an ongoing platform responsibility. A node-pool upgrade can change the gVisor kernel or GPU driver version and invalidate existing snapshots. The documented fallback starts Pods normally without an error, so the performance improvement can disappear without a failed startup.
Restoration is also progressive rather than instantaneous. The gVisor kernel typically resumes within a few seconds, allowing the application to execute while its memory continues loading in the background. Hardware restrictions apply: whole-pod snapshots are unavailable on E2 machine types, multi-GPU Pods are supported only with L4 GPUs, and Multi-Instance GPU sharing is unsupported.
Security and agent sandbox use
Snapshot storage introduces a governance consideration because a Cloud Storage object can contain a workload's complete running memory. For agent sandboxes, that may include memory from executing untrusted, model-generated code. Access relies on Workload Identity Federation and IAM bindings for each Pod's service account; Google notes that those bindings can take time to propagate.
GKE Agent Sandbox, which also reached general availability in May, builds on this capability. Google says its warm pool can allocate up to 300 sandboxes per second per cluster, with 90% allocated within 200 milliseconds. Pod snapshots let it suspend idle agents instead of keeping their compute resources warm.
Google introduced the open-source Agent Substrate project at the same time to explore higher-density suspend-and-resume multiplexing. Its repository states that it is not production-ready. Meet Shah, AVP of cloud platform and AI engineering, distinguished the generally available Agent Sandbox as a foundation for secure execution from Agent Substrate's still-evolving work on density.
Google positions Pod snapshots for workloads beyond AI inference, including Java applications, game servers and legacy monoliths. Teams still need to decide which node pools use gVisor, where snapshots are stored, who can access them, how long they remain available, and which application state must be refreshed after resumption.