NVIDIA Dynamo Restores LLM Inference...

NVIDIA Dynamo’s preview Shadow Engine Recovery feature is designed to restore LLM inference capacity within seconds after a recoverable engine-process failure. It keeps a fully initialized standby engine on the same GPU and uses GPU Memory Service (GMS) to preserve model weights independently of engine processes.

A conventional cold restart must reload weights into HBM, compile kernels, size the KV cache, autotune, and recapture CUDA graphs. For large models, this can take several minutes while surviving workers handle displaced traffic. With shadow recovery, the standby engine takes over while reinitialization occurs off the serving path.

In a test that terminated one worker in a two-worker GLM-5.2 deployment, cold recovery took 283 seconds. Shadow recovery restored a second serving worker in 7.3 seconds, nearly 39 times faster, reducing disruption to time to first token (TTFT) and per-user decode rate.

Bar chart comparing recovery time: 283 seconds for a cold restart, 7.3 seconds with a shadow engine. — Figure 1. Time until a second worker resumes serving after one of two workers fails. A cold restart reloads weights, sizes the KV cache, autotunes, and recaptures CUDA graphs. A preinitialized shadow engine can begin serving much sooner

Why inference recovery is slow

Recoverable failures can include process crashes, CUDA errors, and transient collective failures even when the hardware, drivers, and node remain healthy. Two issues normally force a complete restart:

Weights are attached to the engine’s CUDA context and process. When that process exits, the driver releases the GPU resources, including resident weights.
NCCL and torch.distributed communicators are process-specific, while CUDA graphs depend on virtual addresses captured by the original process. These components must be recreated.

Shadow recovery addresses these problems by separating weight lifetime from the engine process and preparing non-transferable state before failure.

GPU Memory Service and shared weights

GMS is a per-GPU sidecar that owns physical GPU memory for inference engines. It allocates physical pages, provides handles, and controls access, while engines import those handles and map the pages into their own CUDA contexts. This uses the CUDA Virtual Memory Management API, allowing physical allocations and virtual addresses to have separate lifetimes.

Diagram of two engines mapping one shared copy of the weights in HBM, with GMS off to the side. — Figure 2. Each engine maps the weights into its own address space, but there is only one physical copy in HBM. GMS holds the allocation and hands out the handles; it does not sit between an engine and the memory it reads from

Because allocations are reference-counted, weights remain resident while any process retains a mapping. Multiple engines can therefore access the same physical HBM bytes without duplicating weights. GMS is not involved in ordinary reads after mapping, so a GMS-backed weight read costs no more than an engine-allocated read.

vLLM, SGLang, and NVIDIA TensorRT-LLM integrate GMS through a custom torch.cuda.CUDAPluggableAllocator bound to the weight pool. From the engine’s perspective, weights remain ordinary torch.Tensor objects, and enabling GMS requires a startup flag. The preview does not yet support GMS for KV caches, although that capability is under development.

How shadow engines operate

A shadow engine is a fully initialized process that shares the active engine’s GPUs and weights. It establishes NCCL and NIXL communicators, captures CUDA graphs, imports weight mappings, and completes warm-up before becoming dormant.

A fleet of three workers behind a single router. — Figure 3. A fleet of workers behind a single router. The two-engine layout is internal to each worker, so the router, frontend, and orchestrator need no changes to benefit from it

Each worker contains two engine containers, a GMS sidecar, and a shared lock. The active engine holds the lock, serves requests, maintains a materialized KV cache, and registers with the frontend router. The shadow remains initialized but releases reclaimable memory, holds no KV cache, and waits for the lock.

Before parking, the shadow prepares its CUDA context, graphs, communicators, and weight mappings. It reserves the KV-cache address range without physical backing and materializes that cache only after promotion. This keeps its standing footprint small enough to coexist with the active engine.

Failure and promotion sequence

Image of four panels: engine A active, engine A fails, engine B takes over, engine A returns as the shadow. — Figure 4. The four phases of a recovery. Active and shadow roles swap between engine A and engine B, and the worker returns to steady state without either engine reloading weights

Steady state: Engine A serves while Engine B waits.
Failure: Engine A exits after a crash or liveness-probe intervention, and the kernel releases its lock.
Cutover: Engine B acquires the lock, remaps weights through GMS, materializes its KV cache, and registers with the router.
Restart: Engine A is reinitialized and becomes the new shadow.

A POSIX flock provides mutual exclusion and reliable release on shutdown, segmentation fault, or SIGKILL. A deadlocked process is handled by the Kubernetes liveness probe, which triggers the same kernel-managed takeover path.

Memory accounting

Memory diagram: weights shared throughout, KV cache only on the active engine, buffers and graphs on both. — Figure 5. GPU memory across a recovery

Weights: Allocated once by GMS and mapped read-only by both engines.
KV cache: Held only by the active engine and materialized by the promoted shadow.
Buffers and graphs: Retained by each engine, including the dormant shadow.

GLM-5.2 benchmark

The benchmark used two workers serving GLM-5.2 quantized to NVFP4 on NVIDIA B200 nodes, with one worker per node, TP=8, a 200K maximum context, and an FP8 KV cache. Requests contained 32,000 input tokens and 1,000 output tokens and arrived at 0.7 requests per second. After steady state, one worker received SIGKILL and the system was observed for 600 seconds.

Line chart of time to first token (p50): the baseline climbs above 20 seconds during the outage while the shadow arm stays flat. — Figure 6. Time to first token, 60-second rolling median. The baseline climbs for the whole 283 seconds its second worker is missing, crossing the 5-second line about 90 seconds in. With shadow engine recovery, p50 TTFT is more resilient to disruption

Line chart of inter-token latency: the baseline steps to 84 milliseconds and holds there while the shadow arm degrades slightly and recovers. — Figure 7. Inter-token latency, 60-second rolling median. The baseline steps to 84 milliseconds and holds flat for 283 seconds, well above the 50-millisecond line. The shadow arm rises briefly at the fault and settles back

Shadow recovery reduced the time until a second worker served traffic from 283 seconds to 7.3 seconds: 1.7 seconds for fault detection and 5.6 seconds for promotion.

Post-fault p50 TTFT: 23,815 ms with cold restart versus 1,311 ms with shadow recovery.
Post-fault p50 decode rate: 12 versus 46 tok/s/user.
Requests taking more than 5 seconds to reach the first token: 201 of 399 versus 1 of 398.
Requests below 20 tok/s/user: 226 of 399 versus 90 of 398.

Scope and next steps

Shadow Engine Recovery is being stabilized and will roll out incrementally. It addresses engine-process failures, not hardware, node, or multi-node failures, which still require standard rescheduling. Deployment requires Kubernetes 1.34 or newer with Dynamic Resource Allocation (DRA) enabled and the NVIDIA GPU DRA driver installed.

NVIDIA Dynamo Snapshot can be combined with recovery to reduce contention while a shadow initializes. Promoted shadows currently start with empty KV caches, causing a slight post-cutover TTFT increase. Carrying both the prefix-cache index and cache memory across promotion is an active area of development. vLLM is the primary supported backend, with a Kubernetes quickstart, deployment workflow, and vLLM failover example available in the ai-dynamo/dynamo repository.