On-the-fly snapshot compression for elastic inference at scale

When an inference engine starts, it performs several initialization steps before it can handle requests. These include importing runtime dependencies, loading the model weights into GPU memory, allocating the KV cache, running an initial forward pass that triggers just-in-time compilation, and capturing CUDA graphs. Depending on the model size and configuration, this initialization process can take several minutes to complete.

In Cloudburst, we used checkpoint and restore to capture SGLang after this initialization setup had completed. This allows us to create new replicas from that prepared state instead of repeating the same work.

The cost then moves from initialization to storage. A snapshot of a large model can contain tens or hundreds of gigabytes of host memory, which must be written during checkpointing and read during restore. Compression reduces the amount of data stored and read, but it improves restore time only when decompression keeps pace with storage.

CRIU now supports LZ4 compression directly in its memory checkpoint and restore paths. It compresses memory pages while writing the checkpoint and decompresses them during restore, avoiding the need for intermediate storage for uncompressed data and a separate compression pass.

Compressing memory inside CRIU

CRIU stores process memory in pages-*.img files and uses the corresponding pagemap images to describe where those pages belong in the restored address space. To support memory compression, this image format is extended so that CRIU can choose how to store each block of pages.

A block that contains only zeroes is represented in the snapshot without storing any page data. As model weights is the largest component of the memory state and often have a low compression ratio, CRIU stores the compressed bytes as an LZ4 block only if the compressed block size is reduced by more than 12.5%. Otherwise, the original bytes are stored as RAW to avoid the performance overhead of decompression during restore.

This decision is made independently for each block, allowing compressible and low-compression memory to coexist in the same checkpoint.

The pagemap stores the size and representation of each block, while the pages image contains the bytes stored for RAW and LZ4 blocks. During restore, CRIU uses this metadata to locate the stored bytes and reconstruct the original memory pages.

CRIU Data Path

During checkpoint, CRIU stores each memory block as zero, raw, or LZ4-compressed. During restore, it reconstructs the block through the corresponding path.

Checkpoint

1. Process memorycheckpointed mappings
2. Page pipepages + ranges
3. Compress blockszero · raw · LZ4
4. pages-*.imgmemory page data

   Pagemap metadatasizes + pages/block

Restore

1. Inventory + pagemapblock metadata
2. Validate image metadatacounts · sizes · totals · offsets
3. Build restore batcheschoose zero, raw, or LZ4 path

4. No page data read

   Zero blockS = 0 · fill N

   Page data read

   pages-*.img inputmemory page data

   Read stored bytestotal for the planned range

   Raw blockS = N · no decompression

   LZ4 block0 < S < N · decompress N
5. Restored process memoryN bytes from every path

Reading the full flow

Metadata controls the work; the pages image supplies stored bytes

For stored size S and in-memory block size N, restore reads 0 bytes for zero, N for raw, or S for LZ4. Every path writes N bytes into restored process memory.

Restoring compressed memory

The three block representations follow different paths during restore. ZERO blocks require no input data, RAW blocks use the stored bytes without decompression, and LZ4 blocks must be decompressed to their original size.

CRIU cannot leave LZ4 decompression to its final restorer. The restorer is a small position-independent executable (PIE) that runs inside the process being reconstructed without a dynamic loader, so it cannot use liblz4. CRIU therefore decompresses LZ4 blocks in its page reader before the restorer takes over. The page reader already understands the pagemap metadata and can write the decompressed pages into the process memory.

For large model checkpoints, serial decompression can move the bottleneck from storage to a single CPU core. CRIU can group independent blocks into bounded batches and decompress them in parallel. It can also read the next batch while the current one is being processed, overlapping checkpoint I/O with decompression.

For pagemap entries with compression metadata, CRIU limits each restore batch to 32 MiB of memory after decompression. This keeps temporary input buffers and block descriptions bounded even for very large checkpoints. CRIU may keep two batches active, allowing the restore thread to read the next batch while the current one is being decompressed.

CRIU does not always read a checkpoint sequentially. It reconstructs private mappings, shared memory, memfd regions, copy-on-write relationships, and pages inherited from earlier incremental checkpoints. A request that covers only part of an LZ4 block may require the complete block to be decompressed. CRIU caches that block so that later reads can reuse it instead of repeating the same work.

LZ4-compressed page data uses buffered I/O even when CRIU is configured with --image-io-mode direct. CRIU can still use direct I/O when reading aligned RAW page data, while ZERO blocks are restored without reading from the pages image.

Choosing block size and decompression concurrency

CRIU supports page-sized and multi-page compression blocks. The --compress option uses each memory page as a separate block, while the --compress-block SIZE option groups consecutive pages into blocks up to the specified size, which must be page-aligned and no larger than 4 MiB.

Page-sized blocks preserve fine-grained access and work across the widest set of CRIU paths. Because CRIU compresses each block independently, multi-page blocks allow LZ4 matches to cross page boundaries within its roughly 64 KiB window. They also spread per-block processing and metadata overhead across more pages. The trade-off is that asking for one page can require decompressing the complete block, while fewer blocks remain available for parallel decompression.

The best block size depends on the memory contents and restore path. A larger block can improve compression ratio and reduce processing overhead for a local checkpoint, but it can increase the cost of partial reads.

The --decompress-threads N option controls decompression concurrency. The default value of 1 processes each decompression request serially. A value of 0 asks CRIU to choose a limit from the available CPUs and the work in each batch, while a value greater than 1 sets an explicit maximum. CRIU may use fewer threads when a batch contains too little data or too few independent blocks.

Automatic selection uses CPU affinity rather than a cgroup CPU quota. A container with a broad affinity mask should use a cpuset or an explicit limit when fewer CPUs are actually available. A model-serving platform must also account for the combined CPU demand of simultaneous restores. Using more CPUs to restore one replica can delay other replicas or inference workers on the same host.

Checkpoint size and restore latency

We evaluated CRIU memory compression with SGLang running in a container on an AMD EPYC 9335 host with 16 CPUs available to the benchmark and an NVIDIA RTX PRO 6000 Blackwell GPU. Measurements used a warm cache, and model files had already been downloaded before timing began. Each configuration had one warm-up followed by five measurements. The benchmark used deterministic inference and validated the same response before checkpoint and after every restore.

Checkpoint file size below refers to the complete exported checkpoint file, not only the CRIU memory image. No additional compression was applied when the checkpoint was exported. The baseline compared compression disabled with 256 KiB LZ4 blocks and serial decompression.

Across the five model configurations shown below, LZ4 reduced the checkpoint file size by 12.5% to 45.5%.

For model serving, the more useful boundary is the time until the restored server can produce a token. Here, restore to first token starts immediately before the restore command and ends at the first non-empty token in the streamed response. It includes reading the checkpoint file, restoring the container and process state, resuming SGLang’s GPU memory, waiting for the health check, and submitting the first inference request.

LZ4 reduced restore to first token for each of the five model configurations.

The narrower restore measurements show where the balance changes. With serial decompression, the CRIU restore phase and runtime restore latency were higher for Qwen 4B and 9B. The smaller checkpoint file still reduced the time to the first token. For Qwen 27B, Qwen 35B-A3B, and Gemma 26B-A4B, LZ4 also reduced both narrower restore measurements.

Block size for Qwen 3.5 4B

We also compared 4 KiB, 256 KiB, 512 KiB, and 1 MiB blocks for Qwen 3.5 4B, using the same maximum of 16 decompression threads for every size. Complete restore covers the restore command itself, including reading the checkpoint file and reconstructing the container, but ends before SGLang resumes its GPU memory and processes the first request.

For this model and host, 1 MiB had the lowest value for each reported metric. Compared with 256 KiB, it reduced complete restore latency by 11.5% and restore to first token by 10.9%, while checkpoint file size changed by only 0.17%. In this comparison, the larger block mainly reduced processing overhead rather than storage use.

CRIU supports blocks up to 4 MiB. Our evaluation focused on sizes up to 1 MiB to identify how block size affects checkpoint size and overall restore latency. A different model, storage path, or CPU allocation may favour a different block size.

Conclusion

CRIU can now compress memory pages during checkpoint and decompress them during restore, without storing an intermediate uncompressed checkpoint or adding a separate processing step. It uses ZERO blocks to represent all-zero memory pages without storing page data, LZ4 blocks to store compressed bytes, and RAW blocks to store the original bytes when LZ4 reduces the block by no more than 12.5%.

In our SGLang measurements, 256 KiB LZ4 blocks reduced checkpoint file size and restore to first token for all five model configurations. For Qwen 4B, the 1 MiB configuration reduced checkpoint file size and restore latency compared with the smaller block sizes we tested. The best block size will depend on the workload and restore path.

CRIU’s compression is most useful when storage I/O accounts for a significant part of restore latency. Larger blocks can reduce per-block processing overhead, while parallel decompression can use spare CPUs to keep pace with storage.

For snapshot-based model serving, the two optimizations work at different layers. Reusing initialized runtime state avoids repeating model-server startup work. On-the-fly compression reduces the amount of that state that must be stored and read before a restored replica can serve traffic.

The design of the compression mechanism is described in more detail in our EuroMLSys 2026 paper, Towards On-the-Fly Snapshot Memory Compression for Low-Latency Elastic Inference Serving Systems, while the longer version of this article covers the implementation history, additional CRIU details, and the full set of measurements.

@misc{doubleword-efficient-snapshot-compression,
  title        = {On-the-fly snapshot compression for elastic inference at scale},
  author       = {Radostin Stoyanov},
  year         = {2026},
  howpublished = {Doubleword Blog},
  url          = {https://blog.doubleword.ai/efficient-snapshot-compression},
}