Notes on Building AI Systems
Freezing a live CUDA process to host memory and thawing it again, what the driver does — and doesn't — do to make that work, and how understanding that lets us restore CUDA processes up to 4x faster.
Some thinking about how to trade off batching and speculative decoding in a running inference engine.
We rebuilt Moonshot's Kimi agent swarm and pointed it at a real codebase: ~53× fewer tokens and ~45× cheaper than one long-context agent.
Tracing one vector-add kernel from nvcc all the way down to the warps that execute it.
Using artificial analysis benchmarks we try to predict when an open source LLM will be released that matches frontier LLMs.
A breakdown of the modelling choices of three new and popular diffusion language models.
Extending SGLang to improve it's offloading capabiltiies, achieving speedups for compute bound workloads.
A follow-on to the economics of speculative decoding, we run the inference lab simulator on MTP & DFlash drafters with real acceptance data, and find out whether adaptively choosing the draft length is worth it.
How a technically superior but economically isolated solution slowly lost ground to a good-enough one built on infrastructure everyone already owns.
How UCCL reimplements the DeepEP kernels for arbitrary hardware by swapping the transport out from under them.
How expert-parallel dispatch and combine kernels work, built up from scratch: the high-throughput shape and the low-latency one.
Two underexplored axes: what MoE routing does to the decode roofline, and how compressed attention takes away the slack that used to make speculated tokens free.