25September 10, 2025/Jamie Dborin

Behind the Stack, Ep 10: Batched Endpoints

Running LLMs at scale can be expensive. Whether you're building customer-facing chatbots, document extraction pipelines, or research tools, token usage can balloon into thousands of dollars quickly. While infrastructure teams often focus on throughput optimizations (batching requests on the GPU, prefix caching, etc.), there's another lever to pull: endpoint design. One of the most powerful - and under-discussed - endpoint types is the batched endpoint. Instead of prioritizing instant responses, batched endpoints trade latency for cost, cutting your LLM bill in half (or more in some cases).

26September 3, 2025/Jamie Dborin

Behind the Stack, Ep 9: How to Evaluate Open Source LLMs

Choosing the right LLM for your workload isn’t just about picking the latest open-source release or switching to a cheaper closed model. If you’re self-hosting language models - whether for RAG pipelines, agents, or fine-tuned data tasks - knowing how good a model is (and compared to what) is a critical decision.

27July 15, 2025/Jamie Dborin

Behind the Stack, Ep 8: Choosing the Right Inference Engine for Your LLM Deployment

Inference engines are the backbone of self-hosted LLM stacks. They’re responsible for turning model weights into real-time, token-by-token output.

28July 9, 2025/Jamie Dborin

Behind the Stack, Ep 7: Choosing the Right Quantization for Self-Hosted LLMs

Large Language Models (LLMs) are incredibly powerful but also incredibly resource-hungry. Running them efficiently, especially on self-hosted infrastructure, requires squeezing every bit of performance out of limited compute and memory. That’s where quantization comes in.

29July 1, 2025/Jamie Dborin

Behind the Stack, Ep 6: How to Speed up the Inference of AI Agents

AI agents are transforming everything from customer support to autonomous workflows. But under the hood, most AI agent architectures suffer from one major problem: growing latency and cost at scale.

30June 24, 2025/Jamie Dborin

Behind the Stack, Ep 5: Making RAG Work for Multimodal Documents

Most retrieval-augmented generation (RAG) systems assume that documents are clean, structured, and text-based. But in enterprise environments, the reality is different. Documents often contain:

31June 18, 2025/Jamie Dborin

Behind the Stack, Ep 4: Making Your Load Balancer LLM-Aware

If you’re self-hosting LLMs, scaling out to multiple nodes isn’t optional - and neither is load balancing. But conventional strategies like round-robin or least-connections often fail silently when applied to LLM workloads.

32June 10, 2025/Jamie Dborin

Behind the Stack, Ep 3: How to Serve 100 Models on a Single GPU with No Cold Starts

In many orgs, self-hosting LLMs starts with a single model. Then comes a customisation request. Then another. And before long, you’ve got dozens of fine-tuned variants - each trained with a LORA or other parameter-efficient technique.

33June 4, 2025/Jamie Dborin

Behind the Stack, Ep 2: How Many Users Can My GPU Serve?

When self-hosting LLMs and productionising AI, one of the first practical questions you’ll run into is: “How many users can this system actually support?”

34May 28, 2025/Jamie Dborin

Behind the Stack, Ep 1: What Should I Be Observing in my LLM Stack?

It’s easy to default to GPU or CPU utilization to assess LLM system load - but that’s a trap. These metrics were built for traditional compute workflows and fall short in LLM deployments. They can stay flat while your model silently hits capacity, leading to missed scaling signals and degraded performance.