49October 6, 2025/Amanda Milberg

Choosing the Right Model for the Use Case

Selecting the right AI model for deployment is a critical decision that can significantly impact the performance, cost, and user experience of your application. With a wide variety of models available—each with unique strengths and trade-offs—it’s essential to evaluate them carefully based on relevant criteria. In this post, we’ll explore the three key factors to consider when comparing models for deployment: quality, cost, and speed. Understanding how these factors interact and influence your application will help you make informed choices that align with your technical requirements and business goals

50October 6, 2025/Amanda Milberg

Understanding Chargeback in the Context of Self-Hosted Systems

When technology infrastructure—such as GPUs and servers—is owned and managed by a central IT team, the need to allocate costs back to the business units that benefit from these resources becomes a critical consideration. This is particularly relevant in the context of self-hosting AI models, where the initial investment in high-performance GPUs, servers, and supporting infrastructure can be substantial. Without a clear chargeback mechanism, it becomes difficult to ensure accountability, optimize resource usage, and justify the ROI of such investments.

51September 23, 2025/Fergus Finn

LLM guided scheduling

An idea on how to use LLMs to help with scheduling.

52September 22, 2025/Fergus Finn

Scheduling in inference engines

A look into how inference engines choose which requests to process.

53September 19, 2025/Jamie Dborin

Should GPUs make Free Trade Agreements?

Drawing a parallel between optimizing LLM inference and classical economic principles, specifically the concept of comparative advantage.

54September 10, 2025/Jamie Dborin

Behind the Stack, Ep 10: Batched Endpoints

Running LLMs at scale can be expensive. Whether you're building customer-facing chatbots, document extraction pipelines, or research tools, token usage can balloon into thousands of dollars quickly. While infrastructure teams often focus on throughput optimizations (batching requests on the GPU, prefix caching, etc.), there's another lever to pull: endpoint design. One of the most powerful - and under-discussed - endpoint types is the batched endpoint. Instead of prioritizing instant responses, batched endpoints trade latency for cost, cutting your LLM bill in half (or more in some cases).

55September 3, 2025/Jamie Dborin

Behind the Stack, Ep 9: How to Evaluate Open Source LLMs

Choosing the right LLM for your workload isn’t just about picking the latest open-source release or switching to a cheaper closed model. If you’re self-hosting language models - whether for RAG pipelines, agents, or fine-tuned data tasks - knowing how good a model is (and compared to what) is a critical decision.

56July 15, 2025/Jamie Dborin

Behind the Stack, Ep 8: Choosing the Right Inference Engine for Your LLM Deployment

Inference engines are the backbone of self-hosted LLM stacks. They’re responsible for turning model weights into real-time, token-by-token output.

57July 9, 2025/Jamie Dborin

Behind the Stack, Ep 7: Choosing the Right Quantization for Self-Hosted LLMs

Large Language Models (LLMs) are incredibly powerful but also incredibly resource-hungry. Running them efficiently, especially on self-hosted infrastructure, requires squeezing every bit of performance out of limited compute and memory. That’s where quantization comes in.

58July 1, 2025/Jamie Dborin

Behind the Stack, Ep 6: How to Speed up the Inference of AI Agents

AI agents are transforming everything from customer support to autonomous workflows. But under the hood, most AI agent architectures suffer from one major problem: growing latency and cost at scale.

59June 24, 2025/Jamie Dborin

Behind the Stack, Ep 5: Making RAG Work for Multimodal Documents

Most retrieval-augmented generation (RAG) systems assume that documents are clean, structured, and text-based. But in enterprise environments, the reality is different. Documents often contain:

60June 18, 2025/Jamie Dborin

Behind the Stack, Ep 4: Making Your Load Balancer LLM-Aware

If you’re self-hosting LLMs, scaling out to multiple nodes isn’t optional - and neither is load balancing. But conventional strategies like round-robin or least-connections often fail silently when applied to LLM workloads.