Throughput vs Latency: Optimizing LLM Inference Speed

You've probably felt it. You type a prompt into an AI chatbot, and there's that awkward pause before the first word appears. Or maybe you're building a high-volume API service where every millisecond costs money, but users are complaining about lag. This is the core tension in running Large Language Models (LLMs): the battle between throughput and latency.

It’s not just a buzzword issue. It’s a hardware reality. If you optimize for one, you usually hurt the other. Understanding how Transformer architecture dictates this tradeoff is the difference between a snappy user experience and a server bill that eats your profit margin.

The Core Definitions: What Are We Actually Measuring?

Before we tweak any settings, let's clear up the jargon. These two metrics measure completely different things, yet they share the same GPU resources.

Here’s the catch: these metrics fight each other. To get high throughput, you need to process many requests at once (batching). But processing many requests at once slows down the response time for each individual request because the GPU is busy juggling everyone else's data.

The Batch Size Dilemma

If you change only one setting in your inference engine, make it batch size. It is the primary control knob for the throughput-latency tradeoff.

Imagine a GPU as a chef.

• Small Batch: The chef cooks one dish at a time. It comes out instantly (low latency), but the kitchen is mostly idle (low throughput).
• Large Batch: The chef cooks 64 dishes simultaneously. The kitchen is humming (high throughput), but the first customer waits longer for their plate (high latency).

Research on NVIDIA A100 GPUs shows that increasing batch size from small to large (e.g., 64) can boost throughput by 14x, but latency might spike by 4x. For interactive apps like code editors, that 4x latency hit is unacceptable. For background document summarization, who cares if it takes a few seconds longer, as long as you process millions of documents cheaply?

Prefill vs. Decode: Two Different Worlds

Transformer inference happens in two distinct phases, and they behave very differently regarding batching.

1. Prefill Phase: The model processes your entire input prompt at once to build its internal context. This phase is compute-bound. Even a single request can saturate the GPU’s calculation units. Batching helps here, but diminishing returns kick in quickly because the math is already heavy.
2. Decode Phase: The model generates output tokens one by one. This phase is memory-bandwidth-bound. The GPU spends most of its time fetching weights from memory rather than doing complex math. Because the math is light, the GPU sits idle waiting for data. This is where batching shines. Combining 32 decode steps allows the GPU to fetch weights once and apply them to 32 sequences, amortizing the memory cost.

This distinction explains why modern schedulers like vLLM use continuous batching. They don't wait for a whole batch to finish decoding. Instead, they mix new prefills with ongoing decodes, keeping the GPU busy without letting any single request stall indefinitely.

How Tensor Parallelism Impacts Speed

When models get too big for one GPU (like Llama-3-70B), you split them across multiple GPUs using Tensor Parallelism (TP). While TP reduces the computational load per GPU, it introduces a hidden tax: communication overhead.

Every time the model passes data between layers, the GPUs must talk to each other via NVLink or PCIe. This "all-reduce" operation happens twice per layer. Here’s the problem: the amount of data sent doesn’t shrink when you add more GPUs; it stays constant relative to the sequence length. So, while you do less math per GPU, you spend more time waiting for network transfers.

The result? Adding more GPUs via Tensor Parallelism often increases latency scalability issues. You might get lower theoretical compute time, but the network chatter eats up those gains. For low-latency applications, TP can actually make things slower if the interconnect bandwidth isn't massive enough.

Scheduling Strategies: Who Goes First?

Your scheduler decides which requests get GPU cycles. Most systems prioritize either prefill or decode, creating specific performance profiles.

Systems like Sarathi-Serve have shown that smart scheduling can improve throughput by up to 6.9x compared to older methods while keeping latency within acceptable limits. The key is balancing the queue so that no request starves while maintaining high GPU occupancy.

Choosing Your Optimization Target

There is no "best" configuration. There is only the right configuration for your application.

Scenario A: Interactive Chatbot You need low TTFT. Users hate staring at a spinner. Optimize for low latency. Use smaller batch sizes or aggressive prefill prioritization. Accept lower throughput and higher cost per token.

Scenario B: Document Summarization Pipeline You have thousands of PDFs to process overnight. No human is watching. Optimize for max throughput. Use huge batch sizes and decode-prioritizing schedules. Let latency rise; you care about total tokens processed per hour.

Scenario C: Code Completion This is tricky. You need low TTFT (so suggestions appear instantly) AND decent throughput (because developers trigger completions constantly). This requires sophisticated continuous batching and potentially speculative execution techniques.

Practical Heuristics for Engineers

Don't guess. Measure. But start with these rules of thumb:

• Monitor P95 Latency, Not Average: Averages hide outliers. If your average TTFT is 200ms but P95 is 2s, 5% of your users are having a bad time. Fix the tail.
• Watch Memory Bandwidth: If your GPU utilization is high but throughput isn't scaling with batch size, you're likely hitting memory bandwidth limits during the decode phase.
• Prompt Length Matters: Longer prompts increase TTFT significantly because the prefill phase gets heavier. If your users paste 10-page docs, your latency profile changes drastically compared to short queries.
• Test Real Workloads: Synthetic benchmarks often lie. Use your actual distribution of prompt lengths and output lengths to tune batch sizes.

The field is moving fast. New architectures like Mamba and hybrid models aim to reduce the quadratic complexity of attention, potentially loosening the throughput-latency grip. But for now, mastering the Transformer inference pipeline remains a critical skill for any AI engineer.