Decoding Strategies for LLMs: Greedy, Beam, Top-k, and Nucleus Sampling Explained

You type a prompt into your favorite AI tool. You hit enter. The cursor blinks, and then words appear. But have you ever wondered how the model actually decides which word comes next? It isn't magic. It’s math. Specifically, it is a process called decoding, which refers to the set of algorithms that determine how Large Language Models select subsequent tokens from probability distributions during text generation.

Most people think of an LLM as a single entity. In reality, the model itself just calculates probabilities. It says, "Given these previous words, there is a 40% chance the next word is 'cat,' a 30% chance it is 'dog,' and so on." The decoding strategy is the rulebook that tells the system how to act on those numbers. Do we pick the most likely option every time? Do we roll the dice? Do we look ahead?

The choice of decoding strategy changes everything. It affects speed, creativity, factual accuracy, and even whether the model hallucinates. If you are building an application or just trying to get better results from an API, understanding these four main strategies-Greedy Search, Beam Search, Top-k Sampling, and Nucleus (Top-p) Sampling-is essential.

The Deterministic Approach: Greedy Search

Greedy Search is a deterministic decoding method that selects the single highest-probability token at each step without considering future consequences. Imagine you are walking down a path and at every fork, you always take the road that looks the widest right in front of you. You don’t look around the corner. You don’t consider where that wide road might lead five miles later. You just take the best immediate option.

This is exactly what Greedy Search does. At every step, the model picks the token with the highest probability score. It is fast. It is simple. And it is completely predictable. If you run the same prompt through a model using Greedy Search twice, you will get the exact same output both times.

So, why isn't everyone using it? Because it has a fatal flaw: repetition. Since the model always picks the most likely next word, it often gets stuck in loops. If the model thinks "the" is the most likely word after "and," and "is" is the most likely after "the," it might generate "and the is and the is" endlessly. It lacks variety. For creative writing, this is boring. For code generation or strict fact-checking, however, it can be surprisingly effective because it minimizes randomness.

Looking Ahead: Beam Search

If Greedy Search is looking only at the immediate next step, Beam Search is a deterministic algorithm that maintains multiple candidate sequences (beams) simultaneously to find a locally optimal path rather than just the immediate best token. Instead of picking one best word, Beam Search keeps track of the top K best partial sentences. Let's say K is 3. The model generates three possible continuations. Then, for each of those three, it generates three more. It now has nine paths. It scores all nine, cuts the bottom six, and keeps the top three. It repeats this until the sentence is finished.

This approach solves the repetition problem of Greedy Search. By keeping multiple options open, it avoids getting trapped in local minima-those repetitive loops. It produces higher-quality text for tasks like translation or summarization where coherence over the whole sentence matters more than individual word choices.

But there is a cost. Speed. Beam Search is significantly slower than Greedy Search. According to recent benchmarks, maintaining a beam width of 5 can slow down decoding by nearly 13% compared to Greedy Search. As the sequence length grows, the computational overhead increases. If you need real-time chat responses, Beam Search might introduce noticeable latency. It is also still deterministic. It doesn't add creativity; it just adds quality control.

Introducing Randomness: Top-k Sampling

Now we move from deterministic methods to stochastic ones. This means we introduce randomness. Why? Because human language isn't always logical. Sometimes the "best" word is boring. Sometimes we want surprise. Stochastic methods allow the model to explore less probable but potentially more interesting paths.

Top-k Sampling is a stochastic decoding method that restricts token selection to the k most probable candidates before sampling randomly from them. Here is how it works. The model calculates probabilities for all possible next tokens. Then, it sorts them. It keeps only the top K tokens. All other tokens are discarded. Finally, it picks one token randomly from that shortlist.

If you set K to 10, the model ignores the entire vocabulary except for the 10 most likely words. This prevents the model from choosing absurdly unlikely words (like gibberish), but it still allows for variety within the reasonable options. A common value for K is between 10 and 100. Many practitioners use K=40 as a sweet spot.

The problem with Top-k is that it treats all K tokens equally once they are in the list. If the top token has a 90% probability and the 10th token has a 0.1% probability, Top-k still gives them both a chance. That 0.1% token is still wildly unlikely, but it is now in the running. This can sometimes lead to awkward phrasing if the cutoff K is too high, or robotic repetition if K is too low.

The Smart Filter: Nucleus (Top-p) Sampling

Nucleus Sampling, also known as Top-p Sampling, is a dynamic stochastic decoding method that selects tokens from the smallest set whose cumulative probability exceeds a threshold p. This is widely considered the state-of-the-art for general-purpose text generation. Instead of fixing the number of candidates (like Top-k), it fixes the total probability mass.

Imagine you have a pie chart of all possible next words. Nucleus Sampling starts from the biggest slice and keeps adding slices until the total area reaches P percent. If P is 0.9 (or 90%), the model includes enough top tokens to account for 90% of the probability distribution. It then samples randomly from just those tokens.

Why is this better? Because it adapts to the context. In some situations, the model is very sure. The top word might have 80% probability. In that case, the "nucleus" might only contain two or three words. The output will be focused and coherent. In other situations, the model is unsure. The probabilities might be spread out evenly among many words. The nucleus will include more candidates, allowing for greater creativity.

Typical values for P range from 0.7 to 0.95. A lower P makes the output more conservative. A higher P makes it more diverse. Most commercial APIs default to P=0.9. It strikes a balance between coherence and variety that works well for chatbots, creative writing, and general assistance.

Temperature: The Volume Knob

You cannot talk about stochastic sampling without mentioning temperature. Temperature is a parameter that scales the probability distribution before sampling occurs. It applies to both Top-k and Nucleus Sampling.

Think of temperature as a volume knob for randomness. A temperature of 1.0 leaves the probabilities unchanged. A temperature below 1.0 (e.g., 0.1) sharpens the distribution. It makes the most likely tokens even more likely. This reduces randomness and makes the output more deterministic. A temperature above 1.0 (e.g., 1.5) flattens the distribution. It makes unlikely tokens more competitive. This increases creativity but also increases the risk of nonsense.

For factual tasks, you want low temperature (0.1-0.3). For creative brainstorming, you want higher temperature (0.7-1.0). Be careful, though. Recent studies show that performance drops significantly if you deviate too far from the optimal temperature for a specific task. On math reasoning tasks, moving the temperature by just ±0.2 from the optimum can drop accuracy by nearly 20%. It is not a setting you should ignore.

Which Strategy Should You Choose?

There is no single best decoding strategy. The right choice depends entirely on your goal. Here is a quick guide based on current research and industry practices.

• Factual Q&A and Code Generation: Use Greedy Search or low-temperature Nucleus Sampling. Deterministic methods produce fewer hallucinations. Research shows deterministic methods generate 37% fewer hallucinations on factual tasks than stochastic ones.
• Creative Writing and Brainstorming: Use Nucleus Sampling (Top-p) with a moderate temperature (0.7-0.9). This provides the best balance of coherence and novelty.
• Translation and Summarization: Use Beam Search. The ability to look ahead ensures grammatical correctness and contextual flow over longer sequences.
• Real-Time Chat Applications: Use Greedy Search or optimized Nucleus Sampling. Speed is critical. Beam Search may introduce too much latency.

Also, consider your model. Aligned models (like Chat versions) are less sensitive to decoding choices than base models. If you are using a fine-tuned, instruction-following model, Nucleus Sampling with default settings will likely work great. If you are using an open-source base model, you may need to tune your parameters more carefully to avoid gibberish.

Emerging Trends: Beyond the Basics

The field is moving fast. New methods like Fast Sampled Decoding (FSD) and Speculative Sampling are gaining traction. FSD aims to match the speed of Greedy Search while maintaining the quality of more complex methods. Speculative Sampling uses a smaller "draft" model to predict tokens quickly, which the larger model then verifies. This can speed up inference by 2-3x without changing the output quality.

As models become better aligned with human instructions, the importance of decoding strategy may decrease for standard tasks. However, for specialized applications like medical diagnosis or legal analysis, the choice remains critical. Clinical accuracy can vary by nearly 30 percentage points depending on the decoding method used. Don't treat these settings as black boxes. Understand them, test them, and choose wisely.