How Large Language Models Use Probabilities to Choose Words and Phrases

Have you ever wondered how an AI writes a poem, answers a question, or even jokes like a human? It doesn’t know what words mean in the way you do. Instead, it guesses. Not randomly. But with math. Every word an LLM writes is a calculated bet - a probability. And understanding that is the key to understanding how these models really work.

The Core Idea: Next-Token Prediction

Large language models don’t think. They predict. At every step, they look at the words you’ve given them - your prompt - and ask: What’s the most likely next word? Then the next. Then the next. It’s like filling in blanks in a sentence, but with billions of options.

This is called next-token prediction. The model has seen trillions of sentences from books, articles, code, and forums. It learned patterns: after “The cat sat on the,” the word “mat” is way more likely than “asteroid.” That’s not because it understands cats or furniture. It’s because “mat” appeared next to those words millions of times in training data.

The model doesn’t store phrases. It stores probabilities. For every possible next word, it assigns a number between 0 and 1. That number is its confidence. A probability of 0.85 means it’s 85% sure that word comes next. A probability of 0.000001? That word is almost certainly wrong.

From N-Grams to Transformers: A Leap in Context

Older models, like n-gram predictors, only looked at the last 2 or 3 words. If you typed “I love to,” they’d guess “eat,” “read,” or “run” based only on those three. But that’s shallow. It couldn’t handle long conversations or complex ideas.

Transformers changed everything. Introduced in 2017, they let models pay attention to any word in the entire input - even if it’s 100 words back. So if you say, “John lives in Paris. He works at a café near the Eiffel Tower. Yesterday, he bought a croissant and,” the model doesn’t just see “a.” It sees the whole context: Paris, café, Eiffel Tower, croissant. All of it feeds into the probability.

Modern LLMs like GPT-4 Turbo can handle up to 128,000 tokens in one go. That’s about 95,000 words - longer than most novels. The model doesn’t get confused. It weighs every relevant clue, no matter how far back. That’s why it can write coherent stories, debate philosophy, or debug code across multiple files.

How Probabilities Are Calculated: Logits and Softmax

Under the hood, the model doesn’t spit out probabilities right away. First, it calculates logits - raw, unnormalized scores for every word in its vocabulary (which can be over 100,000 words). These scores are like points: “mat” gets +4.2, “asteroid” gets -8.1, “table” gets +1.5.

Then comes the softmax function. It turns those scores into probabilities. The higher the logit, the higher the chance. But here’s the trick: softmax makes sure all probabilities add up to 1. So if “mat” has the highest score, it might get 0.85. The next 10 most likely words might share the other 0.15. The rest? Zero.

This is where the model’s knowledge lives - in the weights of its neural network, shaped by training on massive datasets. It doesn’t “remember” facts. It remembers patterns of word co-occurrence. That’s why it can sound smart without understanding.

Choosing the Word: Decoding Strategies

Now that the model has probabilities, how does it pick the next word? There’s more than one way - and the choice changes the output dramatically.

• Greedy decoding: Always pick the word with the highest probability. Simple. Fast. But boring. You’ll get repetitive, generic text. Like a robot reading a textbook.
• Beam search: Keeps track of the top 5 or 10 possible sequences at each step. Picks the best overall path. Better for structured output - like code or answers with fixed formats. Used in translation tools.
• Top-k sampling: Picks from the top 40 most likely words, ignoring the rest. Balances quality and creativity. Common in chatbots.
• Top-p (nucleus) sampling: Picks from the smallest group of words whose probabilities add up to 90%. If the top 3 words already hit 90%, it only picks from those. If the distribution is spread out, it might pick from 50 words. This adapts to context. Used by OpenAI and Anthropic.
• Temperature: A dial that controls randomness. At 0.2, the model is conservative - almost always picks the top choice. At 1.0, it’s wild - every word has a fair shot. Creative writing? Try 0.8. Technical answers? Stick to 0.3.

For example, if you ask an LLM to write a haiku, a temperature of 0.7 and top-p of 0.95 gives you poetic, surprising lines. Use greedy decoding? You’ll get: “The sun is bright / The sky is blue / The grass is green.” Boring. Predictable. Lifeless.

Why It Sometimes Gets It Wrong

Just because a word is statistically likely doesn’t mean it’s true. LLMs don’t fact-check. They pattern-match. If “the moon is made of cheese” appeared often in fantasy stories, the model might say it’s true - because it’s a plausible sequence.

Studies show LLMs generate factually incorrect but statistically probable statements 18.3% more often than correct ones in knowledge-heavy tasks. That’s not a bug. It’s how they’re built. They’re not truth engines. They’re pattern engines.

Another problem: bias. If “A” is the most common answer in training data, the model will favor it - even if it’s wrong. In multiple-choice tests, models pick option A 23.7% more often than they should, regardless of content. That’s not intelligence. That’s statistical noise.

And then there’s repetition. Long outputs often loop. “The cat sat on the mat. The cat sat on the mat. The cat sat on the mat.” Why? Because after a few repeats, the model’s probability distribution gets skewed - the repeated phrase looks more likely than it should. The fix? Use a repetition penalty - a setting that lowers the score of words that just appeared.

Real-World Use and Tuning

Developers don’t just use the default settings. They tweak them.

• Customer service bots: Temperature 0.3, top-p 0.85. Keep answers accurate, short, and safe.
• Marketing copy: Temperature 0.7, top-k 40. Mix creativity with clarity.
• Code generation: Beam search with K=5. Precision matters more than flair.
• Storytelling: Temperature 0.9, top-p 0.98. Let the model surprise you.

Tools like Hugging Face make this easy. You can adjust these settings in a few lines of code. But getting it right takes testing. What works for poetry won’t work for legal documents.

And the cost? Generating each word takes billions of calculations. GPT-4 uses about 420 GFLOPs per token. Llama 3-70B uses 140. That’s why fast LLMs need powerful GPUs. And why your phone can’t run them locally.

The Future: Beyond Probabilities?

Right now, LLMs are brilliant guessers. But they’re not thinkers. They can’t do math reliably. They don’t understand logic. That’s why researchers are blending probabilities with symbolic reasoning.

IBM’s Neuro-Symbolic AI, released in early 2025, checks LLM outputs against knowledge graphs. If the model says “Einstein invented the telephone,” it checks the facts - and corrects itself. This cuts factual errors by nearly 40%.

OpenAI’s upcoming GPT-5 will adjust its decoding strategy based on the task. If you’re asking for a recipe, it uses low temperature. If you’re asking for a joke, it turns it up. No manual tuning needed.

But here’s the truth: even with all these upgrades, the core remains the same. LLMs don’t know. They guess. And their guesses are based on probabilities - shaped by data, tuned by engineers, and filtered by users.

That’s not magic. It’s math. And it’s powerful enough to rewrite how we write, think, and communicate.