Length Penalties and Repetition Penalties in LLM Decoding: A Practical Guide

Ever noticed your AI chatbot rambling on about the same point three times? Or maybe it gives you a one-sentence answer when you asked for a detailed report? These aren't just random glitches. They are direct results of how Large Language Models (LLMs) calculate the "best" next word. Two specific knobs control this behavior: length penalties and repetition penalties. If you are deploying models or fine-tuning generation parameters, understanding these two hyperparameters is the difference between a usable product and a frustrating demo.

These tools don't change what the model knows; they change how it chooses to speak. They emerged from machine translation systems in the mid-2010s and have since become standard features in major libraries like Hugging Face Transformers. Without them, models tend to produce outputs that are either too short to be useful or stuck in repetitive loops. Let's break down exactly how they work and how to tune them for real-world applications.

The Core Problem: Why Models Get It Wrong

To understand the fix, you first need to see the bug. LLMs generate text autoregressively, meaning they predict one token at a time based on previous tokens. When using Beam Search, a common decoding strategy, the system keeps track of multiple potential sequences (beams) and scores them based on their cumulative log-probability.

Here is the catch: log-probabilities are negative numbers. As a sequence gets longer, you add more negative numbers together, making the total score worse (more negative). This creates a systematic bias where shorter sequences naturally look "better" to the algorithm, even if they are incomplete or less accurate. This is known as length bias. Simultaneously, because the model looks at its own recent output to predict the next step, high-probability phrases can reinforce themselves, leading to repetition loops where the model says "the quick brown fox jumps over the lazy dog" five times in a row.

How Length Penalties Fix Short Outputs

A Length Penalty is a mathematical adjustment applied to the final score of a generated sequence to counteract this preference for brevity. It normalizes the score so that long and short sequences can be compared fairly.

In the original Google Neural Machine Translation (GNMT) system from 2016, researchers defined the length penalty function as:

• lp(Y) = ((5 + |Y|)^α / (5 + 1)^α)

Where |Y| is the length of the output sequence and α (alpha) is an exponent typically tuned between 0.6 and 0.7. The normalized score becomes the log probability divided by this length penalty. In modern libraries like Hugging Face Transformers, this is exposed simply as the length_penalty parameter.

How does the value affect the output?

• Value < 1.0 (e.g., 0.6): Favors shorter sequences. Useful for summarization tasks where conciseness is key.
• Value = 1.0: No length bias correction. This is the default in many configurations but often leads to under-generation.
• Value > 1.0 (e.g., 1.2): Strongly favors longer sequences. Useful for translation or creative writing where completeness matters more than brevity.

If you set this value too high, you risk the model hitting the maximum token limit instead of stopping naturally, resulting in truncated sentences. If you leave it at 0 or very low, you get terse answers that miss context.

How Repetition Penalties Stop Loops

While length penalties adjust the final score, Repetition Penalties operate at the token level during generation. They modify the logits (raw scores) of words that have already appeared in the current output.

The logic is simple: if a word has already been used, make it slightly less likely to appear again. In the Hugging Face implementation, the repetition_penalty parameter works as follows:

• Default (1.0): No penalty. The model treats all tokens equally regardless of history.
• Values > 1.0 (e.g., 1.2): Previously generated tokens have their probabilities scaled down. For example, with a penalty of 1.2, a repeated token's probability is multiplied by roughly 0.83.
• Values < 1.0: Technically possible but rarely used; it would encourage repetition.

This mechanism is crucial for long-form generation. Without it, models often fall into local minima where they repeat a catchy phrase every few seconds. However, this is a blunt instrument. If you set the penalty too high (above 1.5), the model starts avoiding common, necessary words like "the," "is," or "and," leading to awkward, ungrammatical prose. The sweet spot for most conversational and narrative tasks is usually between 1.1 and 1.2.

Comparing the Two Mechanisms

It is easy to confuse these two parameters because they both influence the final text structure. But they solve different problems at different stages of the decoding process. Here is a clear comparison of their attributes and use cases:

Practical Tuning Guidelines

So, how do you actually configure these for your project? There is no single magic number, but there are reliable starting points based on task type. Always validate against a small set of representative prompts rather than relying solely on intuition.

For Translation Tasks: Start with a length_penalty of 0.6 to 0.7. This matches the reference translations' length closely. Keep repetition_penalty at 1.0 unless you notice specific looping issues, as translation requires precise terminology reuse.

For Summarization: Lower the length_penalty to around 0.4 to 0.5. You want the model to prioritize information density over completeness. A slight repetition_penalty of 1.1 can help ensure the summary doesn't restate the same fact in different words.

For Creative Writing or Chatbots: This is where repetition penalties shine. Set repetition_penalty to 1.1 or 1.2 to keep the dialogue fresh. Keep length_penalty near 1.0 or slightly above (1.1) if you want the model to elaborate rather than give brief answers. Monitor for "stilted" language; if the model starts avoiding basic verbs, lower the penalty back to 1.1.

Pro Tip: In Hugging Face Transformers, remember that the repetition_penalty applies to tokens generated *after* the prompt. The input prompt itself is not penalized. This means if you paste a long document into a chat window, the model won't be punished for referencing words from that document in its reply, which is usually the desired behavior.

Common Pitfalls and Troubleshooting

Even with correct settings, you might run into issues. Here are the most common scenarios and how to fix them.

1. The Output is Cut Off Mid-Sentence: This usually means your length_penalty is too high or your max_new_tokens limit is too low. Lower the length penalty by 0.1 increments or increase the token limit.
2. The Model Says "The The The": Your repetition_penalty is likely at the default 1.0. Increase it to 1.2. If it persists, check if your temperature setting is too low (greedy decoding exacerbates repetition loops).
3. The Text Sounds Robotic or Awkward: You probably set the repetition_penalty too high (above 1.5). The model is fighting itself to avoid common words. Drop it back to 1.1.
4. Summaries Are Too Verbose: Increase the length_penalty effect by lowering the value (e.g., from 1.0 to 0.5) or explicitly constrain the maximum length.

Recent research in 2026 suggests that while these penalties are heuristic fixes, they remain the most practical tools for deployment. More advanced methods, like learned reward models, exist but require significant computational overhead. For most production environments, tuning these two simple floats offers the best balance of quality and efficiency.

Frequently Asked Questions