Non-English Evaluation: How to Test LLMs Beyond English

Most large language models (LLMs) speak fluent English. Ask them to write a poem in French or debug code described in Japanese, and the quality often drops sharply. This gap is not just an annoyance; it is a serious risk for businesses and developers deploying AI globally. The field of non-English evaluation is the systematic testing of AI models on tasks conducted in languages other than English to measure performance gaps and improve capabilities has moved from academic curiosity to industry necessity. We need rigorous ways to test these models so they don't fail when users switch languages.

Why LLMs Struggle Outside English

You might assume that because an LLM was trained on the entire internet, it should be equally good at every language. The data says otherwise. The primary culprit is training data imbalance where the overwhelming majority of high-quality pretraining text is in English. English dominates the curated datasets used to build these models. When you train a model mostly on one language, it becomes biased toward that language's structure and vocabulary.

Tokenization also plays a huge role. Many non-English languages use scripts that are not Latin-based or have complex morphology. For example, agglutinative languages like Turkish or Finnish combine many words into single long strings. Current tokenizers often break these into many small pieces, increasing the sequence length and making it harder for the model to learn patterns. This technical friction leads to what localization experts at companies like LILT is a language intelligence and localization company tracking enterprise LLM deployment challenges call "quality degradation." They see consistent errors in translation and content generation when models leave the comfort zone of English.

The Menlo Framework: Measuring Native-Like Quality

Old evaluation methods simply translated English questions into other languages. This approach fails because it ignores cultural nuance. A direct translation might be grammatically correct but sound robotic or rude to a native speaker. Enter the Menlo framework which provides a structured evaluation system for native-like output across 47 language varieties using human-annotated pairs.

Menlo does not just ask if the answer is right. It asks if the answer sounds like it came from a local expert. It uses a four-dimensional rubric:

• Language quality and coherence: Is the grammar solid and the flow natural?
• Cultural alignment: Does it respect local norms, idioms, and politeness levels?
• Localized factuality: Are facts accurate within that specific cultural context?
• Writing style: Is the tone appropriate for the audience?

Researchers built this dataset with 6,423 human-annotated prompt-response preference pairs. They found that even advanced LLM judges struggle here. Zero-shot LLM judges (models evaluating without fine-tuning) perform better when forced to choose between two responses (pairwise evaluation) rather than giving a single score. However, they still fall short of human annotators. To fix this, teams are now using reinforcement learning (RL) to train evaluation models, creating generative reward models that can guide policy models toward better multilingual output.

High-Stakes Testing: Medical Licensing Exams

In healthcare, a subtle mistranslation can be dangerous. You cannot rely on general conversation metrics. Researchers are turning to standardized exams to test safety. One prominent study evaluated LLMs on the Chinese National Medical Licensing Examination (NMLE) as a benchmark for assessing clinical reasoning and knowledge retention in Chinese-language medical contexts.

The results were stark. Models that passed English medical exams often failed the Chinese version. Why? Because they misinterpreted domain-specific terminology, misunderstood locally used abbreviations, or got confused by translation artifacts in the exam questions. These studies use statistical tests, like t-tests, to prove that performance gaps between languages are significant. This proves that passing an English benchmark does not guarantee safety in a non-English clinical setting. If you are building a medical AI tool for China, Japan, or Brazil, you must test against local licensing standards, not just translated USMLE questions.

Coding Across Borders: The LLM of Babel

Developers often describe coding tasks in their native language. "Fix this bug in my Python script" might be said in Spanish or German. The LLM of Babel is a research project that evaluates how well LLMs handle code-related tasks when prompts are given in non-English languages study from Delft University highlights this blind spot.

Even if a model is great at writing Python, describing the task in a different language causes performance to drop. The researchers created an Error Taxonomy for these failures:

1. Incorrect functional behavior: The code doesn't work at all.
2. Partial fulfillment: Only part of the requested feature is implemented.
3. Misinterpretation of terms: The model misunderstands technical jargon in the non-English prompt.
4. Language mixing: Inappropriate switching between languages in comments or variable names.

To measure this automatically, they use cosine similarity metrics to compare generated code and explanations against expected outputs. This shows that non-English evaluation must cover multimodal tasks like "non-English prompt → code," not just pure text generation.

Best Practices for Your Evaluation Strategy

If you want to deploy LLMs globally, stop relying on English-only benchmarks. Here is how to build a robust non-English evaluation pipeline:

• Use native speakers: Hire annotators who live in the target locale. They catch cultural nuances that translators miss.
• Favor pairwise comparison: Asking humans or LLM judges to pick the better of two responses yields higher consistency than scoring each response individually.
• Leverage local exams: For professional domains, use existing national certification exams as your gold standard.
• Create error taxonomies: Categorize failures specifically for your use case (e.g., medical vs. coding) to identify root causes.
• Calibrate automatic judges: Use RL-trained LLM judges to scale evaluation, but always validate them against human preferences to avoid overestimating improvements.

Ignoring these steps means you are flying blind. As LILT notes, enterprises face business risks when they assume English performance translates directly to other markets. By adopting frameworks like Menlo and domain-specific benchmarks, you ensure your AI is truly multilingual, not just multilingual-capable in theory.