Judging output quality in a language you do not speak

The problem nobody puts in the ticket

You ship a support bot, a content pipeline, or a translation feature that needs to work in Portuguese, Hindi, Thai, or Japanese. You do not read Portuguese, Hindi, Thai, or Japanese. You still have to sign off on quality before it goes to production. This is a normal position to be in as an engineer working with large language models, and it is a genuinely different problem from evaluating English output, not a smaller version of it.

The uncomfortable part is that fluency and correctness decouple in ways you cannot see. A model can produce grammatically perfect, native-sounding text that is factually wrong, culturally off, or subtly rude, and you will have no way to tell just by reading it, because you cannot read it. That gap is the whole subject here.

Fluent is not the same as correct

LLMs are trained to predict plausible next tokens, not to verify facts. In a language you speak, you catch a wrong answer partly because it reads wrong, awkward phrasing, a term that does not fit, a claim that contradicts something you know. In a language you do not speak, none of those signals reach you. The output can be perfectly idiomatic and still hallucinate a phone number, invert a date format, or mistranslate a negation, which is one of the most common and dangerous failure modes because “not available” and “available” can differ by a single particle in some languages and the sentence still parses fine either way.

This matters more for low-resource languages, where the model has seen far less training data and is more likely to fall back on statistically common patterns that sound right rather than are right. It matters less, but not zero, for high-resource languages like Spanish or Mandarin where the training data is closer in scale to English.

Tokenization already lied to you before generation starts

Before you even get to evaluating the output, check how the model is tokenizing the input and output. Most tokenizers, including the ones behind GPT, Claude, and Gemini families, were built with byte-pair encoding trained on corpora that skew heavily toward Latin-script, English-heavy text. The practical effect is that a sentence in Thai, Burmese, or Amharic routinely takes several times more tokens to encode than the equivalent English sentence, because the tokenizer has fewer whole-word or whole-syllable merges available for that script and falls back to shorter byte-level fragments.

This is not just a cost issue, though it is that too, since you pay per token regardless of what language it is. It is also a quality signal. When a model has to spend more tokens per unit of meaning, it has less effective context window and less room for the kind of internal “reasoning” tokens that longer chain-of-thought approaches rely on. If you are debugging why a model seems to lose the thread halfway through a longer response in one language but not another, check the token count of the same input in both languages before you blame the model’s reasoning.

Back-translation catches gross errors, not subtle ones

The obvious move is to translate the output back into your own language with the same model, or a different one, and read that. This works, but only for a certain tier of error. Back-translation will reliably surface things like wrong numbers, dropped clauses, or a sentence that has clearly gone off-topic. It will not reliably surface register problems, wrong honorifics, awkward but technically correct phrasing, or a translation that is accurate at the sentence level but wrong in cultural context.

There is also a compounding-error risk: if the same model produced the original text and is now translating it back, systematic biases in that model can cancel out rather than reveal themselves. A wrong idiom translated wrong and then translated back wrong can round-trip to something that looks fine. Using a different model or provider for the back-translation step reduces this but does not eliminate it, since the underlying training data overlaps heavily across major model providers.

Minimal pairs beat one-off spot checks

Instead of eyeballing a handful of outputs through back-translation, build a small set of minimal pairs, prompts that differ in exactly one variable you care about, and run both through the same review pipeline. For example, a support-bot prompt where the only change is whether the customer’s message implies anger versus a neutral question, or a product description prompt where only the price changes. If the model handles one variant well and the other badly, back-translation across the pair will usually make that gap visible even without deep fluency, because you are comparing two outputs against each other rather than judging one in isolation.

This is the same logic as writing a regression test instead of manually clicking through an app. You are not trying to certify “the model speaks Vietnamese well.” You are trying to catch specific, repeatable failure classes that matter for your product.

A tempting shortcut is to ask another LLM to grade the output’s quality in the target language. This works better than nothing, but you have to be honest about what you are actually testing: the judge model’s competence in that language, which you also cannot independently verify, since you still do not read it. If the generator and the judge share training data biases, which they often do given how much web-scraped multilingual data overlaps across providers, the judge can rate a systematically wrong pattern as correct because it made the same mistake during its own training.

LLM-as-judge is more useful for structural checks you can verify without knowing the language: did the response answer the question that was asked, does it contain the required fields, is the length within bounds, did it avoid a banned topic. Treat anything the judge says about fluency, tone, or cultural appropriateness as a low-confidence signal, not a verdict.

Native reviewers are not optional, and they are not free

For anything customer-facing, you need a person who actually speaks the language to review a sample before launch and periodically after. This does not have to be a large program. A few hours from a freelance linguist or a bilingual colleague, reviewing a stratified sample of outputs across your minimal-pair test set, catches the register and cultural errors that back-translation and LLM-as-judge both miss. Budget for this as an ongoing line item, not a one-time launch check, because model updates change behavior in languages you cannot monitor yourself, sometimes without any announced change in the English-language behavior you do test.

If you cannot get a native reviewer for every language you support, prioritize by risk and volume: languages where the model is handling anything financial, medical, or legal get reviewed before languages where it is generating generic marketing copy.

Register, honorifics, and the stuff translation tools miss

Some of the highest-stakes errors are invisible to translation entirely, because they are correct at the sentence level but wrong at the social level. Japanese keigo, Korean speech levels, and the German du versus Sie distinction all carry meaning that a literal back-translation into English flattens out. A model can produce a sentence that translates to “thank you for your inquiry” correctly while using a register that would read as inappropriately casual or stiffly formal to a native reader, and you will never see that in the English round-trip.

The fix is not a clever prompt trick. It is asking your native reviewer specifically about register and formality as a separate checklist item, not folding it into a general “does this look right” pass, because a general pass tends to focus on content correctness and skip tone entirely.

What actually goes in a review checklist

A workable process looks like this: build minimal-pair test prompts covering your real use cases, run them through the target language, back-translate with a different model than the generator, flag anything that looks structurally off using an LLM judge for objective criteria only, and route a sampled subset to a native reviewer with a specific checklist covering factual accuracy, register, and cultural fit rather than an open-ended “how does this read.” Repeat the native review after model version changes, not just at launch, since provider model swaps can shift multilingual behavior without touching your English test suite results at all.

The honest tradeoff

None of this makes evaluating a language you do not speak as reliable as evaluating one you do. You are trading direct judgment for a set of indirect signals, tokenization behavior, back-translation, cross-model disagreement, and periodic native review, each of which catches a different slice of errors and misses the rest. The goal is not certainty. It is knowing which failure modes your process actually covers and which ones you are shipping blind on, so that when something goes wrong in a language you cannot read, you at least know where to look first.

If you want more breakdowns like this on how these tools actually behave under the hood, come find us at AI Tool Gazette.