LLM-as-judge: how you grade an AI when there's no answer key

Here's a problem that sounds simple until you try to solve it: how do you tell whether your AI is doing a good job?

If the AI is doing arithmetic, you can check. If it's classifying emails as spam or not-spam, you can check. But most of what people actually build with LLMs today doesn't have a clean right answer. "Summarise this 80-page contract." "Draft a customer email explaining the refund policy." "Answer this support question using our internal docs." There is no single correct output. There are better and worse outputs, and reasonable people would disagree about which is which.

But most of what people actually build with LLMs today doesn't have a clean right answer.

That's the problem. And the answer that's quietly become industry standard, the one showing up in Palantir's Foundry release notes this month as a built-in workflow, and in almost every serious agent framework, is called LLM-as-judge. You use one LLM to grade another LLM's work.

I want to walk through how this actually works, because it sounds circular when you first hear it (isn't that the blind leading the blind?) and it isn't. There's a real mechanism underneath, with real trade-offs, and if you're going to ship anything built on LLMs you should understand it.

Why we needed this in the first place

For a long time, the standard way to evaluate an NLP system was with metrics like BLEU or ROUGE. These compare the model's output to a reference answer by counting overlapping words and phrases. They work okay for narrow tasks like machine translation, where there's a gold-standard human translation to compare against.

They fall over completely for the things LLMs are now used for. If I ask a model to summarise a document and it produces a summary that's better than the reference, more accurate, better-structured, clearer, BLEU will score it low because the words don't line up. The metric is measuring string overlap, not quality.

Human evaluation is the gold standard. You get humans to read the outputs and rate them. This works. It's also slow, expensive, and doesn't scale. If you're iterating on a prompt twenty times a day, you cannot convene a panel of human raters each time.

So the field needed something in between: cheaper than humans, smarter than BLEU. LLM-as-judge is that thing.

How it actually works

The mechanism is almost embarrassingly simple. You take an LLM, usually a strong frontier model like GPT-5.1 or Claude, and you give it three things:

1. The input that was sent to the model you're evaluating (the question, the document, the task).
2. The output the model produced.
3. A rubric: a written set of criteria describing what good looks like.

Then you ask the judge model to score the output against the rubric. The rubric might be a simple 1–5 scale ("rate factual accuracy from 1 to 5"), a pass/fail ("does this response cite sources correctly? yes or no"), or a structured breakdown ("score on accuracy, completeness, tone, and safety, and return JSON").

That's it. That's the technique. The judge reads, scores, and often explains its reasoning. You run this over a test set of hundreds or thousands of examples and get aggregate scores. When you change a prompt or swap a model, you re-run the evals and see whether the scores went up or down.

The reason this works better than it has any right to is that judging is easier than generating. A model that would struggle to write a flawless legal summary can often tell you whether a given summary is accurate, complete, and well-structured, the same way a reader who couldn't write a novel can still tell a good novel from a bad one. The judge is doing recognition, not production.

Where the metaphor breaks

I just called the LLM a "judge", and that metaphor is useful but leaks in specific places.

A human judge has independent standards. An LLM judge has the biases of its training. This matters in concrete ways:

• Position bias. If you show a judge two responses and ask which is better, many models systematically prefer whichever one came first (or second, it depends on the model). Serious eval setups counteract this by running each comparison both ways and averaging.
• Length bias. Judges tend to prefer longer responses, even when the longer response is padded or worse. You have to watch for this and sometimes explicitly tell the judge not to reward length.
• Self-preference. A model often rates its own outputs higher than those of other models. This is a real finding, reproduced across labs. It's the reason you generally want a different model as judge than the one being evaluated, or at least a different model generation.
• Rubric sensitivity. Change the wording of the rubric slightly and scores can shift meaningfully. The rubric is not a neutral measuring stick; it's a prompt, and prompt engineering applies.

None of these kill the technique. They just mean you can't treat the judge's scores as ground truth. You treat them as a signal, a directionally useful, cheap, repeatable signal, and you calibrate it against human ratings on a smaller sample to make sure it's tracking something real.

A worked example

Say you're building a support bot that answers questions using your company's internal docs. You want to evaluate it. Here's what an LLM-as-judge setup looks like in practice:

You assemble an eval set of 200 representative questions, along with the relevant doc sections. For each question, you run your bot and capture its answer.

You write a rubric with a few criteria: Faithfulness (does the answer only claim things supported by the docs?), Completeness (does it cover what was asked?), Clarity (is it well-written?). Each scored 1–5.

You send each (question, docs, answer, rubric) bundle to a judge model with instructions like: "You are evaluating a support bot. Score the answer on the three criteria below. For each, give a score and a one-sentence justification. Return JSON."

You aggregate: average score per criterion, distribution of scores, flagged cases where faithfulness dropped below 3. That last one is the gold, the judge is now surfacing hallucinations for you automatically.

Now when you change the retrieval strategy or tweak the system prompt, you re-run the 200 evals in a few minutes and see whether faithfulness went up or down. You have a feedback loop.

This is what Palantir shipping "AIP Evals" inside Foundry means in practice: the whole cycle, author a function, define a rubric, run the judge, see the scores, lives in one place, next to the code. It used to be that you'd bolt this together from three different tools. Now it's the default shape of serious LLM development.

Why it matters

Before LLM-as-judge, shipping LLM-powered features was mostly vibes. Someone would try ten prompts, pick the one that looked best on a handful of examples, and ship. When quality degraded later, nobody knew why, because there was no measurement.

Evals change this. They turn LLM development into something that looks more like ordinary software engineering: you have tests, you run them on every change, regressions are visible. You can't ship well without them, and teams that have this machinery move noticeably faster and more safely than teams that don't.

The deeper point is that evals are the product, in a sense. The eval set encodes what you actually mean by "good" for your use case. Writing a good rubric forces you to articulate standards you'd otherwise leave implicit. Most teams who do this discover that they hadn't really decided what they wanted the system to do until they tried to grade it.

What to watch

Two things, mainly.

One: the quality of judge models is climbing fast, and the gap between "cheap judge" and "expensive judge" keeps shifting. Re-test your judge occasionally against human ratings, a setup that was well-calibrated six months ago may not be now.

Two: the frontier of this work is moving toward judge ensembles (multiple judges voting), reference-free evaluation (judging without a gold answer at all), and continuous evals that run in production, not just in testing. The basic mechanism is stable. The sophistication around it is still compounding.

If you're building with LLMs and you don't have evals yet, that's the highest-leverage thing you can fix this week.

---

Footnotes and links

Further reading

• [Paper] Zheng et al., Judging LLM-as-a-Judge, 2023, the foundational empirical study.
• [Docs] OpenAI Evals framework on GitHub, reference implementation of the pattern.
• [Explainer] Eugene Yan, "Evaluating LLM-based applications", practitioner-focused write-up with concrete rubric examples.
• [Docs] Anthropic's guide to evaluating Claude outputs, includes rubric-writing guidance that generalises across models.

¶