Using a language model as a judge

Teams building with language models soon have more answers to check than people can read.

One common answer is to ask another model to grade them. This is often called "LLM as a judge". These notes set out what published research has found about it, and how one open-source evaluation framework handles the details.

How well judges agree with people

A 2023 study by Zheng and colleagues tested strong models as judges of chat answers. As a judge, GPT-4 agreed with human preferences more than 80% of the time. That was about the same rate at which people agreed with each other.

A 2024 study by Thakur and colleagues looked at thirteen judge models on a simpler task, where people agreed closely. Only the largest judge models lined up reasonably well with people. Even they could give scores up to 5 points away from the human scores.

The biases to test for

Zheng and colleagues named three biases. A judge may favour the answer shown first, which is position bias. It may favour the longer answer, which is verbosity bias. It may also favour answers written by itself.

Position bias can be large. Wang and colleagues found that changing the order of two answers was enough to swing a comparison. In one test, a smaller model beat ChatGPT on 66 of 80 questions, with ChatGPT as the judge. One of their fixes was to run each comparison in both orders and combine the results.

Thakur and colleagues also found judges tended to be lenient. Their scores were sensitive to how long and complicated the prompt was.

Measuring agreement properly

The simplest check is to have people grade a sample and count how often the judge agrees. Thakur and colleagues found that this can mislead. Two judges with a high share of agreement could still give very different scores.

So it helps to look past the agreement rate. Compare the scores themselves, and look at the cases where the judge and the people disagree.

Writing the judge's instructions

The UK AI Security Institute built Inspect, an open-source framework for model evaluations. Its model-graded scorers give the judge the question, a grading criterion, the answer and the instructions. By default the judge reasons first and then ends with a grade, such as correct or incorrect.

Inspect reads the last grade in the judge's reply. That stops an earlier grade, written while reasoning or slipped into the answer, from counting. It also rewrites any data markers that appear inside the answer. A model being graded then cannot fake the end of its answer and add grading instructions of its own.

Panels and repeat runs

Inspect can use several judge models and take a majority vote. A grade only wins if more than half of the panel gives it. A judge whose reply cannot be read gives no vote, and the threshold stays the same. If no grade then has a majority, the sample is left unscored.