You've built something and you need to know if it works. So you do what's sensible—you ask an LLM to grade it. Factual accuracy, code quality, agent outputs. The machine judges the machine, and you get a number you can act on. Except that number is lying to you.
Here's the problem. Researchers from the University of Wisconsin–Madison and Yonsei University have quantified exactly how wrong these naive LLM-judge metrics can be. Take a judge with 70% specificity and 90% sensitivity—reasonable numbers for a capable model. If your system has 30% true accuracy, the judge reports 48%. That's an 18 percentage-point overestimate. But if your system actually achieves 90% accuracy, the same judge reports only 84%—now it's underestimating by six points. The bias direction flips depending on whether true accuracy falls below or above 75%.
In other words, the ruler stretches and shrinks depending on what you're measuring.
The fix isn't complicated, but it does require effort. The researchers present a bias-correction formula that needs a calibration dataset—somewhere between 200 and 500 human-verified labels. Once you've paid that upfront cost, you can evaluate unlimited test cases with statistically sound confidence intervals. They've even built an adaptive algorithm that optimally allocates your calibration samples between correct and incorrect examples to minimize uncertainty.
So does this matter for your weekend prototype? Probably not. Rough directional feedback is fine for quick iteration. But for production evaluation systems, prompt optimization experiments, benchmark leaderboards, or any decision with real consequences? An 18 percentage-point error is the difference between shipping and scrapping. The calibration investment is small compared to selecting the wrong model or deploying something broken.
The researchers provide a Python implementation that computes bias-adjusted estimates and confidence intervals. The tools exist. The question is whether you trust a judge who hasn't been sworn in.
