
An evaluator can agree with human reviewers on almost 97% of its decided cases while approving 40% of the outputs those reviewers consider unacceptable. The synthetic example below does exactly that. Nothing is wrong with the arithmetic; the headline metric answers the wrong operational question.
Before using an LLM judge to accept outputs or supply evidence for a release, measure its false approvals among unacceptable cases, the unacceptable share of its passes, and the cases it cannot decide. Qualify those measurements against a frozen, independently labeled sample. When the relevant sample is too small, the outcome is insufficient evidence, even if no errors were observed.
This is a narrower problem than designing an enterprise AI workflow evaluation gate. The gate consumes measurements. Here, the measurement instrument itself needs qualification.
Give the judge one decision it can actually make
Consider an assistant that drafts customer-support replies. A judge checks whether the reply makes a refund commitment unsupported by the policy and case evidence supplied to it. That is a useful, bounded criterion. Asking whether the reply is “good, safe and compliant” combines too many different decisions to diagnose a failure.
Define the object being judged: the complete draft, the authorized evidence snapshot, the policy version and the applicable criterion. A label of acceptable means acceptable for that criterion with that evidence. It does not certify the entire interaction, the customer’s entitlement or the eventual business action.
Use three outcomes:
pass: the evidence supports acceptance under the criterion.fail: the output violates the criterion.abstain: the evidence is missing, contradictory or insufficient to decide.
Missing policy context must not default to a pass. Invalid JSON, timeouts and evaluator exceptions also produce an unresolved evaluation, with their technical cause recorded separately from a semantic abstention. Count every attempted case; a dashboard that silently drops parser failures measures only the convenient part of the workload.
An evaluator platform can configure prompts, models, input mappings and scoring outputs, as illustrated by LangSmith’s judge setup documentation. That configuration is the start of the measurement contract. It is not evidence that the judge makes the right decisions.
For this example, pass accepts a draft on one criterion. It never authorizes a refund. Keep deterministic policy checks and tool authorization contracts in the execution path.
Read the denominator before reading the score
Suppose domain reviewers adjudicate 1,000 drafts. Of those, 950 satisfy the criterion and 50 violate it. The following counts are invented to demonstrate the accounting; they are not results from a deployed model.
| Human reference label | Judge passes | Judge fails | Judge abstains | Total |
|---|---|---|---|---|
| Acceptable | 930 | 10 | 10 | 950 |
| Unacceptable | 20 | 20 | 10 | 50 |
| Total | 950 | 30 | 20 | 1,000 |
On the 980 cases where the judge decides, it matches the reference 950 times: 96.94% agreement. But it passes 20 of the 50 unacceptable drafts: 40% false approvals conditional on an unacceptable output. A judge that always passed every draft would already achieve 95% agreement on this imbalanced sample.
There is a second, different question: of the 950 drafts the judge passes, how many are unacceptable? Here it is 20/950, or 2.11%. That is the contamination of the accepted set. It depends on the prevalence and composition of failures in the evaluated population; a challenge set deliberately enriched with bad outputs cannot directly estimate this rate for production.
The judge decides on 98% of all cases, yet abstains on 20% of unacceptable ones. Reporting only total coverage hides where manual work accumulates. It also hides whether apparent improvement came from recognizing errors or from declining difficult cases.
Keep these quantities separate:
| Measurement | Denominator | Operational question |
|---|---|---|
| False approvals given unacceptable | All reference-unacceptable cases, including abstentions | How often does this judge let a known violation through? |
| Unacceptable given pass | All judge passes | How contaminated is the accepted set in this sampled population? |
| Agreement on decided cases | Passes plus fails | How often do its explicit decisions match the reference? |
| Decision coverage | All attempted cases | How much work receives a decision? |
| Abstention by reference class and failure family | Cases in that class or family | Where does uncertainty move the work? |
Do not label both conditional error rates “false positive rate.” The meaning of positive changes between evaluation libraries. Store the numerator and denominator with a descriptive name.
The research on LLM judges in MT-Bench and Chatbot Arena identifies position, verbosity and self-enhancement biases while studying agreement with human preferences. Agreement in that setting is useful evidence about that setting. It is not a transferable limit on false approvals for an enterprise policy check.
Separate rubric development from qualification
Start by having domain reviewers label cases independently, without seeing the judge’s verdict or explanation. Adjudicate disagreements against the same policy and evidence. Record unresolved reference cases separately; do not force a binary “ground truth” just to fill a matrix. Report how many cases were excluded from error-rate estimation and why. Exclusion can itself bias the result.
Use a development set to improve the rubric, examples and input format. Then freeze them before a held-out qualification run. Keep related cases together when splitting: paraphrases of one ticket, turns from one conversation and mutations of one source document should not straddle development and qualification as though they were independent evidence.
A practical qualification set has two complementary parts. A representative, time-bounded sample estimates behavior under the sampled workload. A failure-enriched set probes specific defects: unsupported commitments, contradictory evidence, omitted qualifiers, forged references and instructions embedded in the draft telling the judge to award a pass. Retain separate results for the two sampling schemes.
Enrichment provides more opportunities to measure false approvals among bad cases. It does not automatically reproduce the mix of bad cases found in production. Fifty spelling variations of one obvious violation cannot establish coverage of five unrelated failure mechanisms.
For each critical family, retain the human label, judge outcome, reason code, evidence reference and case-family identifier. Compare concise and verbose versions that preserve meaning; for pairwise evaluators, swap candidate order. Such mutations test sensitivity to presentation. They do not multiply the number of independent cases available for a confidence calculation.
Two models agreeing is not an independent reference label. They may share a blind spot. Likewise, a generated confidence of 0.99 is an output to validate, not a measured probability of correctness. This article’s calibration concerns judge decisions against reference labels; it does not claim to calibrate model confidence scores.
Zero observed false approvals still needs a sample size
Assume a frozen judge sees 20 independently sampled unacceptable cases from a defined population and passes none. The observed false-approval rate is zero. Under a binomial model, its exact one-sided 95% upper confidence limit is about 13.91%.
For zero observed errors in n independent cases, the upper limit is:
1 | p_upper = 1 - alpha ** (1 / n) |
This follows by setting the probability of observing zero errors, (1 - p_upper) ** n, to alpha. NIST’s treatment of proportion confidence intervals explains binomial limits and the problems with simple normal approximations when failures are rare. Here the calculation uses a one-sided limit because the decision concerns an upper error threshold.
With zero false approvals, 60 unacceptable cases give an upper limit of 4.87%; 299 give 0.997%. The last number illustrates the evidence needed for a predeclared 1% threshold under these assumptions. It is not a recommended enterprise risk limit. The process owner must select a tolerable error rate for the actual consequences.
The denominator is 299 independently sampled unacceptable cases, not 299 total drafts. Three hundred mostly acceptable drafts may contain very little evidence about the failure mode of interest. If errors are observed, use a method that handles a nonzero count; the zero-error formula no longer applies.
These bounds assume independent trials from the population being claimed, a fixed judge and valid reference labels. They do not cover systematic reviewer mistakes, a new policy regime or a shifted failure mix. If several critical slices must pass simultaneously, specify a simultaneous-inference method or allocate the error budget across them. Separate 95% bounds do not provide a 95% guarantee for the whole collection. Repeatedly tuning and retesting the same holdout also invalidates the intended qualification process.
There is substantial research on correcting inference from imperfect judges, including Noisy but Valid. The small calculation here addresses only the judge’s conditional false-approval rate on human-labeled cases. It does not correct an entire benchmark scored by that judge or establish an application’s production failure rate.
Reproduce the qualification receipt
This standard-library Python example computes the three-outcome accounting and the zero-error bound. It makes no model calls. Empty denominators remain None, so a test set with no unacceptable cases cannot become a report of “0% false approvals.” All arithmetic comes from the synthetic table above.
1 | """Synthetic accounting example. No model calls or measured judge accuracy.""" |
The example produces 0.969388 agreement on decided cases, 0.4 false approvals given unacceptable, 0.021053 unacceptable given pass and 0.98 decision coverage. It also checks the boundary: 298 zero-error cases do not support a 1% upper limit at this confidence level; 299 do. The bound was independently checked against a beta-distribution quantile calculation. This verifies the arithmetic, not a judge model.
For an actual release, expand the output into an immutable qualification record. Include the judge model/version, prompt and rubric hashes, input mapping, decoding configuration, parser version, reference-label policy, dataset version, sampling method, case-family grouping, counts per critical slice, excluded reference cases, infrastructure errors and chosen confidence method. Attach the criterion owner’s threshold and the resulting disposition: qualified for this scope, rejected, or insufficient evidence.
Use the agent audit-log pattern to keep evidence references resolvable without copying sensitive customer data into every report. If a provider does not expose an immutable model version, record the available identifier and run time, and treat that limitation explicitly when deciding how long qualification remains useful.
Decide what an abstention is allowed to do
Abstention is an unresolved case, not successful review. For a draft that requires this check before release, abstention must preserve the hold or route to a qualified reviewer. A separate mandatory business review remains mandatory regardless of the judge’s result.
An all-abstaining judge has no automatic false approvals, but also no useful decision coverage. Set an operating requirement for coverage and manual workload alongside the error requirement. Measure eventual human outcomes as well: routing every difficult case to an overloaded queue can merely relocate the failure. The human-review capacity model explains how to budget that demand before admitting more work.
Do not repeatedly ask the judge until it passes. If a workflow retries an evaluation, the qualification experiment must evaluate the complete retry-and-selection rule. A one-call error rate does not describe a system that accepts any pass from several attempts. Correlated retries also make simple independence calculations unreliable.
Fail closed on missing or malformed verdicts for outputs that require the check. Preserve the original case and reason for escalation. A parser that converts an unknown verdict to pass defeats every measurement in the qualification record.
Qualification expires when the measurement changes
Treat changes to the judge model, rubric, input evidence, parser, retry rule or scored application as reasons to assess and rerun the affected qualification slices. An application update can change the population of bad outputs even when the judge itself is untouched. Historical false-approval measurements then describe the old mix.
In production, independently sample passes as well as failures and abstentions. Reviewing only what the judge flags cannot reveal the violations it approves. Keep a representative audit sample separate from incident-driven cases, retain reviewer disagreement, and compare outcomes by policy version and failure family.
The release decision should be narrow enough to write in one sentence: this frozen judge is qualified to evaluate this criterion, on this evidence and population, with these measured errors and unresolved cases. If the team cannot fill in those fields, a high agreement score is not yet a reason to trust the passes.
