Agreement is a measurement, not an impression
Once a workflow routes AI-drafted actions to a human reviewer, it is tempting to judge review quality by whether the queue moves and complaints stay quiet. That tells you throughput, not accuracy.
The direct way to check whether two reviewers are applying the same standard is to have them independently review a shared sample and score the agreement, using a statistic built for that purpose rather than a raw percent-agreement number.
- Sample the same cases across reviewers on a regular cadence, not only during onboarding.
- Report agreement alongside raw percent match, since the two can diverge sharply.
- Track agreement by case type, not only as one blended number.
Cohen's kappa, and where it misleads you
Cohen's kappa is the standard statistic for inter-rater agreement: it adjusts raw percent agreement for the agreement you'd expect from chance alone. It is well established in clinical and research settings where two independent raters classify the same cases.
It also has a documented failure mode operations teams should know before they rely on it. When the case mix is heavily imbalanced, for example when 95% of AI-drafted actions are routine approvals, kappa can come out low even when reviewers agree on almost every case, because the correction for chance agreement grows unstable as one category dominates. Research comparing kappa to alternatives like Gwet's AC1 documents this 'kappa paradox' directly.
The practical takeaway is not to distrust the statistic outright. It is to report the raw agreement rate next to kappa, and to expect a low kappa on a heavily skewed review queue even when reviewers are, in fact, well calibrated.
Disagreement is diagnostic, not a reviewer problem
When agreement drops on a specific case type, the default reaction is often to retrain the reviewer. In a production workflow, check the rubric and the decision boundary first.
A recurring split between two reviewers on the same category of case usually means the instructions describing that category are ambiguous, the evidence the reviewers see is incomplete, or the case genuinely sits on a boundary the workflow has not defined. All three are fixable in the workflow itself, and none of them are solved by asking the reviewer to try harder.
Keep a short log of disputed cases with the evidence each reviewer saw and the reasoning each gave. That log becomes the input for rewriting the rubric section that produced the split, and for deciding whether that case type needs a third reviewer, a tighter decision boundary, or a different automation threshold altogether.
Build calibration into the operating cadence
Reviewer standards drift over time even without any change to the workflow, simply from repeated exposure to similar cases. Regular calibration sessions, where reviewers score the same sample and reconcile differences together, are the standard countermeasure and belong on the same recurring schedule as any other production monitoring.
NIST's AI Risk Management Framework frames this as part of ongoing measurement: documenting the degree of human oversight applied to AI system output, and using that measurement to adjust oversight intensity over time rather than setting a review process once and leaving it fixed.
Treat the review layer as a system component with its own metrics, not as a one-time hire-and-forget control. A workflow that tracks reviewer agreement the same way it tracks latency or error rate catches rubric drift months before it shows up as a customer-visible mistake.