Confidence is not the same thing as correctness
It is common to route a case to automation when the model reports 90% confidence and to a human when it reports less. That design assumes the confidence score is calibrated: that among all the cases where the model says 90%, roughly 90% are actually correct.
Research on modern neural networks found the opposite is common. Larger and deeper models frequently produce confidence scores that are systematically overstated relative to their real accuracy, a problem the paper calls miscalibration. A model can be both more accurate and more overconfident than the smaller model it replaced.
The practical consequence is that a threshold picked by eye, or copied from a vendor's default, is not protecting the workflow the way it looks like it is. It has to be checked against outcomes, not just against the score.
One threshold cannot serve every decision
A workflow usually reuses one confidence cutoff across every case type it handles, because it is simpler to configure. But the cost of a wrong automated action is not the same everywhere in the workflow.
Approving a low-value, easily reversible action on a false positive is a minor cost. Approving a high-value or irreversible action, such as a payment, a policy exception, or a write-back to a system of record, on the same false positive can be expensive or hard to unwind. NIST's AI Risk Management Framework frames this directly as a risk-based approach: the rigor of controls should scale with the magnitude and reversibility of potential harm, not be applied uniformly.
In practice this means setting the threshold per decision type, not per model. A support-ticket categorizer and a refund-approval step calling the same underlying model should not necessarily share a cutoff.
- List each decision the workflow can take automatically.
- Rate the cost of a false approve and a false deny separately for each one.
- Set the threshold, and the fallback to human review, per decision, not once for the whole workflow.
Build the threshold from labeled outcomes, not a demo
A threshold chosen by watching a handful of promising outputs in a demo will not survive contact with the full distribution of real cases. It needs a labeled evaluation set, meaning past cases with a known correct outcome, so the team can measure precision and recall at different cutoffs before picking one.
This is also where calibration gets fixed, not just measured. Simple post-processing methods such as temperature scaling can bring a model's stated confidence back in line with its actual accuracy on that evaluation set, which makes the resulting threshold far more trustworthy than one set against raw, unadjusted scores.
Because production traffic drifts, the threshold is not a one-time decision. Re-run the same evaluation on a regular schedule and whenever the model, prompt, or upstream data source changes, and adjust the cutoff if accuracy at that confidence level has moved.
Govern the threshold like any other production control
A confidence threshold is a decision boundary with real consequences, which puts it in the same category as an approval limit or an access permission. It should have a named owner, live in a reviewed configuration rather than buried in code, and change through the same process as any other production control.
Log every threshold change alongside who approved it and why, next to the evaluation numbers that justified it. When an incident review asks why a case was auto-approved, the answer should be a specific, dated decision, not a guess about what the default happened to be at the time.