When Two Experts Disagree

Opening Scene

When two independent appraisers examine the same object and reach genuinely different conclusions, that disagreement itself is valuable information — it signals the object’s authenticity is genuinely uncertain, worth further investigation, rather than something to confidently assert one way or the other. A single language model’s single answer can’t reveal this same signal, but asking it the same question multiple times, independently, genuinely can.

In Plain English

Self-consistency checking generates a model’s answer to the same question multiple times, independently, and checks whether those answers actually agree. A high rate of disagreement across repeated attempts is a meaningful signal that the model’s confidence in any single answer is genuinely unwarranted, even if any one individual response looked perfectly plausible and confident on its own. This technique surfaces uncertainty a single answer alone would hide entirely.

The Old Way

Before self-consistency checking was widely applied, hallucination detection often relied on a single generated answer alone:

Evaluating a model’s answer often meant looking at a single generated response, with no way to check whether that response was genuinely stable across independent attempts.
There wasn’t yet a well-established practice of using disagreement across repeated generations as a genuine, checkable uncertainty signal.
A confidently phrased single answer provided no visibility into how consistently the model would answer the same question again.

Self-consistency checking emerged specifically as a practical technique for surfacing this hidden instability, giving genuine signal where a single generated answer alone provided none.

What’s Changing (and Why AI Is the Reason)

Self-consistency checking is increasingly used as a practical mitigation technique, generating multiple independent answers and flagging disagreement as a genuine uncertainty signal.
This connects directly to the calibrated uncertainty expression covered in Article 8, since consistent disagreement is one concrete signal that should lower a system’s expressed confidence.
This connects directly to the confabulation risk covered in Article 5, since self-consistency checks are specifically effective at catching reasoning-based errors that fabrication-focused grounding alone might miss.

The Metaphor, Fully Extended

The Antiques Appraiser	Self-Consistency Concept
Two independent appraisers reaching different conclusions	A model generating different answers across independent attempts
Disagreement itself being valuable, genuine information	Disagreement itself being a valuable, genuine uncertainty signal
Signaling that authenticity is genuinely uncertain	Signaling that the model’s answer confidence is genuinely unwarranted
A single appraisal alone not revealing this signal	A single generated answer alone not revealing this signal

For Beginners: What to Actually Do

Practice generating a model’s answer to the same question multiple times independently, checking whether the answers genuinely agree.
Learn to treat high disagreement across repeated attempts as a meaningful signal of genuine uncertainty.
Get comfortable using this technique specifically for high-stakes questions where getting the answer wrong would have real consequences.

For Practitioners and Leaders: The Deeper Layer

Build self-consistency checking into evaluation and, where cost allows, production pipelines for high-stakes questions.
Connect self-consistency signals directly to the calibrated uncertainty expression covered in Article 8.
Recognize self-consistency checking as specifically effective against confabulation, covered in Article 5, complementing grounding’s effectiveness against fabrication.

Quick Recap

Self-consistency checking generates a model’s answer multiple times independently and checks for agreement.
High disagreement is a meaningful signal that confidence in any single answer is genuinely unwarranted.
This surfaces uncertainty a single generated answer alone would hide.
This technique is specifically effective against confabulation, complementing grounding’s effectiveness against fabrication.

Where This Fits in the Series

Article 7 covered self-consistency as a mitigation technique. Article 8 turns to the appraiser who genuinely says “I’m not sure”: calibrated uncertainty expression.