What Counts as a Label

Opening Scene

It sounds trivial: the assistant under the hoop marks a shot as a make or a miss. But watch closely for a full practice and it gets murkier. A shot that rims out and drops in a second late — make or miss? A shot that’s clearly good but gets tipped by a teammate before it lands — whose result is that? The assistant has to make a judgment call, over and over, and every one of those calls becomes part of the record the coach later trusts as fact.

That record is the label, and Article 1 treated it as a given. This article treats it as the thing it actually is: a judgment call, made by someone, using some standard, and it deserves exactly as much scrutiny as any other part of the data.

In Plain English

A label is the “correct answer” attached to a training example — the target a supervised model is trying to learn to predict. It sounds like a simple fact, but a label is really the output of a process: someone or something observed an outcome and recorded a judgment about it. That process has a standard, has edge cases, and can be applied inconsistently — which means the label itself carries all the same risk of error as any other measurement.

The Old Way

Before anyone worried about “label quality” as a formal concept, inconsistent labeling just quietly degraded whatever depended on it:

Two referees calling the same play differently — the “correct” outcome depends on who happened to be watching.
A teacher grading essays more strictly on a Monday than a Friday — the same work, a different label, depending on mood and fatigue.
A store marking returns as “defective” versus “customer changed mind” inconsistently — the label exists, but it’s not reliably measuring what it claims to measure.

Nobody set out to create bad labels in any of these cases. The inconsistency crept in because nobody had explicitly defined the standard being applied.

What’s Changing (and Why AI Is the Reason)

Labels increasingly come from automated systems instead of a person’s judgment call, which changes where the risk hides. An automated label is consistent by default, but it inherits whatever bias or blind spot is built into the system that generated it — consistency isn’t the same thing as correctness.
AI tooling can now audit large labeled datasets for internal inconsistency far faster than a human reviewer could. Flagging examples where similar inputs got contradictory labels used to require a painstaking manual review; it’s increasingly a fast, automatable check.
The definition of a “correct” label is under more pressure as models get deployed into higher-stakes decisions. A label that was good enough for an internal report can be genuinely inadequate once a model trained on it starts making decisions that affect people directly — this series returns to that stakes question directly in Article 19.

The Metaphor, Fully Extended

Basketball Drill	Label Quality Concept
The assistant marking each shot	The labeling process itself
A shot that rims out and drops late	An ambiguous case that tests the labeling standard
Two assistants marking the same shot differently	Inter-rater disagreement — inconsistent labeling
A written rule for what counts as a make	An explicit labeling standard or guideline
A tipped shot recorded under the wrong player’s name	A mislabeled example — attributed incorrectly
Reviewing a stack of marked shots for contradictions	Auditing a labeled dataset for consistency

For Beginners: What to Actually Do

Before trusting a labeled dataset, ask who or what created the labels, and what standard they were applying.
Look directly at a sample of the actual labeling decisions, especially the ambiguous-looking ones — they reveal the standard’s real edges far better than a written definition does.
Treat “the label says so” as a starting point, not a final fact, especially for anything genuinely judgment-based.

For Practitioners and Leaders: The Deeper Layer

Write down the labeling standard explicitly, including how edge cases should be handled — an unwritten standard is an inconsistent one, even with a single careful labeler.
When multiple people label data, measure their agreement with each other directly; low agreement is a real signal the standard itself needs work, not just the labelers.
Revisit labeling standards periodically as the underlying reality shifts — a definition of “defective” or “churned” that made sense two years ago may quietly no longer fit.

Quick Recap

A label is the output of a judgment process, not a raw fact — and that process deserves the same scrutiny as any other data source.
Inconsistent labeling standards quietly degrade everything trained on them, even when no one intended to create bad data.
Automated labeling is consistent but inherits the bias of whatever system generated it — consistency isn’t the same as correctness.
Auditing labels for internal consistency, and writing the standard down explicitly, are two of the highest-leverage habits in any supervised project.

Where This Fits in the Series

Article 2 covered learning with no label at all; this article looked hard at what a label actually is once you have one. Article 4 moves to the next foundational split — what kind of answer a label can even be, a score versus a category.