The Navigator's Logbook

Opening Scene

After a voyage, a navigator’s logbook doesn’t just say “it went pretty well.” It records precisely how far off each estimated position was from the ship’s actual, confirmed position — in specific, comparable units, entry after entry. That precise, comparable record is exactly what a forecasting metric provides: a rigorous way to say exactly how good, or bad, a forecast actually was.

In Plain English

Forecast evaluation metrics quantify the gap between predicted and actual values, in ways that can be compared fairly across methods and sequences. MAPE (Mean Absolute Percentage Error) expresses error as a percentage, making it comparable across sequences with different scales. RMSE (Root Mean Squared Error) penalizes large errors more heavily than small ones. Neither metric is universally “correct” — the right choice depends genuinely on what kind of error actually matters most for a specific decision.

The Old Way

Before formal forecasting metrics existed, accuracy was judged far less precisely:

A rough, informal sense of “the forecast was close enough,” without a specific, comparable number attached.
Simple absolute differences reported without any standardized scale, making comparison across different sequences genuinely difficult.
Success judged anecdotally, based on a handful of memorable good or bad predictions, rather than a systematic accounting across many predictions.

Formal metrics turned vague impressions of accuracy into precise, comparable, and genuinely testable numbers.

What’s Changing (and Why AI Is the Reason)

A now-standard toolkit of metrics — MAPE, RMSE, MAE, and others — lets practitioners compare forecasting methods rigorously across the many candidates covered throughout this series, using the backtesting framework from Article 16.
Different metrics genuinely capture different priorities: MAPE is intuitive but struggles when actual values are near zero; RMSE penalizes large misses more heavily, which matters when big errors are disproportionately costly.
Choosing the right metric deliberately, matched to the actual business consequence of different kinds of errors, has become recognized as a genuine design decision, not an afterthought to compute after the fact.

The Metaphor, Fully Extended

The Voyage	Evaluation Metric Concept
A logbook recording precisely how far off each estimate was	A metric quantifying the gap between forecast and actual
Recording error as a percentage of the actual distance traveled	MAPE, expressing error as a percentage
Weighting a dangerously large navigational miss more heavily than a tiny one	RMSE, penalizing large errors more heavily than small ones
Choosing which kind of error matters most for a specific voyage	Choosing the metric that matches a specific decision’s real stakes

For Beginners: What to Actually Do

Compute at least two different metrics — MAPE and RMSE — on the same backtest results, and notice where they actually disagree about which method is “better.”
Learn MAPE’s specific weakness with values near zero before relying on it for a series that includes small or zero values.
Practice explicitly connecting a metric choice to what kind of error actually matters most for a real decision, not just picking the most familiar metric.

For Practitioners and Leaders: The Deeper Layer

Choose evaluation metrics deliberately, based on the real business cost of different kinds of forecast error, rather than defaulting to whichever metric is most commonly cited.
Report multiple metrics together when comparing methods, since a single metric can obscure genuinely important tradeoffs.
Revisit metric choice periodically as business priorities shift — the “right” metric can genuinely change even when the underlying forecasting problem doesn’t.

Quick Recap

Forecast evaluation metrics quantify the gap between predicted and actual values in comparable, standardized terms.
MAPE and RMSE are common but genuinely different metrics, each with real strengths and weaknesses.
The right metric should be matched deliberately to the actual business cost of different kinds of error.
Reporting multiple metrics together often reveals tradeoffs a single number would hide.

Where This Fits in the Series

Article 17 covered scoring a forecast precisely. Article 18 covers a technique for improving on any single method’s score by combining several together.