Costs
Part of Grading a forecasting cycle honestly before anyone trusts it
Forecasting metrics worth tracking, and the ones to ignore
Forecasting error measures compared: what percentage error hides, why scaled measures travel, and why bias and interval coverage also need reporting.
Every forecast error measure is a choice about which mistakes you mind most. None of them is neutral, and picking one without knowing what it hides is how teams end up optimizing for the wrong failure.
This page is about that choice. It covers what each common measure rewards, where it breaks, and which pairing gives an honest picture.
What to take away
- Error measures disagree with each other on purpose. If two of them rank your models differently, that is information about your series, not a bug.
- Percentage measures are the default almost everywhere and are the worst choice for series that approach zero or that swing across it.
- Accuracy without bias is half a picture. A forecast can be accurate on average and wrong in the same direction every month.
The measures, and what each one is for
| Measure | What it rewards | Where it misleads |
|---|---|---|
| Mean error, or bias | Being right on average | Cancels out large errors in both directions |
| Mean absolute error | Small errors in the units of the series | Not comparable across series of different size |
| Root mean squared error | Avoiding large misses | Dominated by one bad period |
| Mean absolute percentage error | Comparability across series | Explodes near zero, and penalizes over-forecasts differently from under-forecasts |
| Mean absolute scaled error | Comparability without the zero problem | Needs a baseline, and readers have to be taught it |
| Interval coverage | Honest uncertainty | Says nothing about the point forecast |
Two of these deserve more than a row.
Why percentage error is the wrong default
Mean absolute percentage error divides by the actual value, so it becomes unstable as the actual approaches zero and is undefined at zero. That alone rules it out for intermittent demand, new products, and anything counted in small numbers.
The second problem is asymmetry. Under-forecasting is bounded, because you can be wrong by at most one hundred percent of the actual. Over-forecasting is unbounded. A method compared on this measure is therefore quietly rewarded for forecasting low, and over enough cycles a team will drift toward exactly that.
The usual replacement is a scaled measure. Mean absolute scaled error divides your error by the error a naive forecast would have made on the same series. Below one beats the baseline; above one does not.
It handles zeros, compares across series, and makes the baseline comparison unavoidable. That is its real virtue.
Bias is the measure nobody publishes
Absolute measures throw away the sign, which is the whole point of them and also their blind spot. A forecast that is ten percent high in every single period can look respectable on any absolute measure while being systematically wrong.
Track the signed mean error alongside whichever absolute measure you use, and watch its run length: how many consecutive periods it has kept the same sign. Persistent forecast bias is nearly always organizational rather than statistical. Sales forecasts run low where the forecast becomes a target. Demand forecasts run high where a stockout is punished harder than an overstock.
Measuring the interval, not just the point
If you publish a range, the range needs its own measure. Coverage is the simple one: over many periods, what share of actuals fell inside the stated interval. A stated eighty percent interval that contains the actual half the time is not conservative, it is wrong, and nobody will notice unless somebody counts.
The general property is calibration: forecasts assigned a given probability should come true at about that rate. Where you forecast events rather than levels, the Brier score gives a single number combining calibration and sharpness, and it has the useful property of punishing confident wrong calls harder than hedged ones.
Aggregating across series and horizons
Two aggregation mistakes are common enough to name.
Averaging percentage errors across series of very different sizes gives the small series equal weight with the large ones. If the decision is about money, weight by value, and say that you have.
Averaging across horizons hides the shape of the degradation. A one-week-ahead error and a six-month-ahead error mean different things, and a single blended figure lets a model that is terrible at long horizons look adequate. Report the error curve by horizon, even if only annually.
A workable default set
For most internal forecasting, publish four things per series: a scaled accuracy measure against a stated naive baseline, the signed bias with its current run length, interval coverage, and the count of judgment overrides applied. Those four together are hard to game, because improving one at the expense of another shows up immediately in the next.
Review them on a fixed cycle rather than when a forecast misses. A measure only inspected after a bad period is an argument, not a measurement.
Related terms worth keeping straight
Accuracy is how close you were. Bias is whether you were consistently wrong in one direction. Precision, in this setting, is how narrow your intervals were. A forecast can be precise and inaccurate, which is the most damaging combination because it looks confident.
Related reading on this site
The method these measures grade is in how forecasts are built and reviewed, and the pre-publication sequence is in twelve checks before a forecast goes out. For metric design in general, including the guardrail pairing this page applies to bias, see metric shapes and their traps.
The reporting habits that keep an error measure honest in a pack are in building a report worth keeping. The questions to ask when someone shows you an accuracy figure are in asking about a number in the room.
Common questions
Our stakeholders only understand percentage error. Do we have to change?
Keep it as a presentation layer if you must, and select models on a scaled measure underneath. What you cannot do is let a percentage measure decide between methods on series that go near zero.
What is a good accuracy figure?
There is no answer to that question in the abstract, and anyone offering one is selling something. Accuracy is only interpretable against the volatility of the specific series and against the naive baseline for that series.
How many periods before an accuracy number means anything?
Enough to see a full seasonal cycle plus a few periods, and enough that one unusual month cannot move it much. Below that you are reading noise, and you should say so beside the figure.
Should the forecaster choose the measure?
The measure should be agreed with whoever acts on the forecast, before the first one is published. Chosen afterwards, it will be the measure that makes the last quarter look best.
