Where analytics foundations work goes wrong: the starting point

A number in a report is the last link in a chain: something happened, something recorded it, something moved and reshaped the record, someone wrote a counting rule, and someone put it on a page. Any link can be wrong while the others look fine.

So arguments about numbers rarely end by looking harder at the number itself.

This page is about the chain, and about the two questions that decide whether a number deserves to be acted on: what is it actually counting, and what could have made it wrong.

What to take away

Making a question answerable

"Is the product doing well" cannot be answered. Not because the data is inadequate, but because the question does not specify anything that could be counted.

A question becomes answerable when four things are pinned down.

The subject. What thing are we counting? Users, accounts, sessions, orders, workspaces, devices. These produce different numbers from the same data, and all four routinely get called users in the same meeting.

The population. Which of those subjects are in scope, and which are deliberately out. New ones only, paying ones only, one region, excluding internal accounts and test records.

The period. Over what stretch of time, using which calendar, and in whose time zone. A rolling window and a calendar month are different measurements and will never agree.

The comparison. A number alone means nothing. Compared to last period, to a different group, to what we expected, or to what happens when we do nothing.

Write those four down before touching any data. Most of the time this alone resolves the disagreement that prompted the question, because two people wanted two different numbers and neither had said so.

The map above is the standard example of a question made answerable by choosing the right subject and population. Counting deaths by district settled nothing; counting them by the water pump nearest each address settled it immediately.

The chain, and how each link breaks

When two numbers disagree, walk this list from the bottom. Most disagreements resolve at definition, and almost all of the rest resolve at collection.

Measurement, estimate, and proxy

These three get reported in the same typeface and they are not the same kind of thing.

A measurement is a direct record of the event you care about. An order was placed; you have the order.

An estimate is a calculation about a population you did not fully observe, usually built from a sample and scaled up. It has an error range whether or not anyone prints one.

A proxy is a measurement of something else that you believe moves with the thing you care about. Time on page as a proxy for interest. Support tickets as a proxy for confusion.

Proxies are legitimate and often unavoidable. The discipline is naming them as proxies every time, in writing, next to the number. A proxy used long enough stops being described as one. Then someone optimizes it directly.

At that point it stops tracking the thing it stood in for. That is the mechanism behind Goodhart's law, and it applies to any measure that becomes a target. That failure is very hard to see from inside the report.

Instrumentation decides everything downstream

No amount of modeling recovers an event that was never emitted. This makes instrumentation the part of the practice that decides the most and receives the least attention.

A few habits that prevent most of the pain:

That last one deserves writing down permanently. Every collection method has gaps, and a gap you have documented is a limitation, while a gap you have not is a wrong number.

Denominators are where accuracy goes to die

Most disputed metrics are fractions, and the numerator is rarely the problem. A rate is only as meaningful as the population it is divided by, and that population is the part nobody writes down.

Ask three questions of any rate. What is in the denominator exactly? Is every subject in it actually eligible for the thing in the numerator? And do the numerator and denominator cover the same period, or has one been measured over a window the other has not?

The classic error is a rate whose denominator includes subjects that could never have produced the numerator event. It makes the rate look low and it moves whenever the ineligible population grows, which produces a metric that responds to something nobody intended to measure.

Report the numerator and denominator alongside the rate. It costs one line and prevents an entire category of argument.

What a number will not tell you

Analytics tells you what happened, to whom, and how it changed. It does not tell you why, and it does not tell you what would have happened otherwise.

Both of those are answered by different work. Why is answered by talking to people, reading their support tickets, and watching them use the thing. What would have happened otherwise is answered by holding out a comparable group, which requires planning before the change rather than analysis after it.

Presenting a correlation as a cause is the most common way a technically correct report produces a bad decision. The defense is a sentence in the report saying what else could explain this, written by the person who built it, before anyone asks.

A realistic starting sequence

For a team with little in place, in this order:

Teams reliably do this backwards, starting at step seven, and end up with a good-looking page built on a number nobody can trace.

Common questions

Two teams have different numbers for the same metric. Where do we start?

Compare definitions before data. Subject, population, period, time zone, exclusions. If those match, compare the source tables. Recomputing without doing this first tends to produce a third number.

How accurate does a number need to be?

Accurate enough that a plausible error would not change the decision. That standard is answerable, unlike "as accurate as possible," and it usually reveals that a lot of precision was being bought for nothing.

Should we track everything now in case we need it later?

Record events for actions that are meaningful in your product, and be deliberate about it. Recording indiscriminately produces volume nobody has documented, which is not the same as having the data, and it carries real obligations for anything about identifiable people.

Is a small sample worth anything?

Yes, for direction, and it should be labeled as such. The mistake is not using small numbers; it is reporting them with the same confidence as large ones, and then explaining a movement that was noise.