Analytics foundations case study: evidence and results

Somebody is going to send you a case study about a data platform: a company rebuilt its foundations, and something good happened afterwards. It will be well written, it will contain a number, and it will be used in a meeting to argue for a budget.

This page will not tell you about any particular one. It is about the genre: how these documents are built, what they are structurally unable to show, and which sentences in them are load bearing.

What to take away

• A case study reports one outcome from one attempt. The comparison it needs, the same company not doing the thing, does not exist and cannot be supplied.
• The strongest evidence in the document is usually the description of the problem, not the description of the result.
• Ask who chose to write it up. That single question explains most of what is missing.

Who writes these, and why that shapes them

Almost every published analytics case study has one of three authors, and each has a predictable bias.

A vendor or consultancy. The document exists to sell. It is not therefore false, but its selection is not random: the engagements that went badly are not written up, and the ones written up are chosen for the size of the reported change.

The team that did the work. Written after a project they defended internally, often near a promotion cycle or a budget round. Honest people still narrate their own decisions as more deliberate than they were.

A journalist or researcher. Less incentive to flatter, but usually further from the data, and dependent on the same insiders for every fact in the piece.

None of these is disqualifying. All of them mean the document is a sample of one, drawn non randomly, described by an interested party. Cases are published because they went well, which is selection bias applied before you read a word of the arithmetic.

The claim that carries the weight

Nearly every case study in this genre reduces to one sentence: we changed the data foundation, and an outcome improved. The entire argument lives in the word "and."

To move from "and" to "because," you would need something like a control group: a version of the same company over the same period that did not make the change. That does not exist. The case study cannot supply it, and neither can a better written case study. This is a limit of the form, not a flaw in a particular example.

That does not make the document useless. It makes it a source of hypotheses and mechanisms rather than a source of expected results.

The things that changed at the same time

A company that rebuilds its data foundation is a company in motion. In the same period it is likely to have hired, reorganized, changed leadership, launched something, changed pricing, or cut costs. Any of those could move the outcome the case study attributes to the platform.

When you read one, list every change mentioned anywhere in the document, including the ones mentioned in passing as context. Then ask which of them, on its own, could plausibly produce the reported result. If two or more could, the attribution is a choice the author made, not a finding.

Pay particular attention to a new leader arriving. A data platform project and a new executive frequently appear together, and the executive is the more common explanation for a change in how decisions get made.

Numbers inside a case study, read closely

What is stated	What to ask
A percentage improvement	Improvement in what, measured how, and compared to which period
Hours saved	Counted from whose estimate, and did anyone measure the before
Faster reporting	Faster to produce or faster to trust, and were both measured the same way
Adoption or usage	Distinct people acting on it, or logins, which are not the same thing
A time to value figure	Starting from the kickoff, or from the first conversation two years earlier
Reduced cost	Net of the new platform, the migration, and the people who now run it

The most common construction is a before figure that was never measured. Someone recalls that reporting "used to take about a week," the after figure is measured precisely, and the comparison is between a memory and an instrument. That gap is not deception. It is what happens when nobody baselines the old process because nobody expected to have to defend it later.

What the description of the problem is worth

Here is the part worth reading carefully. Case studies are unreliable about results and often very good about symptoms. The account of what went wrong before, the arguments in meetings, the spreadsheet everybody secretly used, the report nobody trusted, is drawn from lived experience and has no incentive to be flattering.

Read the problem section as the useful half. If it describes your situation closely, the mechanism they used is worth understanding. If the problem described is not your problem, the result is irrelevant to you no matter how large it is.

Questions to send the author

If you can reach whoever wrote it, four questions get more than the document contains.

• What else changed in that period?
• How was the before state measured, and when?
• What did you try first that did not work?
• What would you not do again?

An author who answers the last two candidly has given you something the published version could not. An author who cannot name a single thing that failed has written marketing, and you should read the rest accordingly.

Using one honestly in an internal argument

You can cite a case study for the shape of a problem, the sequence someone followed, or a risk they hit. You cannot cite it for what will happen to you. If a proposal rests on a number lifted from someone else's write up, the proposal has no evidence, it has a precedent.

The stronger move is to run the smallest version of the change yourself and measure your own before state first. One week of baselining your current process produces evidence that applies to you, which no external document ever will.

Related reading on this site

The chain that determines whether any reported number means what it says is set out in analytics foundations. For the questions to put to the people who hold the answers, see the interview script for data work. For judging a stated metric by its shape and denominator, see choosing and defining a measure. For why attribution without a comparison group fails, see the analysis sequence, and for the function these projects are usually trying to fix, see business intelligence.

Common questions

Is there any case study worth trusting?

Trust the mechanism, not the magnitude. A clear account of what someone built and in what order transfers. The number attached to it does not.

What about a study with several companies in it?

Better, and still selected. Ask how the companies were chosen and whether any that abandoned the effort were included. Almost none are.

Should we publish our own?

Only if you measured the before state, and only if you are willing to name what did not work. Without both, you are adding another document to the pile this page is about.

How do I push back without sounding obstructive?

Ask for the baseline. It is a neutral, specific question, and the answer tells you and everyone else in the room how much weight the document can carry.