Business intelligence case study: findings and lessons

Write ups of intelligence programs make a small number of recognizable claims. Hours were saved, adoption rose, the company got one version of the truth, decisions got faster, the investment paid back inside a stated period.

This page takes those claims one at a time and asks what would have to be true for each to mean what it appears to mean. No company is described here, and no result is reported. The subject is the arithmetic behind the sentences, so you can do this reading yourself on whatever document lands on your desk.

What to take away

• Every headline claim in this genre rests on a before measurement that was usually never taken.
• Adoption figures count logins far more often than they count decisions.
• The most informative sentence in any write up is the one describing what was tried first and abandoned. If it is absent, treat the rest as a brochure.

Hours saved

The construction is nearly always the same: a per person time estimate, multiplied by a headcount, multiplied by a frequency, annualized.

Three questions dismantle or confirm it.

• Where did the per person estimate come from? A recalled figure gathered after the project, from people who know the answer is supposed to be large, is not a measurement.
• Were the hours reallocated or removed? Time freed from assembling a report is only a saving if the person stopped working those hours or did something more valuable with them. Usually they absorbed other work, which is a real benefit but a different one.
• Does the figure net off the new work? Somebody now maintains models, definitions, and pipelines. If that time is not subtracted, the number is gross, not net.

An honest hours claim will state the before measurement, its date, and how it was taken. Very few do.

The four datasets above share almost every summary statistic and look nothing alike. Anscombe's quartet is the standard demonstration of why a headline figure in a write up tells you very little about the data underneath it.

Adoption

Adoption is the easiest number to produce and the least likely to mean anything, because the instrument sits inside the tool being praised.

Watch which unit is used. Accounts provisioned is a purchasing decision. Monthly logins measure curiosity. Weekly active builders is closer to something. Distinct people who changed a decision is what you actually want, and it is nearly never reported because it cannot be collected automatically.

Also ask what adoption is being compared to. A rollout that replaces an unpopular predecessor starts from a floor that flatters everything that follows.

One version of the truth

This claim usually means a technical fact: everything now reads from one modeled layer. That is a real achievement and it is worth reporting.

It is often presented as an organizational fact: disagreements about numbers stopped. Those are different things, and the second does not follow from the first. Teams with a shared modeled layer still argue, because the arguments were about which definition should win, and a shared layer does not settle that. It only makes the disagreement visible in one place instead of five.

When you read this claim, look for evidence of the organizational half: a named owner per metric, a change process, a decision about restating history. If those are absent, the claim describes plumbing.

Faster decisions

Almost always measured as time to produce a number rather than time to reach a decision. Those diverge sharply, because the slow part of most decisions is agreement, not data.

If a write up reports cycle time, ask which clock was running. From request to answer is a data team metric. From question to committed choice is a business metric. A program can improve the first considerably and leave the second untouched, and both outcomes are worth knowing about.

Payback and return

A payback figure requires a cost side and a benefit side, and the two are rarely built with the same rigor.

The cost side is under counted in predictable places, and the discipline that names them is total cost of ownership: the internal engineering time, the migration months when both systems ran, the training, the ongoing maintenance, and the analyst time spent on the project rather than on analysis. The benefit side usually rests on the hours saved figure discussed above.

You do not need to reject the number. You need to know which side was measured and which was estimated, because a precise cost against an estimated benefit produces a ratio with a false air of accuracy.

The compressed timeline

Case studies narrate a clean sequence, and the sequence is a reconstruction. Real programs stall, restart under a new sponsor, change scope twice, and spend months on something later dropped.

Two things get lost in compression. The dead ends, which are the most transferable content, and the elapsed time, which is what determines whether your organization can sustain the effort. A program described in five paragraphs may have taken three years and two sponsors.

What is systematically missing

Nobody writes up the program that was quietly wound down after eighteen months. Those are not rare. They simply produce no document, so the entire published record of this field is drawn from the successful tail.

The practical consequence is that you cannot estimate your odds from reading case studies, no matter how many you read. You can only learn mechanisms.

A reading procedure

1. Underline every number. Beside each, write measured or estimated.
2. Find the before state for each measured figure. Note its date.
3. List every other change mentioned anywhere in the document.
4. Find the sentence describing something that did not work. If there is none, downgrade the whole piece.
5. Write one sentence on what the document tells you about your own situation. If you cannot, it was entertainment.

That takes about fifteen minutes and it is the difference between citing a document and being persuaded by one.

Related reading on this site

The function these programs are trying to build is described in business intelligence. For the same skeptical reading applied to platform and data foundation write ups, see reading a platform case study. Several of the claims above are proposed as reasons to buy something: the counter method is in running your own tool evaluation. For judging a reported metric by its shape and denominator, see metric definitions and their traps, and for the organizational moves that these documents are often used to justify, see nine mistakes made around a data team.

Common questions

Should I baseline before our own project so we can report properly?

Yes, and do it even if you never publish. A measured before state is the only thing that lets you tell later whether the work helped.

A vendor gave us a study from a company like ours. Does similarity help?

It helps with the mechanism and not with the magnitude. Similar companies still differ in the ways that drive results: sponsorship, existing data quality, and how decisions get made.

Is it fair to ask a vendor for an unsuccessful engagement?

Yes, and the quality of the answer is informative. A candid account of one that went badly, with reasons, is a better signal than any success story.

What if leadership has already been convinced by one?

Do not attack the document. Ask what the before state was, and offer to measure your own for two weeks. That converts a debate about someone else's numbers into a decision about yours.