Rules
Part of Where analytics foundations work goes wrong: the starting point
How to Test Causation in an Analytics Foundations Case Study
Use an analytics foundations case study to test cause, confounders, and comparisons before citing a platform outcome in an internal budget argument.
An analytics foundations case study can describe a platform change followed by an improved outcome without establishing that the change caused it. This guide gives you a practical way to assess that causal claim: define the change and outcome, trace the proposed mechanism, map competing events, and examine the comparisons the case actually provides. The result is a defensible description of what the case supports—and what it cannot establish—before you use it in an internal budget argument.
State the causal claim precisely
Begin by translating the case’s conclusion into a sentence with three parts: what changed in the data foundation, what work or behavior was supposed to change as a result, and which outcome followed. If the case moves straight from “platform delivered” to “business result improved,” the mechanism connecting those events is missing. A causal conclusion requires more than a sequence in which one event happened after another.
Be specific about the intervention. “We modernized analytics” could mean a change to data pipelines, a shared model, a reporting process, or several things at once. If the case bundles them together, note that it cannot show which part of the package mattered. Likewise, name the outcome the case actually reports, rather than substituting a broader claim such as “better decisions.”
Case Study Numbers: What to Ask
What is stated
- Percentage improvement
- Measured how, which period
- Hours saved
- Whose estimate, before measured
- Faster reporting
- Produce or trust, both measured
- Adoption or usage
- Distinct people or logins
- Time to value
- From kickoff or first talk
- Reduced cost
- Net of platform and staff
What to ask
- Percentage improvement
- Hours saved
- Faster reporting
- Adoption or usage
- Time to value
- Reduced cost
The figure can help identify the outcome claim you are assessing. It does not establish why the outcome changed; the causal test is whether the rest of the account connects the foundation change to that result and rules out plausible alternatives.
Run a five-part attribution test
- Define the intervention and outcome.Write down what changed and what result is being attributed to it. If either is vague, the claim is not yet testable. If the case describes several foundation changes together, treat the bundle as the intervention rather than assigning the result to a single component the case does not distinguish.
- Write the proposed mechanism.Complete this chain in plain language: “The foundation change altered ___; that changed ___; which contributed to ___.” The missing middle should describe an observable work practice or decision, not merely repeat “better data.” If the case does not explain how the change could produce the result, it offers an outcome claim without a causal account.
- Reconstruct the sequence of events.Put the foundation work, the relevant changes in work or decision-making, and the outcome in chronological order. Mark concurrent events mentioned even briefly: a new leader, hiring, reorganization, product launch, pricing change, or cost cutting. A company that rebuilds its data foundation is often a company in motion; the question is whether another change could explain the result as well.
- Identify the comparison.Ask what the case compares with what. It may describe a before-and-after result, compare teams, or provide no meaningful comparison at all. A case cannot observe the same company in the same period both making and not making the change. If there is a comparison team, ask how it was selected and whether its circumstances differed in ways that could also affect the outcome.
- Test the strongest rival explanation.Choose the concurrent event most capable of producing the reported result. Then ask what pattern the platform explanation predicts and what pattern the rival explanation predicts. Does the case provide evidence that distinguishes them, or does it report only a combined company-wide result? If the account is silent, record the alternative as unresolved rather than assuming it has been ruled out.
This procedure is not a request to reject every case without a perfect comparison. It is a way to separate what the case observed from the reason its author gives for the observation. The conclusion should become more cautious as the mechanism, chronology, or comparison becomes less informative.
Separate a competing cause from part of the mechanism
A concurrent change is not automatically a confounder. It may be an independent explanation for the outcome, or it may be part of the route through which the foundation change had an effect. For example, a new work process introduced because of the data change could be a mechanism; a reorganization that would have happened regardless could be a competing explanation. The distinction depends on how the events are connected, not just on whether they happened at the same time.
Ask whether the event could plausibly have happened without the foundation project and whether it could independently have changed the outcome. If both answers are yes, it deserves scrutiny as a rival cause. If the case says the event followed from the project, look for evidence supporting that relationship instead of treating the author's description as proof.
Leadership changes need particular care. A new executive may sponsor the platform work while also changing priorities, staffing, or decision routines. The case then needs to distinguish the platform’s contribution from the leadership change; crediting the platform alone does not do that. Symptoms such as unreliable reports or competing spreadsheets may resemble issues described in business intelligence examples or analytics foundations mistakes, but resemblance to a problem does not establish the cause of another organization’s outcome.
Judge the comparison, not just the timeline
A before-and-after account shows that an outcome followed an intervention; it does not by itself show what would have happened without that intervention. An existing trend, a leadership change, or another event in the same period may explain the movement. The original case’s comparison is therefore central to the causal claim, not a technical detail to skim past.
If the case compares teams or groups, ask whether the groups were already different in readiness, leadership, or the problems they faced. A group that adopted a new foundation earlier may also have been more prepared to improve. If the case tracks an outcome over time, ask whether the account gives enough context to tell whether the change began with the intervention or was already underway. A detailed chart does not resolve competing explanations by itself.
When no comparison can distinguish the platform from a rival cause, use precise language: the case reports an outcome after the foundation change, but does not isolate the change’s causal effect. That does not mean the outcome is false. It means the evidence cannot support a stronger attribution.
Use the case in a budget argument
An external case can show that a proposed mechanism is plausible, describe a sequence another team tried, or flag an implementation risk. It cannot establish that your organization will achieve the same result. Keep the observed outcome separate from the author’s explanation, and name the strongest competing event the account does not rule out.
A useful budget argument says what the case observed, how well its evidence supports the causal explanation, and which features of your own situation are different. If the case describes a work practice that your team does not have, its mechanism may not transfer. If it describes a relevant problem and a plausible way the foundation change addressed it, that is a reason to investigate the approach—not a forecast of your result.
Avoid turning a precedent into a promised outcome. A case with a plausible chain and a useful comparison may support a more confident claim than one with only a before-and-after story, but neither substitutes for evidence about your own project. The point is to state the strength and limits of the evidence accurately, not to treat uncertainty as proof that the investment will fail.
Questions the causal review should answer
Can one case study prove that a foundation change caused an outcome?
A case can make a causal explanation more or less plausible. A simple account of a change followed by an improvement does not establish the effect when it cannot distinguish the intervention from other changes or an existing trend.
Is a concurrent leadership change always a confounder?
No. It may be an independent rival explanation, or it may be part of the proposed pathway. The case needs to explain the relationship and provide evidence that supports it; timing alone does not settle the question.
What if the case offers no comparison?
You can still use its account of the problem or work sequence to assess whether a mechanism is relevant. State that the case does not isolate the platform’s causal effect, and do not present its reported outcome as a prediction for your organization.
