Data analysis mistakes that can derail your plans

The errors that cost an analyst credibility are rarely statistical. They are errors of process: how the work was scoped, where it was done, how the answer was handed over, and what happened when it turned out to be wrong.

Nine of them are below. All are survivable, all are common, and every one has a small habit that prevents it.

What to take away

• Most analytical damage happens before the first query and after the last one.
• A number without its population, window, and unit attached will be misused within a week. Attaching them is a five second habit.
• The fastest way to lose trust is a quiet correction. The fastest way to build it is a loud one.

1. Answering exactly what was asked

The request is a compressed guess at what would help. Delivering it literally is technically responsive and frequently useless, because the person did not know what was available when they wrote it.

Instead: send back the question in your own words before you start, with the population and window spelled out, and get one line of confirmation.

2. Agreeing a deadline without agreeing a precision

"By Thursday" is half a specification. A directional read and a defensible figure are different amounts of work, and the requester will assume whichever suits the use they had in mind.

Instead: settle both at once. Ask how wrong it can be before the decision changes, and say what you can deliver at that level by that date.

3. Building on a number somebody pasted into a message

A figure arrives in a chat thread, gets used as an input, and turns out to have come from a filtered view of a dashboard that excluded a region.

Instead: trace any input back to a source you can query yourself before it becomes load bearing, and say in the write up which inputs you verified and which you took on trust.

4. Working somewhere you cannot rerun from

Ad hoc queries in a console, transformations done by hand in a spreadsheet, a filter applied and then forgotten. It works, the answer is right, and six weeks later nobody can reproduce it, including you.

Instead: do the work in a file that runs start to finish from nothing, even for small questions. The overhead is minutes and it is what lets you defend the number later.

5. Correcting a published figure quietly

A number went out, it was wrong, and the corrected version simply replaces it. This feels tidy and it is the single most reliable way to make people distrust everything you produce, because someone will eventually notice. Statistical offices treat correction as routine and publish how revisions and corrections are handled, which is a better model than silence.

Instead: announce the change in the same channel the original went out in, say what was wrong, what the new figure is, and what the earlier one would have changed. Then keep both visible.

6. Sending a number naked

A figure with no unit, no population, and no window will be recombined with other figures by people who assume they match. It ends up in a slide beside a number counted differently.

Instead: every number you hand over travels with one line: what is counted, who is included, over what period, in what timezone.

7. Leading with method

Analysts present in the order they worked: sources, joins, definitions, then finally the answer. Readers stop paying attention well before the answer and then ask a question you already covered.

Instead: answer, then confidence, then method. Anyone who wants the method will read on, and they are exactly the audience the method section is for.

8. Filing the caveat at the end

The assumption that would flip the conclusion sits in a closing section headed limitations, which nobody reads and which does not survive being forwarded.

Instead: put the one caveat that matters into the same paragraph as the answer. Keep the rest at the end if you like, but the load bearing one belongs beside the claim it qualifies.

9. Explaining a movement you have not shown is real

A number moves, somebody asks why, and an explanation gets produced. Often the movement is inside the range the number wanders through anyway.

Two things moving together in one period is the weakest evidence there is, and correlation does not imply causation is worth restating precisely because it gets nodded at and then ignored.

Instead: before writing a cause, look at how much the measure has moved week to week over the past year with nothing happening. If this week's change fits comfortably inside that, the honest answer is that nothing has happened yet.

The habit underneath most of these

Six of the nine are fixed by writing one extra sentence: the refined question, the precision, the caveat, the definition line, the correction notice. Analysts skip them because each feels like overhead on a small task, and the cost only shows up later, in a meeting, with an audience.

The other three are fixed by choosing where the work lives. Anything done in a place that cannot be rerun will eventually have to be redone under pressure.

Related reading on this site

The end to end sequence these mistakes interrupt is in the analysis process. Points one to three are caught by a second reader working through the twelve point review. For the conversations that produce a well formed request in the first place, see the questions to ask before you build. For why the request queue keeps producing badly framed asks, see business intelligence, and for the difference between a one off answer and something recurring, see reporting.

Common questions

How do I push back on a request without seeming difficult?

Do not push back. Restate it and ask for confirmation. Most badly framed requests improve on their own the moment the requester reads them written out.

What if I find an error in something I published months ago?

Announce it anyway. The cost of a late correction is much lower than the cost of someone else finding it, and it is lower still than the cost of the wrong number continuing to circulate.

Is it ever fine to deliver a number without the definition line?

Only if you are certain it will not be forwarded, and you will be wrong about that regularly. Write the line.

My stakeholders want a cause every time a number moves.

Give them the movement, the normal range, and a clear statement of which one this is. Doing that consistently trains people to ask the better question, and it takes a couple of months.