Data analysis checklist: 12 points to review in 2027

This is a reviewer's checklist. It is what you run over somebody else's analysis before your name goes near it, or what you hand to a colleague and ask them to run over yours.

Reviewing analysis is not reading the code. Almost every damaging error is a correct query answering a question nobody meant to ask, and code review will not find it. These twelve points look for that class of problem, and they can be done in about twenty minutes without opening the query at all.

What to take away

• Review the framing, not the syntax. Wrong code fails loudly; a wrong population publishes cleanly.
• Three of these catch most of it: the population rule, the completeness of the last period, and the caveat that would flip the conclusion.
• A review is a conversation, not a verdict. Every point below is phrased as a question you can ask out loud.

Before you look at any numbers

1. Is the question in the document the question that was asked? Find the original request and compare. Drift between the two is the single most common defect, and it is invisible once the analysis is written up, because everything inside is internally consistent.

2. Is there a written inclusion rule for the population? Not a description, a rule: the condition that decides whether a row is in. Ask specifically what happens to trials, internal accounts, test data, and anything canceled or refunded.

3. Is the window stated with a timezone, and is the last period complete? A trailing period still receiving data reads as a decline, and this error has a strong tendency to appear one day before a board meeting.

4. Is the unit of analysis the same everywhere? Users, accounts, sessions, orders. A document that starts in one and finishes in another produces a conclusion that is true of neither.

Once you have the numbers in front of you

5. Does the denominator move on its own? A rate has two moving parts, so ask what would happen to the headline figure if the denominator fell and the numerator stayed flat. If the answer is "it improves," the rate needs its two components shown beside it.

6. Does anything here reconcile to a figure the business already trusts? One tie out to finance, to a source system total, or to a well established report is worth more than any amount of internal consistency. If nothing ties out, ask why not.

7. Was this found by exploring? A result discovered after slicing the data many ways is a hypothesis. It should be labeled as one, and ideally checked against a period the exploration did not touch.

8. Is there a comparison group, and was it defined before the result was known? Comparisons chosen after seeing the answer are not comparisons. Ask when the groups were fixed, and accept a straight answer.

9. Would the conclusion survive removing the largest contributor? Ask the analyst to run it once without their biggest customer, account, or day. If the finding disappears, the finding was that entity, which is still worth reporting but is a different statement.

Before it leaves the building

10. Are observation, interpretation, and recommendation visually separated? Readers quote interpretation as fact. If the three are woven into the same paragraphs, they will be flattened into one claim by the second forward.

11. Can this be rerun? Reproducibility is the whole of this check: ask for the link to the query or notebook, then ask whether it runs from a clean session. Work that lives only in a scrolled back terminal cannot be defended in six weeks when somebody produces a different number.

12. Is the fatal caveat at the top? Every analysis has one assumption that, if wrong, reverses the conclusion. It belongs above the fold, in the same font as everything else. An appendix is where caveats go to be ignored.

What this checklist deliberately does not do

It does not check arithmetic, and it does not check the SQL. Both matter, and both are better caught by the person who wrote them rerunning their own work. A second reader adds value in exactly the places where familiarity is a disadvantage: what was assumed, what was excluded, and what the words in the summary actually claim.

It also does not check formatting. Do not spend a review on chart choices when the population rule is unwritten.

Running the review without a fight

Ask, do not assert. "What happens to trials here" invites an answer; "you forgot to exclude trials" invites defense, and is often wrong.

Time box it. Twenty minutes, twelve questions, and a written note of which ones could not be answered. The unanswerable ones are the output, and they are usually two or three.

Agree in advance what a failed review means. In most teams it should mean the caveat gets stated more prominently, not that the work gets blocked. Reviews that can only ever stop things stop being requested.

Related reading on this site

The full sequence a request travels, from vague ask to delivered answer, is in the analysis process. For what makes a number trustworthy at all, see analytics foundations. Points five and nine are properties of the measure itself, covered in choosing a metric shape. Who owns a definition and how it changes is the subject of data governance, and if the answer is heading for a screen rather than a document, see dashboard design.

Common questions

Who should do the review?

Someone who knows the data but was not involved in this piece of work. Domain knowledge without ownership is the useful combination.

Is this overkill for a quick answer?

For a number going into a message to one person, run points one, two, and three. For anything that will be forwarded, run all twelve. The distinction is whether it will outlive the conversation.

What if the analyst cannot answer several points?

That is a normal and useful outcome. Write down which ones, and let the reader see the list. Unanswered questions attached to a number are far safer than a number that looks finished.

Should this be a form?

Keep it as questions. A form gets ticked; questions get asked. The moment it becomes a checkbox exercise it stops catching anything.