city street, woman, motion scene, business style, asian culture, determined, photography, blurred, smiling, asian city, busy street, focus area, daylight, outdoor, modern city, long sleeve shirt, person focused on object, city interaction, relaxed atmosphere, german tourist. 6 things nobody tells you about data analysis checklist
Photo by exxposures on Pixabay

Rules

Part of Data analysis in six steps, from the request to the answer with limits

6 things nobody tells you about data analysis checklist

Data analysis checklist in twelve points, grouped by when to run them: before you look at numbers, once you have them, and before the work leaves.

Analysis Desk · · 6 min read

What to take away

This is a reviewer's checklist. It is what you run over somebody else's analysis before your name goes near it, or what you hand to a colleague and ask them to run over yours.

Reviewing analysis is not reading the code. Almost every damaging error is a correct query answering a question nobody meant to ask, and code review will not find it. The twelve points below look for that class of problem. They take about twenty minutes and do not require opening the query.

The six things nobody tells you about a data analysis checklist:

  • The expensive defect is usually a question mismatch, not a broken query. Wrong code fails loudly. A wrong population publishes cleanly.
  • The population rule must be written as a condition, not described. Ask what happens to trials, internal accounts, test data, canceled orders, and refunds.
  • A trailing period that is still receiving data reads as a decline. That error tends to appear one day before a board meeting.
  • A rate improves when its denominator shrinks and its numerator stays flat. Show both components beside the rate.
  • A comparison group chosen after the result is known is not a comparison. Ask when the groups were fixed.
  • The one assumption that could reverse the conclusion belongs above the fold, in the same font as the rest.

Three items catch most of it: the population rule, the completeness of the last period, and the caveat that would flip the conclusion. A review is a conversation, not a verdict. Every point below is phrased as a question you can ask out loud.

Before you look at any numbers: question drift, population, window, unit

1. Is the question in the document the question that was asked? Find the original request and compare. Drift between the two is the most common defect. It is invisible once the analysis is written up, because everything inside is internally consistent.

Before you look at numbers

  • Question in document matches original request
  • Written inclusion rule for the population
  • Window stated with timezone, last period complete
  • Same unit of analysis throughout
2. Is there a written inclusion rule for the population?
Not a description, a rule: the condition that decides whether a row is in. Ask what happens to trials, internal accounts, test data, and anything canceled or refunded.
3. Is the window stated with a timezone, and is the last period complete?
A trailing period still receiving data reads as a decline. This error tends to appear one day before a board meeting.
4. Is the unit of analysis the same everywhere?
Users, accounts, sessions, orders. A document that starts in one unit and finishes in another produces a conclusion true of neither.

Once you have the numbers in front of you: denominators, tie-outs, exploration, comparisons, largest contributors

5. Does the denominator move on its own? A rate has two moving parts. Ask what would happen to the headline figure if the denominator fell and the numerator stayed flat. If the answer is 'it improves,' the rate needs its two components shown beside it.

Checks once numbers are in

  • Does the denominator move on its own?
  • Does anything reconcile to a trusted figure?
  • Was this found by exploring?
  • Is there a pre-defined comparison group?
  • Would it survive removing the largest contributor?
6. Does anything here reconcile to a figure the business already trusts?
One tie out to finance, to a source system total, or to a well established report is worth more than any amount of internal consistency. Named examples include the finance close in QuickBooks, a Salesforce opportunity total, or a Stripe payout report. If nothing ties out, ask why not.
7. Was this found by exploring?
A result discovered after slicing the data many ways is a hypothesis. Label it as one, and check it against a period the exploration did not touch.
8. Is there a comparison group, and was it defined before the result was known?
Comparisons chosen after seeing the answer are not comparisons. Ask when the groups were fixed, and accept a straight answer.
9. Would the conclusion survive removing the largest contributor?
Ask the analyst to run it once without their biggest customer, account, or day. If the finding disappears, the finding was that entity. That is still worth reporting, but it is a different statement.

Before it leaves the building: separated claims, reruns, and the fatal caveat

10. Are observation, interpretation, and recommendation visually separated? Readers quote interpretation as fact. If the three are woven into the same paragraphs, they will be flattened into one claim by the second forward.

Before it leaves the building

  • Observation, interpretation, recommendation visually separated
  • Can this be rerun from a clean session?
  • Is the fatal caveat at the top?
11. Can this be rerun?
Reproducibility is the whole of this check. Ask for the link to the SQL query, dbt model, or Jupyter notebook, then ask whether it runs from a clean session. Work that lives only in a scrolled back terminal cannot be defended in six weeks when somebody produces a different number.
12. Is the fatal caveat at the top?
Every analysis has one assumption that, if wrong, reverses the conclusion. It belongs above the fold, in the same font as everything else. An appendix is where caveats go to be ignored.

What this checklist deliberately does not do

It does not check arithmetic, and it does not check the SQL. Both matter, and both are better caught by the person who wrote them rerunning their own work. A second reader adds value in exactly the places where familiarity is a disadvantage: what was assumed, what was excluded, and what the words in the summary actually claim.

It also does not check formatting. Do not spend a review on chart choices when the population rule is unwritten.

Running the review without a fight

Ask, do not assert. 'What happens to trials here' invites an answer; 'you forgot to exclude trials' invites defense, and is often wrong.

Running the review without a fight

  1. Ask, do not assert
  2. Time box ittwenty minutes, twelve questions
  3. Write down which questions could not be answered
  4. Agree in advance what a failed review means

Time box it. Twenty minutes, twelve questions, and a written note of which ones could not be answered. The unanswerable ones are the output, and they are usually two or three.

Agree in advance what a failed review means. In most teams it should mean the caveat gets stated more prominently, not that the work gets blocked. Reviews that can only ever stop things stop being requested.

Related reading on this site

The full sequence a request travels, from vague ask to delivered answer, is in the analysis process. For what makes a number trustworthy, see analytics foundations, and points five and nine are properties of the measure itself, covered in choosing a metric shape.

Who owns a definition and how it changes is the subject of data governance. If the answer is heading for a screen rather than a document, see dashboard design.

Common questions

Who should do the review?

Someone who knows the data but was not involved in this piece of work. Domain knowledge without ownership is the useful combination.

Is this overkill for a quick answer?

For a number going into a message to one person, run points one, two, and three. For anything that will be forwarded, run all twelve. The distinction is whether it will outlive the conversation.

What if the analyst cannot answer several points?

That is a normal and useful outcome. Write down which ones, and let the reader see the list. Unanswered questions attached to a number are far safer than a number that looks finished.

Should this be a form?

Keep it as questions. A form gets ticked; questions get asked. The moment it becomes a checkbox exercise it stops catching anything.

Filed underdata analysis checklist

More in Rules

Latest from Analysis Desk