Data analysis questions: facts, examples and context

Numbers arrive at you from elsewhere: a slide, a supplier's report, another team's summary, a chart in a meeting. You did not build them, you cannot see the query, and you have about two minutes to decide how much weight to put on them.

This page is a reading protocol for that situation. The questions are grouped by what part of the claim they interrogate, and most of them can be answered from the artifact itself if you know where to look.

What to take away

• You can assess a number you did not build. The information is nearly always on the page, unlabeled.
• Ask about the denominator and the window before anything else. Those two carry most of the error.
• A presenter who cannot say what is excluded has not read the definition either, and that is the useful finding.

Questions about the number itself

• What is being counted, one row at a time? Push until you get a sentence like "one row per completed order." Vague subjects hide the disagreement.
• Percent of what? Ask for the denominator out loud. If it is something like active users, ask what happens to the rate when activity falls.
• What is excluded? Test accounts, refunds, internal traffic, canceled records, one enormous customer. Exclusions are where two versions of the same number diverge.
• Is this a mean, and what does the distribution look like? For anything skewed, ask for a median or a high percentile as well. The gap between them is the real story.
• Where did it come from? A modeled table, a source system, or somebody's spreadsheet. All three appear identical on a slide.

Questions about the picture

• Where does the vertical axis start? A truncated axis turns a small change into a dramatic one. It is not always wrong, but it should be visible and acknowledged.
• Why does the series begin there? A start date chosen after seeing the data is an argument, not a window. Ask what the chart looks like with a year more history.
• What are the two axes on this dual axis chart actually scaled to? The apparent relationship between two lines on independent scales is a property of the scaling, not of the data.
• Is the last point complete? A partial current period plotted beside complete ones produces a fall that does not exist.
• What does the color mean, and does it mean the same on the next chart? Color that changes meaning across a deck is how people misread a whole document without noticing.

Questions about the comparison

• Compared to what, and who chose it? Prior period, same period last year, plan, or a peer group. Each answers a different question, and the choice is usually the most consequential decision in the whole chart.
• Were the groups defined before or after the result was seen? Post hoc groupings are common, rarely disclosed, and hard to interpret.
• Is the comparison at the same grain? A weekly figure against a monthly one, or per session against per user, produces a ratio with no meaning.
• Does the mix explain it? A total can move entirely because the share of a segment changed, with nothing changing inside any segment, which is the everyday form of Simpson's paradox. Ask whether anyone decomposed it.

Questions about what is not shown

• How large is the base under each percentage? Small denominators produce the largest percentage moves on any page, every time.
• What happened to the categories that are not here? An "other" bucket that is larger than the named categories is doing more work than the chart admits.
• Who is missing entirely? People who never entered the system, customers who churned before the window opened, records that failed to be recorded at all. Nothing in the artifact will mention them.
• What was measured before? If a comparison is against a remembered baseline rather than a measured one, the claim is between an instrument and a memory.

Questions about a movement

• Is this bigger than the usual wobble? Ask to see the same measure over a longer period with nothing happening. Most single period changes are inside the normal range.
• Did anything change in how it is measured? A tracking change, a migration, a definition update, or a new client version will move a series without anything real changing.
• Did anything else change at the same time? If three things launched that month, attributing the move to one of them is a choice, not a finding.

Asking these without stopping the meeting

Pick one. In a room, the highest yield question is almost always the denominator or the exclusions, because the answer is short and it either lands or it does not.

Frame it as curiosity about the definition rather than doubt about the conclusion. "What counts as active here" is a question anyone can ask without implying anything. "Are you sure about this" starts a different meeting.

And write down the questions nobody could answer. Those are the follow ups, and they are worth more than the number was.

Related reading on this site

For producing an answer rather than receiving one, see the analysis sequence. The chain that determines whether any figure means what it says is described in analytics foundations. Several questions above exist because of the process errors in nine analysis mistakes. For denominators, averages, and the shapes that mislead, see choosing a metric, and for numbers that describe a future rather than a past, see forecasting.

Common questions

What if I only get one question?

Ask what is excluded. It is short, it is answerable, and the answer separates people who have read the definition from people who have inherited the slide.

Is this not just being difficult?

Only if you ask all of them. One well chosen question improves the discussion; a full interrogation makes everyone defensive and teaches people not to show you work early.

The presenter did not build the number either. Now what?

Then you have found the real issue, which is that a figure is circulating with no owner. Ask who does own it, and treat that as the deliverable of the meeting.

How do I apply this to a supplier's report?

The same way, plus one addition: ask who selected the sample and whether anyone who dropped out is included. Reports built from customers who stayed describe customers who stayed.