Data analysis: a clear guide with practical examples

Most analysis requests arrive badly formed. "Can you pull the numbers on churn?" is not a question: it is a gesture in the direction of a question. The work of analysis starts before any query runs, and it starts with turning that gesture into something a query can actually answer.

This guide follows one request from the moment it lands to the moment someone acts on it.

Step 1: find the decision underneath the request

Ask what changes depending on the answer. If nothing changes, the request is curiosity, and curiosity is fine, but it should be queued behind work that moves something.

Three follow-up questions usually convert a vague ask into a real one:

What will you do differently if the number is high versus low? This exposes the decision. If the person cannot answer, the analysis has no target.
What number were you expecting? This surfaces the prior. If your result lands far from it, you now know to check your work before you check their assumptions.
When do you need it, and how good does it have to be? A directional read for a Tuesday standup and a figure that will appear in a board deck require different levels of rigor. Do not silently deliver one when the reader assumes the other.

Write the refined question down and send it back before you start. "You want to know whether accounts that started on the self-serve plan in the first half of the year cancel at a higher rate than accounts that started with a sales conversation, so you can decide where to put onboarding effort." That sentence is worth more than an hour of querying.

Step 2: define the population and the window before you count

Almost every disputed analysis in a company traces back to two teams counting different rows and calling it the same word. Nail these down explicitly:

Choice	The question to settle	What goes wrong if you skip it
Unit of analysis	Are you counting users, accounts, sessions, orders, or seats?	Per-user and per-account results can point in opposite directions
Population	Who is in and who is out: trials, internal accounts, test data, churned users, refunds?	Internal and test traffic quietly inflates every activity metric
Time grain	Days, weeks, months, and in whose timezone?	Weekly seasonality gets read as a trend when grains are mixed
Cohort anchor	Is a row assigned to when the account signed up, or to the calendar period the event happened in?	Signup-cohort and calendar-period views of the same data disagree, correctly
Denominator	Percent of what, exactly?	A rate can rise because the numerator grew or because the denominator shrank
Late data	How long does the source keep writing to a period after it closes?	Recent periods look artificially low and get read as a decline

The last row deserves particular attention. Many pipelines backfill. Refunds, corrections, delayed webhooks, and offline conversions all land after the fact. If the trailing seven days are still filling in, exclude them from the trend or mark them clearly, or you will report a drop that does not exist.

Step 3: look at the raw data before you aggregate it

Aggregation hides the things most likely to make you wrong. Before computing anything, spend a few minutes with the rows themselves.

Count the rows and the distinct keys. If they differ when they should not, you have duplicates from a join and every sum is inflated.
Check the range of every date column. Dates in 1970 or 2099 mean nulls have been cast to defaults somewhere upstream.
Look at the tails. Sort ascending and descending. The largest customer, the negative quantity, the order with 400 line items: these are the rows that will drive your average.
Plot the row count per day for the whole history. Steps, gaps, and cliffs in the volume line are the fingerprints of a tracking change, a migration, or an outage. If something changed six months ago, comparing across that boundary compares two different measurement systems.
Check nulls per column, over time. A field that was almost always populated and is now missing for a large share of rows usually means a form changed or a client stopped sending it.

If you find one of these, stop and resolve it. Analysis built on a silently broken column is worse than no analysis, because it carries authority.

Step 4: watch for the failure modes that survive good SQL

Correct queries produce wrong conclusions all the time. These are the recurring ways:

The mix shifts. A metric can improve in every segment while the overall number falls, simply because the share of a low-performing segment grew. Whenever an aggregate moves, decompose it: how much came from within-group change, how much from the mix between groups? If you never do this decomposition, you will attribute mix shifts to performance.

The survivors answer for everyone. Analyze only accounts still active today and you will conclude that your product works wonderfully. The accounts that would have told you otherwise are gone. Define the population at the start of the period, not at the end.

The comparison group is missing. "Users who used feature X retained better" is nearly always true, because engaged users use more features. Without a comparison built to be similar in the ways that matter, you have described engagement, not measured a feature.

The window flatters. Start a chart the week after an unusual dip and any recovery looks like growth. Pick period boundaries before you see the result, and show enough history that the reader can judge the baseline themselves.

Small denominators shout. A segment with a handful of accounts will swing wildly and produce the most exciting percentage on the page. Set a minimum sample size for any cell you are willing to draw a conclusion from, and gray out the rest.

Repeated slicing finds patterns in noise. Cut the same data twenty ways and something will look significant. If you found a result by exploration, say so, and treat it as a hypothesis to test on fresh data rather than a finding.

Step 5: separate what you measured from what you think it means

Keep three things visually distinct in whatever you deliver:

The observation. What the data shows, stated with its population and window attached.
The interpretation. Your explanation of why. This is judgment, and it should be labeled as judgment.
The recommendation. What you think should happen next.

Readers routinely absorb the third and quote it as the first. Making the boundary explicit protects you when the interpretation later turns out to be wrong, and it usually is, at least partly.

State the size of the effect, not just its direction. "Retention is lower for self-serve accounts" is much less useful than the same claim with the gap quantified from your own data and a note on how much it moves when you vary a defensible assumption.

Step 6: give the answer with its limits attached

A finished analysis should be readable in a minute and reproducible in an hour. Deliver:

The refined question, in one sentence.
The answer, with the number and its unit.
The population, window, and any exclusions.
The main caveat: the assumption that, if wrong, would flip the conclusion.
The link to the query or notebook that produced it.

That last item matters more than it looks. Six weeks later someone will ask why the number differs from a newer one. Without the saved logic, you cannot tell whether the data changed or your definition did.

Sensitivity: the cheapest rigor available

Before publishing, take the assumption you are least sure about and change it. Exclude the largest customer. Move the cutoff by a week. Switch from mean to median. Include the accounts you excluded.

If the conclusion survives all of that, say so: it is a strong statement, and it is one you can defend. If the conclusion flips when one account is removed, you have not found a pattern, you have found an account. Both are worth reporting, but they call for different actions.

When to say the data cannot answer it

Some questions are unanswerable with what you have. Attribution across channels with no shared identifier. The effect of a change that shipped alongside three other changes. Anything about people who never showed up in your system.

Saying so is a real deliverable. Follow it with the cost of making it answerable: the tracking that would need to exist, the holdout that would need to be reserved, the experiment that would need to run. That converts a dead end into a decision about whether the answer is worth buying.

Common questions

How long should an analysis take?

Set the budget before you start, in conversation with the person asking. Most requests deserve a couple of hours; a few deserve a week. The failure mode is spending a week on something that needed two hours, and the other failure mode is spending two hours on something that will drive a large commitment.

What if the result contradicts what leadership believes?

Check your work twice as hard, then present it plainly with the sensitivity analysis attached. Lead with the method rather than the conclusion so the discussion is about the evidence. Expect the first meeting to be about whether the number is right, not about what to do, that is normal and worth budgeting for.

Should I show the query?

Link it, do not paste it. Most readers will never open it; the ones who do are exactly the ones whose scrutiny you want.

When is an analysis worth turning into something recurring?

When the same question is asked a third time, or when the answer needs to be watched rather than known once. Recurring work carries a maintenance cost, so it should be a deliberate decision, not the default outcome of every request.