Analytics foundations questions: what to know and why

Data work is mostly interviewing. Before a query gets written, somebody has to find out what a field really contains, who decided the current definition, and what the person asking actually plans to do. Nobody volunteers that. You have to ask.

This page is a set of questions organized by who you are talking to. They are worded to be asked out loud, in a fifteen minute conversation, without preparation.

What to take away

• The expensive unknowns live in other people's heads, not in the data. Ask before you build.
• Ask what a field contains, never what it is called. The name is a summary written by someone who has since left.
• Any question whose answer nobody knows is itself a finding. Write it down and say so.

Asking the person who requested it

The request as delivered is a compressed version of something longer. These reopen it.

• What will you do differently depending on the answer? If nothing, the request is curiosity. That is allowed, but it should not outrank work that changes something.
• What do you expect the number to be? A stated prior is free insurance. When your result lands far from it, you check your query before you challenge their belief, and that order saves a lot of embarrassment.
• Who else will see this, and what will they do with it? A figure built for one person's judgment gets forwarded into a deck by Thursday. Knowing that in advance changes how much you hedge and how much context you attach.
• What decision is currently being made without it? There is always a current method, usually a guess or a spreadsheet. Ask to see it. It tells you the shape of the answer they can use.
• How wrong can this be before it matters? Directional, within ten percent, or exact. Each is a different amount of work, and nobody volunteers which one they need.
• What is the deadline, and what happens if it slips? Deadlines attached to a meeting are real. Deadlines attached to nothing are preferences.

Asking whoever owns the source system

This is the conversation most teams skip, and it is where the surprises live. The person to find is whoever writes the code that creates the record, not whoever administers the database.

• When exactly does this row get written? At the click, at the server response, at the nightly batch. The gap between the event and the record is a timing error you will otherwise attribute to seasonality.
• Can a row be updated after it is created, and which columns change? A status field that mutates in place destroys any history you thought you had. If it mutates, ask whether anything logs the previous value.
• Are rows ever deleted, and what happens to the ones that are? Hard deletes mean your counts of the past will change every time you run them.
• What is in here that is not a real customer? Internal accounts, load tests, monitoring, staff logins, demo data, the support team impersonating users. Ask for the rule that identifies them, not just the assurance that they are rare.
• What changed in the last two years? Migrations, renames, a new client version that stopped sending a field. Any boundary you compare across is a boundary between two measurement systems.
• What would break if I depended on this? Engineers will tell you honestly if you ask this way. They will not volunteer it, because from their side the system is working correctly.

Asking whoever built the number that already exists

Most of the time you are not measuring something new. You are being asked to reconcile with, or replace, a figure somebody already publishes.

• Where is the logic, and can I read it? If the answer is a spreadsheet on a laptop, you have found the real problem. What you are trying to establish is the data lineage: every hop between the originating system and the published figure is a place the meaning can change.
• What does it exclude, and why? Exclusions are the fingerprint of a definition. Two numbers that disagree almost always disagree here rather than in the arithmetic.
• When did the definition last change, and was the history restated? A metric that has quietly meant three things across two years cannot be trended, and someone should say so before it appears on a slide.
• Who signs off when it moves? If nobody does, the number is decorative. That is worth knowing before you spend a week matching it.

Asking yourself before you publish

• Could this number be right for a boring reason, like a join that duplicated rows?
• If I removed the single largest contributor, would the conclusion survive?
• Is the most recent period complete, or is it still filling in?
• What is the one assumption that, if wrong, flips the answer? Is it written down where the reader will see it?
• Six weeks from now, can I reproduce this exactly? If the query is not saved, the answer is no, and reproducibility is the property that separates a result somebody can check from one they have to believe.

How to ask without sounding like an audit

These questions can land badly if they arrive as a list. Three things help. Ask about the system rather than the person: "when does this row get written" invites explanation, "why is this data wrong" invites defense. Say what you plan to do with the answer, because people are more forthcoming when they can see the use. And write down what you learn somewhere shared, so the next person does not have to hold the same conversation again.

The record you build from these conversations is worth more than any single analysis. It is the only place in most companies where the difference between what a field is called and what it contains is written down.

Related reading on this site

The chain a number travels, and how each link fails, is covered in the analytics foundations overview. For the order these answers get built into, see the six layers a project passes through. For turning the answers into a definition two people would implement identically, see how to write a metric definition. For what happens when nobody asks, see the setup mistakes these questions prevent. And for converting a vague request into something a query can answer, see the analysis request sequence.

Common questions

How long should this take?

Half a day of conversation for a piece of work that will take a week. The ratio feels wrong until the first time a source system question saves you from publishing something wrong.

What if the source system owner has left?

Read the code that writes the row, and if that is not available, test the system yourself: create a record, then look at what appeared. An hour of experiment beats a week of assumption.

Nobody can answer half of these. Is that a failure?

No, it is the output. A list of unanswerable questions about a data source is a legitimate finding, and it is usually the strongest argument you will ever have for fixing the source rather than working around it.

Do I need to ask all of this every time?

Only the first time you touch a source. After that you are updating a record you already hold, which takes minutes.