Reviews
Part of Where analytics foundations work goes wrong: the starting point
Analytics foundations questions to ask requesters, source owners and yourself
Analytics foundations questions to ask the requester, the source system owner, whoever built the existing number, and yourself before you publish.
"""
Data work is mostly interviewing: before a query gets written, somebody has to find out what a field really contains, who decided the current definition, and what the person asking plans to do. Nobody volunteers that. You have to ask.
This page is a set of questions organized by who you are talking to. They are worded to be asked out loud, in a fifteen minute conversation, without preparation.
What to take away
- The expensive unknowns live in other people's heads, not in the data. Ask before you build.
- Ask what a field contains, never what it is called. The name is a summary written by someone who has since left.
- Any question whose answer nobody knows is itself a finding. Write it down and say so.
Asking the person who requested it
The request as delivered is a compressed version of something longer. These reopen it.
Requester questions before you start
- What will you do differently?
- What do you expect the number to be?
- Who else will see this?
- What decision is being made without it?
- How wrong can this be before it matters?
- What is the deadline, and what if it slips?
Asking the requester
- What will you do differently depending on the answer?If nothing, the request is curiosity. That is allowed, but it should not outrank work that changes something.
- What do you expect the number to be?A stated prior is free insurance. When your result lands far from it, you check your query before you challenge their belief, and that order saves a lot of embarrassment.
- Who else will see this, and what will they do with it?A figure built for one person's judgment gets forwarded into a deck by Thursday. Knowing that in advance changes how much you hedge and how much context you attach.
- What decision is currently being made without it?There is always a current method, usually a guess or a spreadsheet. Ask to see it. It tells you the shape of the answer they can use.
- How wrong can this be before it matters?Directional, within ten percent, or exact. Each is a different amount of work, and nobody volunteers which one they need.
- What is the deadline, and what happens if it slips?Deadlines attached to a meeting are real. Deadlines attached to nothing are preferences.
Asking whoever owns the source system
This is the conversation most teams skip, and it is where the surprises live. The person to find is whoever writes the code that creates the record, not whoever administers the database.
Source system owner questions
- When exactly does this row get written?
- Can a row be updated after creation?
- Are rows ever deleted, and what happens?
- What is in here that is not a real customer?
- What changed in the last two years?
- What would break if I depended on this?
When the person who wrote the code is unavailable, work down a short list before you give up. The analyst who last queried the table usually knows its quirks. Application support knows what the screens do. Vendor documentation covers the fields the vendor owns.
Asking the source owner
- What triggers this row into existence?A user action, a batch job and a scheduled import look identical in a table. Ask which one it is, because the answer sets how current your counts can be.
- What values can this field hold?Ask for the full list or the constraint, not a sample. A field that shows five values in your test extract may hold fifty in production.
- Can a record be changed after it is written?Immutable and editable records need different handling. If a value can change, ask which timestamp marks the change, because the earlier value is lost.
- What is the key, and can it repeat?A column that looks unique in a small extract often is not. Ask what the code treats as the identifier, and whether one entity can appear on more than one row.
- Does anything get deleted?Some systems remove the row, some set a flag, and some move it to an archive. Each of those makes your history mean something different.
- Which field do you trust least?System owners know their own weak points. This question is free to ask and it often names the field that will break your analysis.
Asking whoever built the number that already exists
Most of the time you are not measuring something new. You are being asked to reconcile with, or replace, a figure somebody already publishes.
Questions for the existing number
- Where is the logic, and can I read it?
- What does it exclude, and why?
- When did the definition last change?
- Was the history restated?
- Who signs off when it moves?
Asking the number builder
- Where is the logic, and can I read it?If the answer is a spreadsheet on a laptop, you have found the real problem. What you are trying to establish is the data lineage: every hop between the originating system and the published figure is a place the meaning can change.
- What does it exclude, and why?Exclusions are the fingerprint of a definition. Two numbers that disagree almost always disagree here rather than in the arithmetic.
- When did the definition last change, and was the history restated?A metric that has quietly meant three things across two years cannot be trended, and someone should say so before it appears on a slide.
- Who signs off when it moves?If nobody does, the number is decorative. That is worth knowing before you spend a week matching it.
Asking yourself before you publish
- Could this number be right for a boring reason, like a join that duplicated rows?
- If I removed the single largest contributor, would the conclusion survive?
- Is the most recent period complete, or is it still filling in?
- What is the one assumption that, if wrong, flips the answer? Is it written down where the reader will see it?
- Six weeks from now, can I reproduce this exactly? If the query is not saved, the answer is no, and reproducibility is the property that separates a result somebody can check from one they have to believe.
How to ask without sounding like an audit
Questions can land badly if they arrive as a list; three things help. Ask about the system, not the person: "when does this row get written" invites explanation; "why is this data wrong" invites defense.
Say what you will do with the answer, since people are more forthcoming when they see the use. Write down what you learn somewhere shared, so the next person does not have to repeat it.
The shared record can be one table, one row per field. This is the whole template:
How to ask
| Field | What it contains | Who writes it | Last confirmed |
|---|---|---|---|
| order_status | Lifecycle state of the order, written when the row is created | Orders service | the month you confirmed |
| shipped_at | Carrier handoff time, empty until the order ships | Orders service | the month you confirmed |
The record you build from these conversations is worth more than any single analysis.
Related reading on this site
The chain a number travels, and how each link fails, is covered in the analytics foundations overview. For the order these answers get built into, see the six layers a project passes through. For turning the answers into a definition two people would implement identically, see how to write a metric definition.
For what happens when nobody asks, see the setup mistakes these questions prevent. For converting a vague request into something a query can answer, see the analysis request sequence.
Common questions
How long should this take?
Half a day of conversation for a piece of work that will take a week. The ratio feels wrong until the first time a source system question saves you from publishing something wrong.
What if the source system owner has left?
Read the code that writes the row, and if that is not available, test the system yourself: create a record, then look at what appeared. An hour of experiment beats a week of assumption.
Nobody can answer half of these. Is that a failure?
No, it is the output. A list of unanswerable questions about a data source is a legitimate finding, and it is usually the strongest argument you will ever have for fixing the source rather than working around it.
Do I need to ask all of this every time?
Only the first time you touch a source. After that you are updating a record you already hold, which takes minutes.
"""
