David Spergel, chair of NASA's independent study on unidentified anomalous phenomena and President of the Simons Foundation, answers a question during a public meeting of NASA’s unidentified anomalous phenomena (UAP) independent study team, Wednesday, May 31, 2023 at the Mary W. Jackson NASA Headqua. How to dodge the analytics foundations mistakes that stall new teams
Photo by NASA Headquarters / NASA/Joel Kowsky on Wikimedia Commons, Public domain

Rules

Part of Where analytics foundations work goes wrong: the starting point

How to dodge the analytics foundations mistakes that stall new teams

Analytics foundations mistakes made while a data function is being set up, from instrumenting everything first to treating the first dashboard as the end.

The mistakes that hurt most when a company sets up analytics are not analytical. They are decisions about sequence, ownership, and scope made in the first few months, when nobody thinks they are making a decision at all.

Each of the nine below is described the same way: what you see, what actually caused it, and what to do instead. They are drawn from the setup phase, the period between "we should use our data" and "we have a working data function."

What to take away

  • What you seeHundreds of event types, dozens of properties, and no analysis anyone actually uses.
  • What you seeA polished BI tool, a handful of dashboards, and persistent disagreement about whether the numbers are right.
  • What you seeMarketing, sales, and finance each report a different number for the same concept, and every cross-functional meeting turns into a reconciliation exercise.
  • What you seeA leadership dashboard that is reviewed monthly, and the people doing the daily work still running their own spreadsheets.

1. Instrumenting everything before naming a single question

What you see: Hundreds of event types, dozens of properties, and no analysis anyone actually uses.

Capture-All vs Question-Led Schema

Capture everything

Starting point
All events
Event count
Hundreds
Naming
Unnavigable
Analysis used
None
Additions
Later, no context

Three decisions first

Starting point
Three decisions
Event count
Smallest set
Naming
Well-named
Analysis used
Informs decisions
Additions
When question demands

The cause: Tracking felt like a technical task with a technical answer, so it was scoped as "capture it all, we will figure out what we need later." Later never has enough context to reconstruct intent.

Instead: Pick three decisions the business makes repeatedly. Work backwards to the smallest set of events that would inform them. Add the rest when a question demands it. A short, well-named event schema is worth more than a comprehensive one nobody can navigate.

2. Buying the visualization layer first

What you see: A polished BI tool, a handful of dashboards, and persistent disagreement about whether the numbers are right.

Correct Build Order

  1. Centralize the data
  2. Model and clean it
  3. Agree on definitions
  4. Then choose a display tool
  5. Dashboards inherit trust

The cause: The delivery layer is the visible one, so it is what gets budget. But dashboards do not create trust; they inherit it from the data underneath.

Instead: Get data centralized and modeled before you shop for a way to display it. If your data lives in one place and is clean, almost any tool will do. If it does not, no tool will save you.

3. Letting each team define its own metrics

What you see: Marketing, sales, and finance each report a different number for the same concept, and every cross-functional meeting turns into a reconciliation exercise.

One Name or Two?

Do two teams genuinely need different calculations?

Yes

Give the metrics different names

No

Agree one shared definition early

The cause: Nobody was told they could not. Each team built the definition that fit its own workflow, and each is internally consistent. The general answer to this is master data management, which most companies need in a much smaller dose than the term suggests.

Instead: Agree definitions early, while there are only a handful and no one is emotionally attached. Where two teams genuinely need different calculations, give them different names, that is a legitimate outcome, and it is the ambiguity of a shared name that causes the damage.

4. Building for executives and ignoring the operators

What you see: A leadership dashboard that is reviewed monthly, and the people doing the daily work still running their own spreadsheets.

Operators First, Executives Second

Executive reporting

Review cadence
Monthly
Requests
Visible, sponsored
Questions
Vary
Feedback loop
Slow
Build order
On top of operations

Operator reporting

Review cadence
Daily
Requests
Diffuse, quiet
Questions
Repeat
Feedback loop
Fast
Build order
First

The cause: The loudest requester got served first. Executive requests are visible and sponsored; operational needs are diffuse and quiet.

Instead: Serve the people who make frequent decisions. Their questions repeat, which makes them automatable, and their feedback loop is fast enough to tell you quickly whether you built the right thing. Executive reporting built on top of well-served operations is easy; the reverse is not.

5. Treating raw collected data as trustworthy

What you see: Engagement numbers that look great and never quite match what anyone observes.

Before Publishing Any Number

  • Plot volume over time
  • Look for release-shaped steps
  • Exclude internal and test traffic
  • Build the rule into the modeled layer
  • Write down what the exclusion covers

The cause: Internal accounts, test traffic, automated monitoring, and bot activity are all in the data. Nobody excluded them because nobody knew they were there.

Instead: Before publishing anything, plot volume over time and look for steps that correspond to releases rather than to real behavior. Build an exclusion rule for internal and test activity into the modeled layer once, so every consumer inherits it. Write down what the exclusion covers, because someone will ask.

6. Making analytics whoever's spare time

What you see: Requests handled inconsistently, pipelines that break in ways nobody notices, and knowledge concentrated in one person's memory.

Spare Time vs Named Owner

Whoever's spare time

Requests
Inconsistent
Pipelines
Break unnoticed
Knowledge
One person's memory
Cost
Side task

Named owner

Requests
Accountable
Pipelines
Monitored
Knowledge
Known to everyone
Cost
Fraction of time

The cause: Analytics started as a side task for a capable generalist and was never converted into an owned responsibility.

Instead: Name an owner for the function, even if it is a fraction of one person's time. Ownership does not require a headcount; it requires that someone is accountable for whether the numbers are right and that everyone knows who that is.

7. Measuring what is easy rather than what matters

What you see: Reports full of page views, ticket counts, and totals that go up and to the right, and no obvious connection between any of them and a business outcome.

Is This Metric Vanity?

Would someone do anything differently if it moved?

Yes

Measure it, even approximately

No

Keep it out of action reporting

The cause: Easy metrics are the ones already available in the source systems. Hard metrics require joining, defining, and agreeing. Searching where the light is rather than where the keys are is the streetlight effect, and it shapes reporting estates more than any deliberate choice.

Instead: For each metric, ask what someone would do differently if it moved. If the answer is nothing, it is a vanity number, keep it out of the reporting that people are supposed to act on. Then do the harder work of measuring the thing that actually connects to a decision, even if it is only approximate at first.

8. Making every request permanent

What you see: A growing set of dashboards and recurring reports, and a team that spends most of its week maintaining rather than analyzing.

The cause: One-off questions were answered by building something recurring, because that seemed more helpful.

Instead: Answer the question first. Build something permanent only when the same question arrives a third time, or when someone needs to watch rather than know. Every recurring artifact should have a named reader and a review date, and review means being willing to switch it off.

9. Treating the first dashboard as the finish line

What you see: The launch happened, the announcement went out, and nothing about how the business runs has changed.

The cause: Success was defined as shipping the artifact rather than as changing a decision. Once it shipped, everyone moved on.

Instead: Define success as an observable behavior change, a meeting that now starts from the data, a manual export that stopped, a decision made faster. Check for it a few weeks after launch. If nothing changed, find out whether the artifact is wrong, unreachable, or answering a question nobody had.

The pattern underneath all nine

Every one of these is a case of optimizing the visible part. Instrumentation, tools, dashboards, and launches are visible; definitions, ownership, exclusions, and follow-through are not. Setup phases reward the visible work because that is what can be demonstrated in a status update.

The correction is a habit rather than a process: for anything you are about to build, name the decision it serves and the person who owns it. Both answers should be specific, and if either one is hard to produce, that is the finding.

Related reading on this site

Build order for these pieces: see analytics foundations framework. For a definition precise enough to prevent mistake three, see analytics foundations metrics. Pillar overview: analytics foundations.

For the function these mistakes damage, see business intelligence; for the analysis habits that survive them, see how to work through a question; and for the delivery layer where mistake four shows up, see dashboard design.

Common questions

Which of these is most expensive to fix later?

Definitions. Instrumentation gaps can be filled going forward, and tools can be replaced. A metric that has meant three different things across two years of history cannot be retroactively reconciled without rebuilding the history.

We already made several of these. Where do we start?

With ownership, because everything else needs someone to decide it. Then definitions for the small number of metrics that appear in meetings. Then exclusions. The rest can wait.

Is it a mistake to start with spreadsheets?

No. Spreadsheets are a reasonable first step and they surface real requirements cheaply. It becomes a mistake when the logic inside them is never moved into something reviewable, and the business ends up depending on a file on someone's laptop.

More in Rules

Latest from Analysis Desk