Guides
Part of Why does an analytics certification actually matter before you pay?
Analytics certifications metrics as practitioners see it
Analytics certifications judged afterwards: measure the mechanism you named, not satisfaction, and why every published outcome figure flatters the program.
Almost nobody checks whether a credential worked. The exam is passed, the line is added, and the question of what changed is never asked, partly because nobody agrees what would count as an answer.
This page proposes what to measure, when to measure it, and why the published evidence about credential outcomes is systematically flattering.
What to take away
- Measure the mechanism you expected, not your general career satisfaction. You expected something specific; check that specific thing.
- Six months is the earliest any of this reads, and the screening measures read earliest of all.
- Every published outcome figure about a credential is drawn from people who completed it, which makes it useless for deciding whether to start.
Measure the mechanism you named
Before starting, you should have written one sentence about what the credential was for. Each version of that sentence has its own check.
A worked example. One such sentence might read: "I am taking this so the recruiter screen stops rejecting me before a human sees the application." The matching measure is response rate to applications, counted three months before and three months after.
What to measure by expectation
What you expected
- Automated screening
- Response rate
- Named gate
- Whether gate opened
- Structured learning
- Unsupervised capability
- Career-change signal
- Interviews reached
- Partner-tier requirement
- Requirement met
What to measure
- Automated screening
- Three to six months
- Named gate
- Immediately
- Structured learning
- Six months
- Career-change signal
- Six months
- Partner-tier requirement
- Immediately
When it reads
- Automated screening
- Named gate
- Structured learning
- Career-change signal
- Partner-tier requirement
| What you expected | What to measure | When it reads |
|---|---|---|
| Getting past automated screening | Response rate to applications, before and after | Three to six months |
| A specific gate, named by a specific party | Whether the gate opened | Immediately |
| Structured learning you would not otherwise do | What you can now do unsupervised | Six months |
| A career-change signal | Interviews reached, not offers received | Six months |
| Employer partner-tier requirement | Whether the requirement was met | Immediately |
The middle row is the only one that measures capability rather than access, and it is the one most people claim as their reason. Test it honestly: list three things you can now do without supervision that you could not do before, with an example of each. A short or vague list is the answer.
The screening measure, done properly
Response rate is the cleanest available signal, and it needs a before period. Count applications sent and responses received for the three months before, then for the three months after, in the same market and for the same kind of role.
Screening response rate, before and after
Three months before
- Applications sent
- Count all
- Responses received
- Count all
- Same market
- Yes
- Same role type
- Yes
- Application quality
- Baseline
- Sample size
- Small
Three months after
- Applications sent
- Count all
- Responses received
- Count all
- Same market
- Yes
- Same role type
- Yes
- Application quality
- Usually improves
- Sample size
- Small
Two cautions. Application quality usually improves at the same time, so any change is confounded. And the sample is small, which means a difference of one or two responses is noise. Report it as a direction rather than as a measurement.
Roughly twenty applications per period is the practical floor for screening data, because one response at that volume still moves the rate by five percentage points.
Why published outcome claims are unusable
Programs publish figures about what their graduates went on to do. Those figures are computed over people who finished, who responded to a survey, and who had an outcome worth reporting.
That is selection bias at three separate stages, and no amount of care in the arithmetic afterwards repairs it. The population you belong to when deciding whether to start includes everyone who registered, and nobody publishes figures over that population.
The underlying reason credentials carry any information at all is that they are costly to obtain and easy to verify, which is the standard account of signaling. That mechanism says nothing about your individual return, and it predicts that as more people hold a marker, the marker distinguishes less.
A blank certificate is a fair image of the problem. The document is identical for everyone who holds it, so whatever distinguishes two holders is somewhere else entirely.
Measures not worth taking
Salary change. Too many other things move at the same time, market conditions dominate, and the comparison you would need does not exist. Any figure you compute will be a story you already believed.
Recruiter contact volume. Driven by profile keywords and market conditions, not by the credential.
Confidence. Rises with study and is uncorrelated with the thing you care about.
A six-month review worth doing
Half an hour, in writing.
Six-month review, in writing
- What was the sentence I wrote?
- Did that specific thing happen?
- Name three things I can now do unsupervised
- What did I build with it?
- What is the renewal commitment?
- Would I do it again?
Four questions, answered on paper:
- Did the gate I named actually open?
- Can I now do something unsupervised that I could not before?
- Did my application response rate move against a matched before period?
- Have I used it on real work since I finished?
The fourth question, have I used it on real work since I finished, decides most of the value. A credential with no application behind it will not survive a technical conversation, and six months is roughly when the study starts to fade.
Related reading on this site
The decision itself is covered in analytics certifications, with the pre-purchase sequence in twelve checks before you pay and the errors to avoid in nine credential mistakes. For measuring your own progress more broadly, see measuring your own progress honestly, and for placing the credential against what you are short of, see four axes and a planning pass.
Common questions
Six months feels slow. Can I tell earlier?
Only for the gate cases, which read immediately. Everything else needs enough applications or enough work to say anything, and a verdict at six weeks will be a mood rather than a measurement.
What if the honest answer is that nothing changed?
Then you have learned something worth far more than the fee: that in your market, this credential is not the constraint. Spend the next block of hours on the thing that is.
Can I compare myself to people who did not take it?
Not usefully. You have no comparison group and no way to construct one, so any such comparison is an anecdote. Measure the mechanism you named instead.
Should I put a lapsed credential on my profile?
Only with the dates shown. A lapsed credential presented as current is a small dishonesty that is trivially checkable, and a dated one is simply a record of study, which is fine.
