Where does a p-value actually come from?

We never get to measure everyone

Suppose we want to know how much people game. Not these 200 people specifically — people, in general. The trouble is obvious: we can't survey the whole planet. So we grab whoever we can, measure them, and hope our little group reflects the bigger picture.

On the left is a stand-in for “everyone” — 200 gamers and their daily hours. Pretend this is the truth we wish we knew but normally can't.

The blue line is the true average — 2.86 hours. In real life this line is invisible. We only ever see a handful of people and have to guess where it sits.

Let's take a small sample

Taking a sample is like scooping a cup of water from a lake to guess its temperature. Watch the left: we repeatedly grab just 5 people at a time. Each orange line is that scoop's guess at the average.

See how wildly the guesses jump around the true blue line — and how scattered the purple diamonds at the bottom are. With only 5 people, a single sample can be way off.

Now a much larger sample

Same population, same scooping — but this time we grab 120 people each draw. Watch the diamonds pile up again on the left.

The cloud is far tighter now, hugging the true average. The individual people up top haven't changed at all — only the size of each scoop did. Bigger samples give steadier guesses.

Do league players game more?

Now the question we actually care about. We have two groups: 80 casual players and 80 competitive league players. We suspect the league players play more. Let's look.

So league players do play more — about an hour a day more, on average. Case closed? Not quite. Remember the last section: even identical groups would show some gap, just from the luck of who we sampled. The honest question is sharper:

“Is this gap real — or is it the kind of gap luck hands out for free?”

Imagine nothing is going on

To answer that, we play devil's advocate. We assume the boring explanation is true: the two groups are really the same, and the gap we saw is pure coincidence. Statisticians call this the null hypothesis — but you can just call it the skeptic's assumption.

If the skeptic is right, then the labels “casual” and “league” don't mean anything. Everyone is really from one big crowd. Watch on the left: we dump all 160 people into a single pool and forget who was who. That pool is our null world.

Deal the labels at random

This is our first resampling move — shuffling. If the labels are meaningless, we can deal them out at random: 80 people get an “A” tag, 80 get a “B” tag, purely by chance.

Watch the left: each shuffle recolours the pool and measures the gap between those fake groups. A few gaps drop onto the pile at the bottom — every one of them is something luck produced on its own.

A thousand shuffles, one verdict

One shuffle proves nothing, so we let it run a thousand times. The histogram filling in on the left is a portrait of what chance alone can do.

Our real gap is the dashed orange line. To get the p-value, we count: of all those luck-only gaps, how many reached the orange line or beyond (the orange bars)? Divide by the number of shuffles — that fraction is the p-value.

Run it yourself

The board is cleared. Build the pile by hand and watch the p-value emerge.

First, a tiny effect

A p-value only answers “is it real?” It says nothing about “is it big?” To feel the difference, watch two groups with the same spread as our data slide apart on the left.

Right now the separation is tiny — the two bells almost completely overlap. A person from one group is nearly indistinguishable from the other. This is a negligible effect.

Now a large effect

Watch the curves pull apart on the left. The grey overlap shrinks: the groups are now clearly distinct, though — notice — even a “large” effect keeps a surprising amount of overlap.

The number driving this is Cohen's d: the raw gap divided by the data's own spread, so it's comparable across any study. Roughly, 0.2 is small, 0.5 medium, 0.8+ large.

And our actual data

The curves have now settled on the effect size measured in our real groups. That's the honest picture of how far apart casual and league players actually sit.

Drawing with replacement

Shuffling answered “is it real?”. The second flavour answers “how big, and how sure?” — and it leans on drawing with replacement. Watch the small bag on the left: we draw 8 picks, but every pick is independent, so some values get grabbed twice and others not at all.

Why allow repeats? Because we only have one sample, but we want to know how much our answer would bounce around if we could collect fresh samples over and over. Treating our data as a stand-in “population” and drawing from it with replacement manufactures that wobble. That's the bootstrap.

Bootstrap both groups

Now we bootstrap both groups at once. Watch the left: each resample draws a fresh Group A and Group B from our data (with replacement), recomputes each average, and measures the gap.

See the two mean lines jitter and the gap change every time? Each gap drops onto the pile below — that bounce is exactly the uncertainty we want to capture.

A thousand resamples

Let it run a thousand times and the gaps stack into a histogram. The shaded band on the left is the middle 95% — a 95% confidence interval for the true difference.

It tells a richer story than p alone: not just “there's a difference,” but roughly how big, and how sure we are. If the whole band sits above zero, “no difference” looks unlikely.

Build the interval yourself

The pile is cleared — your turn to fill it.

This is one recipe, not the only one

Everything here computed a p-value by simulation — shuffling and resampling until the answer appeared. That's the bootstrap / permutation approach, and it's wonderfully literal: you can watch it happen. But it's not the only way, and historically it's not even the usual one.

Long before computers could shuffle a thousand times in a blink, statisticians derived formulas for the null distribution by assuming it had a particular shape — usually a bell curve. The famous t-test does exactly that: it skips the shuffling and reads the p-value off a formula. See how our shuffled histogram and the smooth curve trace the same shape?

That's the punchline: the formula is a shortcut for the simulation we just ran by hand. Each named test you'll meet — the t-test for two means, the chi-square for counts, Mann–Whitney for ranks, an ANOVA for several groups — is just a different recipe for the same question: how surprising is my data if nothing is going on?

Simulation makes few assumptions and you can see it work. Formulas offer speed and a century of practice. Now that you've watched the machinery turn, those formulas should feel less like magic and more like a convenient shortcut.