A p-value answers one question: if nothing were really going on, how often would chance alone produce a result as big as mine? Here you get to run the "nothing going on" world thousands of times and check.
1 ยท Run YOUR study
Two groups (say, treatment vs. control), n per group. The slider sets the true effect in the world your study samples from โ in real research you never get to see this slider. Scores ~ N(0 or d, 1).
True effect d0.5
n per group
25
โ
My observed difference
โ
Group means (B โ A)
2 ยท Now run the null world
Same design, same n โ but the true effect is forced to zero. Every difference below is pure luck of the draw.
Null-world differences (luck only)As extreme as mine (two-tailed)My result
0
Null studies run
โ
At least as extreme as mine
โ
p (simulated, two-tailed)
โ
At ฮฑ = .05
Try this:
Set true d = 0, run your study, then run 10,000 null studies. Sometimes "your" totally-null study still gets p < .05 โ that's a Type I error, and it happens about 5% of the time. Re-run your study a bunch of times and watch.
Set d = 0.5 with n = 10. Run your study several times: notice how wildly p bounces around with small samples, even when a real effect exists.
Keep d = 0.5 but raise n to 200. Now the observed difference barely moves and p is almost always tiny. Sample size buys stability.
Sigma says: A p-value is NOT the probability your hypothesis is wrong. It's the proportion of null-world studies that beat your result. You just watched that proportion get counted, one study at a time.