The courtroom logic behind every "significant" result in psychology.
Every study in psychology ends with the same nagging question — is this effect real, or did we just get lucky? This unit hands you the formal procedure researchers use to answer it, called null-hypothesis significance testing (NHST). The move is sneaky and brilliant: you start by assuming nothing is going on, then ask how surprising your data would be if that were true. Master this logic and you can read a results section, smell a bad claim, and never again be fooled by the word "significant." The arithmetic is a single formula. The reasoning is the prize.
NHST runs on a maneuver borrowed straight from a courtroom. A jury doesn't start by asking "is this person guilty?" It starts by assuming the defendant is innocent and then asks whether the evidence is too strong to keep believing that. Statistics does the identical thing. We assume there is no effect — the dullest possible state of the world — and then ask: if that were true, how unlikely would data like ours be? If our data would be a freak accident under "no effect," we stop believing "no effect."
That's the whole engine, and it fits in four moves:
You can never directly prove there's an effect — there are too many ways to be wrong. But you can show that "no effect" makes the data look absurd. So we set up the boring claim (H0) as a target and try to knock it down. It's proof by contradiction, dressed in numbers.
Every test names two rival claims about a population (never just the sample). They are mutually exclusive and they cover all the possibilities between them.
| Hypothesis | Symbols (one-sample mean) | Plain English |
|---|---|---|
| H0 (null) | μ = 18 | Nothing's going on. No difference from the known value. The skeptical default. |
| H1 (alternative) | μ ≠ 18 | Something's going on. The true mean really differs. |
Notice the hypotheses are about μ (the population mean), not x̄ (our sample mean). We already know our sample mean — the question is what it implies about the population that produced it. H0 gets the equals sign; H1 gets "different from" (and as you'll see in section 5, "different" can mean ≠, >, or <).
You write H0 and H1 — and pick your cutoff — before looking at the data. Choosing your hypothesis after seeing which way the numbers fell is a cardinal sin (it's called HARKing), because it lets you "win" no matter what happens.
The p-value is the heart of the test, and it is also the single most misstated number in all of science. Here is the only correct definition:
The p-value is the probability of obtaining data at least as extreme as ours, assuming the null hypothesis is true. In symbols: p = P(data this extreme | H0 true). A small p means: "data like this would almost never happen by chance — so doubting H0 makes sense."
Read that conditional carefully — the order matters enormously. The p-value assumes H0 and asks about the data. It does not assume the data and tell you about H0. So a p of .03 does NOT mean "there's a 3% chance H0 is true" or "a 97% chance the effect is real." Those are flipped conditionals, and the flip is wrong. The p-value can only ever speak about how the data behave in a world where nothing is going on.
Before we collect a single data point, we choose a threshold for "surprising enough." It's called alpha (α), and by long convention in psychology α = .05. Crucially, alpha is not just a cutoff — it is the rate at which we'll falsely cry "effect!" when H0 is actually true. Set α = .05 and you've agreed to be fooled by random noise about 5% of the time. That's the price of admission, and it's exactly why we keep α small.
Now the payoff. Compare your p-value to alpha and the verdict is automatic:
p < α → the data are too surprising to credit to luck. We REJECT H0. The result is statistically significant.
p ≥ α → the data aren't surprising enough. We FAIL TO REJECT H0.
You'll often see the same decision made with a test statistic instead of a p-value: compute something like a z or t, then compare it to a critical value that marks the edge of the rejection region. If the statistic lands past the critical value (out in the tail), p is automatically below α and you reject. The two routes — "p vs α" and "statistic vs critical value" — always agree; they're the same boundary described two ways.
When p ≥ α we say "fail to reject H0," never "accept H0" or "prove H0 true." Failing to reject only means we didn't gather enough evidence — like a jury returning "not guilty," which is not the same as "proven innocent." Absence of evidence isn't evidence of absence. A bigger or better study might still find the effect.
Before collecting data you also decide which direction counts as surprising. This sets where the rejection region lives on the null distribution.
Here is the picture for a two-tailed test — the null distribution with both rejection regions shaded and our observed statistic marked out in the right tail:
Even with flawless arithmetic, a decision can be wrong, because we only ever see a sample, not the whole truth. Reality is either "H0 true" or "H0 false," and we either reject or don't — so there are four outcomes. Two are correct; two are mistakes. This 2×2 table is the one to commit to memory:
| H0 is really TRUE (no effect exists) | H0 is really FALSE (an effect exists) | |
|---|---|---|
| We reject H0 | ❌ Type I error false positive · false alarm · rate = α | ✅ Correct decision (a "hit" — rate = power, 1 − β) |
| We fail to reject H0 | ✅ Correct decision (rate = 1 − α) | ❌ Type II error false negative · a miss · rate = β |
Type I error = a false positive. We yelled "effect!" when there was none. Its rate is alpha (α) — which is precisely why we keep α small (usually .05). Type II error = a false negative. A real effect was there and we walked right past it. Its rate is called beta (β), and power = 1 − β is your chance of catching a real effect when one truly exists (we aim for power ≥ .80).
The two errors trade off against each other. Shrink α to almost zero and you'll almost never raise a false alarm (low Type I) — but you'll also miss more real effects (higher Type II). The biggest lever you actually control is sample size: a larger n shrinks the standard error, sharpens the test, and lowers β without touching α. That's why bigger studies have more power.
A criminal trial runs on this exact logic. The jury starts with H0 = "innocent" — the default, the boring assumption. To reject it and convict, the prosecution needs strong evidence, beyond a reasonable doubt. Weak evidence? You don't convict; you "fail to reject" — which is not the same as declaring the defendant truly innocent. So Type I error = convicting an innocent person (a false alarm — we rejected a true null), and Type II error = letting a guilty person walk (a miss — we failed to reject a false null).
Alpha is how much false-conviction risk the system tolerates. Setting α = .05 is like saying, "We'll convict only when the evidence is so strong it would fool us less than 5% of the time." We keep α small because convicting the innocent is the scarier mistake — just as a false positive is usually the scarier mistake in science.
Here is the distinction that separates a careful reader from a gullible one. "Statistically significant" means only one thing: the effect is probably not zero. It says nothing about whether the effect is big enough to matter. Those are two different questions, and the p-value answers only the first.
Why does this matter so much? Because the p-value depends heavily on sample size. The standard error shrinks as n grows, so with a large enough sample any difference from H0 — no matter how trivial — will eventually cross p < .05. Pour in 50,000 participants and a difference of half a point on a 100-point scale can be "highly significant" and utterly meaningless. Significance tells you an effect is real; it does not tell you it is large.
That's what effect size is for — a number that captures how big the effect is, independent of sample size (e.g., Cohen's d for mean differences, where ≈ 0.2 is small, 0.5 medium, 0.8 large; or r for correlations). Best practice is to report both: the p-value answers "is it real?" and the effect size answers "is it big enough to care about?" A complete result needs both.
The flip side is just as important. A non-significant result with a tiny sample might be hiding a real, even large, effect — the study simply lacked the power to detect it. So "p > .05" never means "no effect"; it means "this study couldn't rule chance out." Sample size cuts both ways.
The question: Are SJNY students more stressed than the national norm? The national Perceived-Stress norm is μ = 18, with a known population σ = 6. Our sample of n = 200 students has a mean stress of 20.09 (column J). Because we know σ, we can use a z-test, where the test statistic is z = (x̄ − μ) / (σ / √n). Walk the four steps — predict each before you reveal it.
"Reject H0" here means the data are incompatible with "no difference" — not that we've proven the exact size of the gap, and not that there's only a 0.1% chance H0 is true. The p-value (< .001) is the probability of data this extreme if μ really were 18.
You won't compute z by hand on exams — Sheets does it. Open the SJNY Student Life dataset and type these in any empty cell (perceived stress is column J):
=(AVERAGE(J2:J201) - 18) / (6 / SQRT(COUNT(J2:J201))) → the z-score (≈ 4.91)
=2*(1 - NORM.S.DIST(ABS(z), TRUE)) → the two-tailed p-value (< .001)In the second formula, put your z-value where it says z. The 2* at the front makes it two-tailed (it counts both tails); drop the 2* for a one-tailed p. A handy gut check: a z beyond about ±1.96 always yields a two-tailed p below .05 — that's the same ±1.96 boundary drawn in the figure above.
"Fail to reject" is not "prove H0 true" — it only means not enough evidence; absence of proof isn't proof of absence, so never write "accept H0." The p-value is P(data | H0), not P(H0 | data) — a small p does not mean H0 has a small chance of being true. Significant ≠ large: a huge sample can make a trivial difference "significant," so always pair p with an effect size. And pick alpha and your tails before you see the data — never after, to make a result "work."
Ten quick true/false items with instant explanations. (Different from your Canvas quiz — real practice.)