Hypothesis Testing

P1.statistics-advanced.04 · Audience: it-ml · Prerequisites: Bayesian Inference

Bayesian Inference Correlation, Causation & Confounding

Interview-depth testing for the data-scientist track: confidence vs credible intervals, the t-test's assumptions and its rank-based alternatives, chi-square goodness-of-fit and independence, and why ANOVA replaces a pile of pairwise t-tests. Back to the foundations: Statistical Inference and Statistical Inference Deep Dive.

Step 1 / 5 — Interpreting a 95% CI; contrast with a credible interval (s11)

A 95% confidence interval is a frequentist statement about the procedure, not this interval: if you repeated the experiment many times and built an interval each time, 95% of those intervals would contain the true parameter. For the specific interval you computed, θ is either in it or not — the '95%' is a property of the method's long-run coverage.

\bar{x} \pm t_{n-1,\,0.975}\cdot \frac{s}{\sqrt{n}} \quad\text{(one-sample mean)}

Common misreading (wrong): 'there's a 95% probability the true mean is in this interval.' That is a Bayesian credible-interval statement — legitimate only under the Bayesian framework, where θ is random with a posterior.

	Confidence interval (frequentist)	Credible interval (Bayesian)
θ	fixed unknown constant	random variable with a posterior
Random thing	the interval (via the data)	θ
Correct reading	95% of such intervals cover θ	P(θ ∈ interval \| data) = 0.95
Needs a prior	no	yes

With a flat prior and symmetric likelihood the two often coincide numerically — but their interpretation differs, and this distinction is a classic interview probe.

📚 Go Further

Rigorous references for hypothesis testing.

Type	Resource
Book	Wasserman, All of Statistics — testing, chi-square, ANOVA
Book	Casella & Berger, Statistical Inference — CI theory, nonparametric tests
In-app	Statistical Inference Deep Dive, for the t-statistic and multiple-comparison correction
In-app	Bayesian Inference, for the credible-interval contrast

✅ Key takeaways

A 95% CI is a frequentist statement about the procedure's long-run coverage: 95% of such intervals contain the true parameter. Only a Bayesian credible interval licenses '95% probability θ is in this interval' — and it needs a prior.
The t-test assumes independence, approximate normality, and (two-sample) equal variances — default to Welch's t-test. When normality fails, use rank-based alternatives: Wilcoxon signed-rank, Mann–Whitney U, Kruskal–Wallis.
Chi-square compares observed vs expected counts: goodness-of-fit has df = k−1, independence has df = (r−1)(c−1). Expected counts should be ≳ 5 per cell; otherwise use Fisher's exact test. Report Cramér's V for effect size.
ANOVA's F-statistic is a single omnibus test for 3+ group means; running all pairwise t-tests inflates the family-wise error rate (5 groups → 10 tests → ≈40% false-positive chance). Run post-hoc comparisons with correction only after a significant F.

Next: This closes the statistics-advanced track. The deep-dive tails of S1–S6 in statistics-core carry the module-specific advanced material.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Correlation, Causation & Confounding →

Data & Statistics Literacy · P1

Go up a level