Bayesian Inference

P1.statistics-advanced.03 · Audience: it-ml · Prerequisites: Estimation Theory

Interview-depth Bayesian reasoning for the data-scientist track: Bayes' theorem and the base-rate fallacy, the frequentist–Bayesian contrast and its inferential consequence, and how posteriors are computed — conjugacy, MCMC, and variational inference. (Back to the foundation: M5 — Probability and Statistics Basics.)

Step 1 / 4 — Bayes' Theorem; Prior, Likelihood, Posterior; Base-Rate Fallacy

\underbrace{P(H\mid D)}_{\text{posterior}} = \frac{\overbrace{P(D\mid H)}^{\text{likelihood}}\;\overbrace{P(H)}^{\text{prior}}}{\underbrace{P(D)}_{\text{evidence}}}, \qquad P(D)=\sum_{H'} P(D\mid H')P(H')

Prior P(H) — belief in the hypothesis before seeing data (here, the base rate).
Likelihood P(D|H) — how probable the data are if the hypothesis holds.
Posterior P(H|D) — updated belief after the data.
Evidence P(D) — normaliser (total probability over all hypotheses).

The base-rate fallacy is ignoring the prior P(H) and reading a high likelihood / test-sensitivity as a high posterior. When the base rate is tiny, even an accurate test yields mostly false positives.

Worked example — a 99%-accurate test for a disease with 1% prevalence:

Quantity	Value
Prior P(D) — prevalence	1.00%
Sensitivity P(+\|D)	99%
Specificity P(−\|¬D)	99%
P(+) = evidence	0.0198
Posterior P(D\|+) — PPV	50.00%

⚠️ A positive result gives only a 50% chance of actually having the disease — not 99% — because the tiny 1% base rate is swamped by false positives from the 99% healthy majority. That gap is the base-rate fallacy.

📚 Go Further

Rigorous references for Bayesian inference.

Type	Resource
Book	Gelman et al., Bayesian Data Analysis — conjugacy, MCMC, hierarchical models
Book	McElreath, Statistical Rethinking — Bayesian modelling with intuition
In-app	Advanced Probability, for the MCMC / Markov-chain machinery
In-app	Estimation Theory, for MAP as regularised MLE

✅ Key takeaways

Posterior ∝ likelihood × prior, normalised by the evidence P(D). The base-rate fallacy is dropping the prior: a 99%-accurate test at 1% prevalence gives only a 50% posterior on a positive result.
Frequentist: θ fixed, data random — a 95% CI is a statement about the procedure's long-run coverage. Bayesian: θ random, data fixed — a 95% credible interval directly supports 'P(θ in interval) = 95%'.
Conjugate priors give closed-form posteriors: Beta–Bernoulli updating is pseudo-counts, exactly additive (Laplace) smoothing in n-gram and Naive-Bayes models.
Without conjugacy, MCMC samples the posterior exactly but slowly (check R̂ and effective sample size); variational inference optimises the ELBO — fast, scalable, but biased and variance-underestimating.
As data grows the likelihood dominates the prior and the posterior concentrates on the MLE (Bernstein–von Mises) — the two paradigms converge in the large-sample limit.

Next: Hypothesis Testing in Depth: Neyman–Pearson, likelihood ratios, power, and multiple testing.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Hypothesis Testing →

Statistics Advanced · P1

Later in Statistics Advanced

Hypothesis Testing

Go up a level