The Reverend Bayes and His Theorem
Thomas Bayes derived his result in the context of a thought experiment about a billiard table. He wanted to know how to reason about an unknown parameter (the position of a ball rolled onto the table) given observed evidence (the outcomes of subsequent rolls). His approach โ assigning a prior probability distribution over unknown quantities, then updating it using observed evidence โ was essentially the complete Bayesian framework, arrived at in one intellectual leap. Bayes' posthumous paper sat largely unnoticed until Pierre-Simon Laplace independently rediscovered and greatly extended the same ideas in the 1810s. Laplace applied Bayesian reasoning to everything from judicial inference to celestial mechanics. The mathematical framework then fell into controversy during the late 19th and early 20th centuries as the frequentist approach (probability as long-run frequency, not degree of belief) became dominant. The modern Bayesian revival began in the 1950s and accelerated dramatically with the development of computational methods โ particularly Markov Chain Monte Carlo โ in the 1990s.
Medical Testing and the Base Rate Fallacy
The most striking application of Bayes' theorem is in medical testing, where it reveals an almost universally underappreciated truth: even a highly accurate test can produce mostly false positives if the underlying condition is rare. This result has profound implications for public health policy, medical screening programmes, and individual diagnosis decisions. Screening for rare diseases with even very accurate tests produces mostly false positives โ causing anxiety, unnecessary follow-up procedures, and resource waste. Bayes' theorem explains why, and provides the quantitative framework for deciding when screening is worthwhile.
Bayesian vs Frequentist: The Philosophical Divide
The frequentist interpretation of probability defines it as the long-run frequency of an event in repeated trials. A frequentist cannot assign a probability to a one-off event ("probability that it rained in Rome on July 4, 1893" โ it either did or it didn't). The Bayesian interpretation defines probability as a degree of belief, which can be assigned to any proposition regardless of whether it is repeatable. This allows Bayesian methods to answer questions that frequentist methods cannot: "given this data, what is the probability that the true parameter is greater than X?" Frequentist methods (p-values, confidence intervals, maximum likelihood estimation) remain dominant in many fields because they are computationally simpler and don't require specifying a prior. But Bayesian methods have become increasingly practical as computing power grew, and they are increasingly preferred in complex models, small samples, and situations where incorporating prior information is valuable.
MCMC: Computing Posteriors Numerically
For most real problems, the posterior distribution cannot be computed analytically โ the integrals required are intractable. Markov Chain Monte Carlo (MCMC) methods โ Metropolis-Hastings, Gibbs sampling, Hamiltonian Monte Carlo โ generate samples from the posterior distribution by constructing a Markov chain whose stationary distribution is the posterior. Given enough samples, expectations, credible intervals, and predictions can be estimated to arbitrary precision. Modern probabilistic programming languages (Stan, PyMC, Turing.jl) make MCMC accessible without requiring users to implement the algorithms themselves. EngForge's hypothesis testing and Monte Carlo tools let you run frequentist tests and compare them with simulation-based approaches โ building intuition for how sample size, effect size, and prior assumptions affect statistical conclusions.