Probability and Statistics: A Practical Guide to Models, Inference, and Computation

Probability theory begins with a probability space (Ω, F, P). Here Ω contains possible outcomes, F is a sigma-algebra of measurable events, and P assigns probabilities satisfying nonnegativity, P(Ω) = 1, and countable additivity for disjoint events. The sigma-algebra specifies which collections of outcomes have probabilities. A random variable X is a measurable function from outcomes to numbers, allowing events such as {X ≤ x} to be assigned probabilities. In practice, defining the observational unit, possible outcomes, and sampling mechanism often matters more than choosing a familiar formula.

Conditional probability updates the reference population: P(A | B) = P(A ∩ B)/P(B), provided P(B) > 0. Suppose a hypothetical machine alarm detects 90% of faults and activates on 5% of nonfaulty cycles, while faults occur on 1% of cycles. Bayes’ rule gives P(fault | alarm) = 0.90 × 0.01 / (0.90 × 0.01 + 0.05 × 0.99), approximately 0.154. An alarm therefore indicates a fault only about 15.4% of the time under these assumptions. This base-rate effect explains why sensitivity alone cannot establish an alarm’s operational usefulness.

Probability distributions describe how random variables allocate probability. A discrete variable has a probability mass function; an absolutely continuous variable has a density f whose integral over an interval gives its probability. The cumulative distribution function F(x) = P(X ≤ x) applies to both. For a continuous X, E[g(X)] = ∫g(x)f(x) dx when the expectation exists, and Var(X) = E[X²] − E[X]² when the second moment is finite. A density value is not a point probability and can exceed one; its total integral must equal one.

For a worked derivation, let X have an exponential distribution with rate λ > 0, so f_X(x) = λe^(−λx) for x ≥ 0, and define Y = X². Because X is nonnegative, F_Y(y) = P(X ≤ √y) = 1 − e^(−λ√y) for y ≥ 0. Differentiating yields f_Y(y) = λe^(−λ√y)/(2√y) for y > 0. Although this density diverges near zero, it remains integrable. Moreover, E[Y] = E[X²] = 2/λ². This change-of-variables technique helps derive distributions of transformed waiting times, energies, and error measures.

Real systems usually produce several measurements together. A random vector X = (X₁, …, Xₚ) has a joint distribution that captures both marginal behavior and dependence. For a joint density f(x, y), integrating over y gives the marginal density of X; dividing f(x, y) by a positive marginal density f_X(x) gives the conditional density of Y given X = x. Covariance measures linear co-variation, while correlation standardizes it. Independence implies zero covariance when the relevant moments exist, but zero covariance does not generally imply independence. Jointly Gaussian variables are an important exception.

Suppose two unbiased sensors measure the same fixed quantity. Their error variances are 4 and 9, and their error covariance is 1, all in consistent squared units. The simple average has error variance (4 + 9 + 2 × 1)/4 = 3.75, not the 3.25 obtained by incorrectly assuming independence. For a weighted estimate wX₁ + (1 − w)X₂, the variance is 4w² + 9(1 − w)² + 2w(1 − w). Differentiation gives the minimizing weight w = 8/11. Accounting for dependence directly changes how measurements should be combined.

Statistical inference depends on the sampling distribution of an estimator: the distribution it would have across repeated samples generated by the same design. For independent, identically distributed observations with finite mean μ, the law of large numbers says the sample mean converges to μ. With finite, positive variance σ², the central limit theorem states that √n(X̄ − μ)/σ converges in distribution to a standard normal variable. This does not say the observations themselves become normally distributed. Strong dependence, heavy tails, and small samples can make routine normal approximations unreliable.

Suppose 64 independent measurements yield a sample mean of 12.4 and sample standard deviation of 2.4. The estimated standard error is 2.4/√64 = 0.3. If the population is normal with unknown variance, an exact 95% confidence interval uses a t critical value with 63 degrees of freedom: 12.4 ± approximately 2.00 × 0.3, or about [11.80, 13.00]. For nonnormal populations this is generally an approximation. Confidence intervals describe a procedure’s long-run coverage, not a 95% frequentist probability that this particular fixed parameter lies inside the observed interval.

Likelihood treats observed data as fixed and measures their compatibility with candidate parameter values. For independent Bernoulli observations with success probability p, the likelihood is proportional to p^s(1 − p)^(n − s), where s is the number of successes. Maximizing it yields p̂ = s/n. Estimators should be judged by bias, variance, consistency, and suitability for the sampling design—not convenience alone. In regression, likelihood or least squares links outcomes to predictors, but coefficient interpretation still depends on functional form, confounding, measurement quality, and whether observations are independent.

Hypothesis testing asks how incompatible a statistic is with a specified null model. Suppose an independent sample contains 56 successes in 100 trials, and H₀ states p = 0.5. The normal-approximation statistic is z = (0.56 − 0.50)/√(0.5 × 0.5/100) = 1.2, giving a two-sided p-value of approximately 0.23 without a continuity correction. An exact binomial test gives a somewhat different result. Neither p-value is the probability that H₀ is true. Report effect sizes and uncertainty, define the alternative in advance, and address multiplicity when testing many claims. Nonrejection is not proof of equivalence.

Bootstrap resampling approximates an estimator’s sampling behavior by repeatedly sampling with replacement from the observed data and recomputing the statistic. For independent observations, a practical workflow draws B samples of the original size, calculates one estimate per sample, and uses their empirical standard deviation as a bootstrap standard error. Percentile intervals use quantiles of those estimates, although their coverage can be poor for biased or skewed estimators. Bootstrap validity depends on the statistic, sampling process, and regularity conditions; it is not a universal substitute for mathematical analysis.

For a reproducible exercise, use the illustrative observations [8, 9, 10, 10, 13]. Their mean is 10. In Python with NumPy, create rng = np.random.default_rng(42), generate samples with rng.choice(x, size=(10000, len(x)), replace=True), and compute their row means. The resulting spread estimates uncertainty in the mean, conditional on using the empirical distribution as a population approximation. With only five observations, conclusions remain fragile regardless of the number of replicates. Resample whole participants for repeated measurements, clusters for clustered samples, or suitable blocks for time series. Permutation tests instead rearrange labels under a null-supported exchangeability assumption.

Bayesian methods combine a likelihood with a prior distribution to obtain a posterior: p(θ | data) ∝ p(data | θ)p(θ). For Bernoulli data, a Beta(a, b) prior produces a Beta(a + s, b + n − s) posterior. With a Beta(2, 2) prior and 14 successes in 20 trials, the posterior is Beta(16, 8), whose mean is 16/24, approximately 0.667. This is also the posterior predictive probability of success on the next exchangeable trial. A 95% credible interval can be obtained from posterior quantiles and has a conditional probability interpretation under the specified model.

More complex Bayesian models often require numerical integration or Markov chain Monte Carlo. Check chain mixing, convergence diagnostics, and effective sample sizes before interpreting simulations. Across Bayesian and frequentist workflows, begin by documenting the estimand, collection process, missingness, and dependence structure. Visualize distributions, examine residuals or posterior predictive simulations, and test sensitivity to consequential assumptions. Reserve held-out data when evaluating predictive performance, and avoid preprocessing that leaks test information into training. Reproducible analysis preserves code, random seeds, software versions, and a clear record of decisions; defensible conclusions explain both what the model establishes and what the data cannot resolve.