Understanding the Bayes Theorem Formula

Bayes' theorem is a fundamental result in probability theory that allows the computation of the probability of a cause, given an observed outcome, by incorporating prior probabilities and conditional likelihoods. This theorem establishes a precise relationship between conditional and marginal probabilities of stochastic events.

Statement and Mathematical Formula of Bayes' Theorem

Let $A_1, A_2, \ldots, A_n$ be a finite partition of the sample space $S$, where each $A_i$ is mutually exclusive and collectively exhaustive, and $\Pr(A_i) > 0$ for each $i = 1, 2, \ldots, n$. Let $B$ be any event such that $\Pr(B) > 0$. Then, for any $j$ $(1 \leq j \leq n)$, Bayes' theorem gives the probability of $A_j$ given $B$ as:

\[ \Pr(A_j|B) = \frac{\Pr(A_j)\Pr(B|A_j)}{\displaystyle\sum_{i=1}^{n} \Pr(A_i)\Pr(B|A_i)} \]

In this equation, $\Pr(A_j)$ is the prior probability of $A_j$, $\Pr(B|A_j)$ is the likelihood of observing $B$ if $A_j$ has occurred, and the denominator is the total probability of observing $B$ across all possible cases. This formula applies directly to cases where the events $A_1, \ldots, A_n$ form a complete partition.

Derivation of Bayes' Theorem for Two Events

Consider two events $A$ and $B$ with $\Pr(A) > 0$ and $\Pr(B) > 0$. The conditional probability of $A$ given $B$ is defined as:

\[ \Pr(A|B) = \frac{\Pr(A \cap B)}{\Pr(B)} \]

Similarly, the conditional probability of $B$ given $A$ is:

\[ \Pr(B|A) = \frac{\Pr(B \cap A)}{\Pr(A)} \]

Observe that $A \cap B = B \cap A$, so $\Pr(A \cap B) = \Pr(B \cap A)$. Thus, substituting for $\Pr(A \cap B)$ from the second equation yields:

\[ \Pr(A|B) = \frac{\Pr(B|A) \Pr(A)}{\Pr(B)} \]

This is the fundamental form of Bayes' theorem for two events. If the sample space admits a partition into more than two events, the denominator is replaced by the total probability of $B$ over the partition.

Total Probability Theorem as the Basis for Bayes' Formula

For mutually exclusive and exhaustive events $A_1, A_2, \ldots, A_n$, and for any event $B$, the probability of $B$ is:

\[ \Pr(B) = \sum_{i=1}^{n} \Pr(A_i)\Pr(B|A_i) \]

This is the law of total probability. Substituting this in the Bayes' formula numerator leads to the general form for partitions.

Bayes' Theorem for Complementary Events

When $A$ and $A'$ are complementary and $\Pr(B) > 0$, the theorem reads:

\[ \Pr(A|B) = \frac{\Pr(A)\Pr(B|A)}{\Pr(A)\Pr(B|A) + \Pr(A')\Pr(B|A')} \]

This form is frequently encountered in binary hypothesis testing and medical diagnostics. For a more detailed treatment of event relations, refer to the Mutually Exclusive vs Independent Events page.

Application: Calculation of Posterior Probabilities Using Bayes' Theorem

Bayes' theorem updates the probability of an underlying cause or hypothesis in the light of new evidence or information. The quantity $\Pr(A_j|B)$ is called the posterior probability of $A_j$ after observing $B$. The value $\Pr(A_j)$ is the prior. This principle is central to inferential reasoning in both classical statistics and Bayesian statistics. For related foundational concepts, consult the Multiplication Theorem of Probability section.

Worked Example: Three Box Problem Using Bayes' Theorem Formula

Given: Three boxes $B_1$, $B_2$, $B_3$ contain the following balls: $B_1$ has $3$ red and $2$ white, $B_2$ has $4$ red and $5$ white, and $B_3$ has $2$ red and $4$ white balls. A box is selected at random and a ball is drawn. The drawn ball is red. Find the probability that $B_2$ was chosen.

Define events:

- $A_1$: Box $B_1$ is chosen

- $A_2$: Box $B_2$ is chosen

- $A_3$: Box $B_3$ is chosen

- $R$: A red ball is drawn

Required: $\Pr(A_2|R)$

First, the probability of selecting each box:

\[ \Pr(A_1) = \Pr(A_2) = \Pr(A_3) = \frac{1}{3} \]

Compute the probability of drawing a red ball from each box:

\[ \Pr(R|A_1) = \frac{3}{5} \]

\[ \Pr(R|A_2) = \frac{4}{9} \]

\[ \Pr(R|A_3) = \frac{2}{6} = \frac{1}{3} \]

By the law of total probability, the probability of drawing a red ball:

\[ \Pr(R) = \Pr(A_1)\Pr(R|A_1) + \Pr(A_2)\Pr(R|A_2) + \Pr(A_3)\Pr(R|A_3) \]

Calculate each term:

\[ \Pr(R) = \frac{1}{3} \times \frac{3}{5} + \frac{1}{3} \times \frac{4}{9} + \frac{1}{3} \times \frac{1}{3} \]

\[ = \frac{3}{15} + \frac{4}{27} + \frac{1}{9} \]

Convert all terms to a common denominator ($135$):

\[ \frac{3}{15} = \frac{27}{135} \]

\[ \frac{4}{27} = \frac{20}{135} \]

\[ \frac{1}{9} = \frac{15}{135} \]

Therefore:

\[ \Pr(R) = \frac{27 + 20 + 15}{135} = \frac{62}{135} \]

Now, using Bayes' theorem:

\[ \Pr(A_2|R) = \frac{\Pr(A_2)\Pr(R|A_2)}{\Pr(R)} \]

\[ \Pr(A_2|R) = \frac{\frac{1}{3} \times \frac{4}{9}}{\frac{62}{135}} \]

\[ = \frac{\frac{4}{27}}{\frac{62}{135}} \]

\[ = \frac{4}{27} \times \frac{135}{62} \]

\[ = \frac{4 \times 135}{27 \times 62} \]

\[ = \frac{540}{1674} \]

\[ = \frac{10}{31} \]

Result: The required probability is $\frac{10}{31}$.

Bayes' Theorem for Sequential Events and Its Computation

If a sequence of dependent draws or transfers is involved, Bayes' theorem still applies by appropriately defining the partition events and the evidence. Each possible history defines an event $A_i$, and the subsequent evidence $B$ is computed conditioned on each $A_i$. For a comprehensive overview of related probability rules, refer to the Statistics and Probability Overview page.

Bayes' Theorem Interpretation in Terms of Prior and Posterior Probabilities

The prior probability of an event (e.g., $\Pr(A)$) represents the belief about the event before any evidence is taken into account. The likelihood ($\Pr(B|A)$) represents the probability of observing the evidence under the assumption that $A$ occurred. The posterior probability ($\Pr(A|B)$) updates the prior in light of the new evidence $B$. This updating mechanism is the core concept of Bayesian inference, which finds application in various scientific, industrial, and data-driven domains.

Formulation for Bayes' Theorem with Three Events

If events $A$, $B$, and $C$ form a partition of $S$, and $D$ is any event with $\Pr(D) > 0$, then for any $X \in \{A, B, C\}$:

\[ \Pr(X|D) = \frac{\Pr(X)\Pr(D|X)}{\Pr(A)\Pr(D|A) + \Pr(B)\Pr(D|B) + \Pr(C)\Pr(D|C)} \]

This extension is crucial for multi-class classification and modeling with more than two causes or hypotheses.

Summary of Bayes' Theorem Formulae

Bayes' theorem expresses the posterior probability $\Pr(A|B)$ as a function of the prior $\Pr(A)$, the likelihood $\Pr(B|A)$, and the marginal probability $\Pr(B)$. In cases with multiple hypotheses, the denominator is the sum over all possible causes of the product of prior and likelihood for each cause. For concise reference to related formulas, the Basic Math Formulas page is suggested.