Bayes' theorem

Which is larger, $P(A \mid B)$ or $P(B \mid A)$ ?

Explanation

One card is drawn from a shuffled deck. Is it more likely to be red given that it is an ace, or an ace given that it is red?

Half of the aces are red, but very few red cards are aces. The diagram below finds both conditional probabilities from the same four regions.

0,0

Ω

A

B

Aces (4)

Red (26)

2

2

24

24

Step 1: The four regions

Each region of $Ω$ is labelled with its number of cards: aces that are not red $2$ , red aces $2$ , red cards that are not aces $24$ , and neither $24$ . So there are $4$ aces and $26$ red cards.

Both conditional probabilities use the same overlap, the $2$ red aces.

Step 2: Given an ace

We learn that the card is an ace, so every card outside the aces is ruled out, and the $4$ aces become the whole. The overlap is half of it:

P (red ∣ ace) = \frac{2}{2 + 2} = \frac{2}{4} = \frac{1}{2}

Step 3: Given red

Now we learn instead that the card is red, so every card outside red is ruled out, and the $26$ red cards become the whole. The same overlap is only a small part of it:

P (ace ∣ red) = \frac{2}{2 + 24} = \frac{2}{26} = \frac{1}{13}

Step 4: Same overlap, different wholes

Both conditional probabilities divide the same overlap, $2$ cards, by the number of cards in the event we are given. The overlap is half of the small aces circle but only a small part of the large red circle.

So the conditional probability given the rarer event, ace, is the larger.

The same holds for any events $A$ and $B$ . The conditional probability formula gives $P(A \mid B)$ and $P(B \mid A)$ the same numerator:

P(A \mid B) = \dfrac{P(A \cap B)}{P(B)} \qquad P(B \mid A) = \dfrac{P(A \cap B)}{P(A)}

Dividing the same numerator by the smaller denominator gives the larger probability:

Theorem

For events $A$ and $B$ that can both happen, if $P(A) < P(B)$ then

P(B \mid A) > P(A \mid B)

and if $P(A) = P(B)$ then the two are equal.

In words, the conditional probability given the rarer event is the larger. Deciding needs only which event is rarer, not the value of $P(A \cap B)$ .

Practice questions

4 questions

Events $A$ and $B$ can both happen, with $P(A) = 0.1$ and $P(B) = 0.4$ .

Which is true?

Select the correct answer:

$P(A \mid B) > P(B \mid A)$

$P(A \mid B) = P(B \mid A)$

$P(B \mid A) > P(A \mid B)$

It cannot be decided without $P(A \cap B)$ .

+ 3 more questions

Bayes' rule

Explanation

Often we know the probability of the evidence given a cause, such as how often a fire makes smoke, but want the probability of the cause given the evidence: how likely a fire is, given smoke. Bayes' rule turns one of these conditional probabilities into the other.

It is one of the most used results in probability, behind spam filters, medical diagnosis and models that update their probabilities as data arrives.

It needs nothing new, only the multiplication rule for probabilities and the conditional probability formula.

The overlap $A \cap B$ can be reached from either event, so the multiplication rule writes its probability in two ways:

P(A \cap B) = P(A \mid B)\,P(B) \qquad P(A \cap B) = P(B \mid A)\,P(A)

Both right-hand sides equal the same probability, so they equal each other:

P(A \mid B)\,P(B) = P(B \mid A)\,P(A)

Dividing both sides by $P(B)$ gives Bayes' rule, also called Bayes' theorem:

Bayes' rule

Theorem

For events $A$ and $B$ with $P(A) > 0$ and $P(B) > 0$ ,

P(A \mid B) = \dfrac{P(B \mid A)\,P(A)}{P(B)}

In words, Bayes' rule finds $P(A \mid B)$ from the reverse conditional probability, $P(B \mid A)$ , and the probability of each event.

Example

In a town, $0.1$ of days are rainy, and $0.5$ of rainy days start cloudy. Of all days, $0.6$ do not start cloudy. Given that a day starts cloudy, find the probability that it is rainy.

Solution

We name the events for a day chosen at random:

$R$ : the day is rainy, with $P(R) = 0.1$ .
$C$ : the day starts cloudy.

The $0.5$ is a share of rainy days, so $P(C \mid R) = 0.5$ . We want $P(R \mid C)$ , the reverse conditional probability, so we use Bayes' rule:

P(R \mid C) = \dfrac{P(C \mid R)\,P(R)}{P(C)}

We are given the probability that a day does not start cloudy, so the complement rule gives the denominator:

P(C) = 1 - P(C^c) = 1 - 0.6 = 0.4

We substitute the three values:

\begin{align*} P(R \mid C) &= \dfrac{0.5 \times 0.1}{0.4} \\[6pt] &= \dfrac{0.05}{0.4} \\[6pt] &= 0.125 \end{align*}

Given that a day starts cloudy, it is rainy with probability $0.125$ .

Practice questions

4 questions

On $0.01$ of days there is a fire in a warehouse, and on $0.1$ of days its smoke alarm sounds. The alarm sounds on $0.9$ of the days with a fire.

Given that the alarm sounds on a day, what is the probability that there is a fire?

Select the correct answer:

$0.9$

$0.09$

$0.009$

$0.1$

+ 3 more questions

Prior, likelihood and posterior

Explanation

Bayes' rule describes how a probability changes when we learn something. It turns the probability of an event $A$ before we learn that $B$ happened into its probability after. Each part of the rule has a name for its role in this update.

Prior, likelihood and posterior

Definition

In Bayes' rule, with $A$ the event we ask about and $B$ the event we learn,

P(A \mid B) = \dfrac{P(B \mid A)\,P(A)}{P(B)}

The prior is $P(A)$ , the probability of $A$ before we learn $B$ .
The likelihood is $P(B \mid A)$ , the probability of what we learn, given $A$ .
The posterior is $P(A \mid B)$ , the probability of $A$ after we learn $B$ .

The denominator $P(B)$ , the probability of what we learn, has no name here.

"Prior" and "posterior" mean before and after learning $B$ : the prior is where we start, and the posterior is where we end up once $B$ is known.

The likelihood is a probability of $B$ , not of $A$ : it says how probable what we learned is if $A$ holds. Bayes' rule multiplies the prior by a factor built from it:

P(A \mid B) = P(A) \times \dfrac{P(B \mid A)}{P(B)}

The factor compares how probable $B$ is when $A$ holds with how probable $B$ is overall. If it is above $1$ , learning $B$ raises the probability of $A$ ; if it is below $1$ , learning $B$ lowers it.

ML Context

In machine learning, a model makes predictions using numbers called parameters, whose values are learned from data. Write $\theta$ for one choice of parameter values and $D$ for the observed data. Bayes' rule then reads

P(\theta \mid D) = \dfrac{P(D \mid \theta)\,P(\theta)}{P(D)}

Many models are fitted by maximum likelihood: among all choices of $\theta$ , it picks the one with the largest likelihood $P(D \mid \theta)$ , the one that makes the observed data most probable. The prior plays no part.

Bayesian inference uses the whole rule. It starts from a prior $P(\theta)$ , saying which values are plausible before seeing the data, and finds the posterior $P(\theta \mid D)$ : how plausible each value is afterwards, not only which one fits best.

Example

Of the items a factory makes, $0.04$ come from a machine with a fault. Of the items from a faulty machine, $0.5$ are defective, and of all items, $0.1$ are defective. An item is found to be defective. For the probability that it came from a faulty machine, name the prior, likelihood and posterior.

Solution

We start with the event asked about. We learn that the item is defective and ask whether it came from a faulty machine, so:

$A$ : the item came from a faulty machine.
$B$ : the item is defective.

The fault came before the defect, but the names follow the event we ask about, not the order in which events happen, so the fault is $A$ .

The $0.04$ and the $0.1$ are shares of all items, and the $0.5$ is a share of items from a faulty machine:

P(A) = 0.04 \qquad P(B) = 0.1 \qquad P(B \mid A) = 0.5

Then we label the roles:

The prior is $P(A) = 0.04$ .
The likelihood is $P(B \mid A) = 0.5$ .
The posterior is $P(A \mid B)$ , the probability that a defective item came from a faulty machine.

The $0.1$ is $P(B)$ and has no name here.

Practice questions

4 questions

In a region, $0.01$ of houses flood each year. Of the houses that flood, $0.7$ stand by the river, and of all houses, $0.1$ stand by the river.

We learn that a house stands by the river and ask how likely it is to flood. What is the likelihood?

Select the correct answer:

$0.1$ , the probability that a house stands by the river.

$0.7$ , the probability that a house stands by the river given that it floods.

$0.01$ , the probability that a house floods.

The probability that a house floods given that it stands by the river.

+ 3 more questions

Master the fundamentals

Explanation

Step 1: The four regions

Step 2: Given an ace

Step 3: Given red

Step 4: Same overlap, different wholes

Explanation

Example

Explanation

Example

Bayes' theorem and evidence