ConditionalConditionalProbability,Probability,Bayes'Bayes'RuleRule
ConditionalConditionalProbabilityProbability
Conditional probability is the probability of an event occurring given that another event has already occurred. Observing an event allows us to update our belief about the probabilities of all other events. The rule for these updates is called Bayes' rule. To build intuition for conditional probability, we will start with examples.
Apple trees on a farm in Saint-Joseph-du-Lac:$$ \begin{array}{l|cc|c} & \text { Soft } & \text { Hard } & \\ \hline \text { Red } & 500 & 2500 & 3000 \\ \text { Green } & 1500 & 500 & 2000 \\ \hline & 2000 & 3000 & 5000 \end{array}$$What is the probability that a randomly selected apple tree bears red apples given that it bears hard apples?
Given the observation that the tree bears hard apples, we can restrict our attention to the second column of the table. We can see that there are 3000 hard apples in total, of which 2500 are red. Therefore, the probability that a randomly selected apple tree bears red apples given that it bears hard apples is$$P(R | H)=\frac{2500}{3000}=0.833 \quad \Rightarrow \quad 83.3\% $$Notice the notation $P(R | H)$ for this conditional probability. There are two events,$\ R$ and $H$, and the vertical bar $|$ separates the event we are interested in$\ R$ from the event that we are conditioning on $H$.
Notice the same conditional probability can be computed as a ratio of probabilities (instead of ratio of counts):$$P(R | H)=\frac{P(R\cap H)}{P(H)} = \frac{2500/5000}{3000/5000}=0.833 $$
Let us compute a few more conditional probabilities:$$P(S | R)=\frac{P(S\cap R)}{P(R)} = \frac{500/5000}{3000/5000}=0.167 $$$$P(R | S)=\frac{P(S\cap R)}{P(S)} = \frac{500/5000}{2000/5000}=0.25 $$
This example is based on sample of fish in the Orinoco river. The sample contains 200 fish. The labels of the column headings are as follows:
$ P(Y | O) = \frac{1}{70} = 0.0143: $ The probability that an Omnivore has an empty stomach is $1.43\% $.
$ P(Y | D) = \frac{12}{50} = 0.24: $ The probability that a Detritivore has an empty stomach is $24\% $.
$ P(N | P) = \frac{35}{40} = 0.875: $ The probability that an Piscivore has a non-empty stomach is $87.5\% $.
$ P(O | Y) = \frac{1}{42} = 0.0238: $ The probability that a fish that has an empty stomach is Omnivore is $2.38\% $.
$ P(D | Y) = \frac{12}{42} = 0.286: $ The probability that a fish that has an empty stomach is Detritivore is $28.6\% $.
$ P(P | N) = \frac{35}{158} = 0.222: $ The probability that a fish that has an non-empty stomach is Piscivore is $22.2\% $.
IndependenceIndependence
Consider the following data on eye color and gender.$$ \begin{array}{l|ccc|c} & \text{ Brown } & \text{ Blue } & \text{Green} \\ \hline \text{ M } & 40 & 30 & 10 & 80 \\ \text{ F } & 60 & 45 & 15 & 120 \\ \hline & 100 & 75 & 25 & 200 \end{array}$$We can compare unconditional and conditional probabilities to check for independence.$$ P(Br) = \frac{100}{200} = 0.5, \; P(Br | M) = \frac{40}{80} = 0.5, \; P(Br | F) = \frac{60}{120} = 0.5 $$$$ P(Bl) = \frac{75}{200} = 0.375, \; P(Bl | M) = \frac{30}{80} = 0.375, \; P(Bl | F) = \frac{45}{120} = 0.375 $$$$ P(Gr) = \frac{25}{200} = 0.125, \; P(Gr | M) = \frac{10}{80} = 0.125, \; P(Gr | F) = \frac{15}{120} = 0.125 $$This sample supports independence of eye color and gender. This is indeed approximately true in reality. Later in the course we will develop a robust test for independence, which takes into account the randomness of the sample.
Bayes'Bayes'RuleRule
Bayes' ruleis a way to update our beliefs about the probability of an event given new information. It is a fundamental tool in data analysis, in statistics, in cognitive science and in machine learning. It is named after the Reverend Thomas Bayes, who first formulated it in the 18th century. The rule is a direct consequence of the definition of conditional probability.$$P(A | B)=\frac{P(A\cap B)}{P(B)}$$To obtain the rule, note that the event $B$ can be written as the union of $A\cap B$ and $A'\cap B$, where $A'$ is the complement of $A$. Indeed, any sample point in $B$ is either in $A$ and $B$ or in $A'$ and $B$. Correspondingly,$$P(B)=P(A\cap B)+P(A'\cap B) = P(B|A)P(A) + P(B|A')P(A')$$Substituting the second equation into the first, we get Bayes' rule:$$P(A | B)=\frac{P(B|A)\ P(A)}{P(B|A)\ P(A)+P(B|A')\ P(A')}$$Notice that Bayes' rule allows us to update our belief about the probability of $A$ given that $B$ has occurred: $P(A)$ is updated to $P(A | B)$.