Free preview

109 lessons

Essential Probability & Statistics for ML

Free preview

Essential Probability & Statistics for ML · 109 lessons

Master the fundamentals

No surprise gaps

Actually remember it

Skip what you know

One subscription. All learning paths included.

Our content is best on a larger screen

The variance of a random variable

Spread about the mean

Explanation

The PMF of a random variable gives its distribution. Often we want to summarise a distribution in a single number. The mean, or expected value, is one such summary: it is what the random variable averages over many repetitions. But two different random variables can have the same mean and very different distributions. What single number measures how spread out a distribution is?

Since XX averages to its mean over many repetitions, a natural measure of spread is how far XX lies from its mean. We write μ\mu, the Greek letter mu, for E[X]\mathbb{E}[X], and take the squared distance (X−μ)2(X - \mu)^2. As a function of XX, it is a random variable, so it has an expected value.

Variance

Definition

For a random variable XX with mean μ\mu, the variance of XX is the expected value of its squared distance from μ\mu,

Var⁡(X)=E[(X−μ)2].\operatorname{Var}(X) = \mathbb{E}\big[(X - \mu)^2\big].

So over many repetitions, the average squared distance of XX from μ\mu is close to Var⁡(X)\operatorname{Var}(X).

Applying the definition of the expected value gives

Var⁡(X)=∑x(x−μ)2 p(x).\operatorname{Var}(X) = \sum_x (x - \mu)^2\,p(x).

The steps below show what this means for the distribution given by a PMF.

0
1
2
3
4
0.1
0.2
0.3
0.4
0.5
0.6
μ=2
0
1
2
3
4
0.1
0.2
0.3
0.4
0.5
0.6
μ=2
X
Y

Step 1: One mean, two distributions

X takes 1,2,3 and Y takes 0,2,4, each with probabilities 0.25,0.5,0.25. Both have mean μ=2, but the values of Y lie further from it.

Step 2: The squared distance from the mean

The middle value of each lies at the mean, at distance 0. The outer values of X lie at distance 1, so their squared distance is 1. The outer values of Y lie at distance 2, so their squared distance is 4.

Step 3: Far values count most

Each value adds its squared distance times its probability. The outer bars have the same heights in both charts, but those of Y are at squared distance 4 rather than 1, so Y has the larger variance.

So probability far from μ makes the variance large, and probability at μ adds nothing.

Why square the distance?

Details

Practice questions

4 questions

A random variable XX has Var⁡(X)=9\operatorname{Var}(X) = 9. Which of these must be true?

Select the correct answer:

+ 3 more questions

Properties of variance

Explanation

We've seen that variance measures how spread out a distribution is about its mean. What can we say about it without calculating it?

Its definition as an expected squared distance from the mean gives several useful properties:

PropertyWhy
Var⁡(X)≥0\operatorname{Var}(X) \ge 0Squared distances are never negative.
Var⁡(X)=0\operatorname{Var}(X) = 0 only when XX always takes one valueThe squared distances are all 00 only when XX always equals its mean.
Var⁡(X+b)=Var⁡(X)\operatorname{Var}(X + b) = \operatorname{Var}(X)Adding a number moves every value and the mean together, so the distances from the mean stay the same.
Var⁡(aX)=a2Var⁡(X)\operatorname{Var}(aX) = a^2 \operatorname{Var}(X)Multiplying by a number multiplies each distance from the mean by the size of aa, so each squared distance is multiplied by a2a^2.

Combining the last two gives the variance of any scaled and shifted random variable:

The variance of aX+baX + b

Theorem

For a random variable XX and numbers aa and bb,

Var⁡(aX+b)=a2Var⁡(X)\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)

Practice questions

4 questions

A random variable XX has Var⁡(X)=9\operatorname{Var}(X) = 9. What is Var⁡(X+10)\operatorname{Var}(X + 10)?

Select the correct answer:

+ 3 more questions