Free preview

59 lessons

Essential Linear Algebra for ML

Free preview

Essential Linear Algebra for ML · 59 lessons

Master the linear algebra that powers modern ML

No surprise gaps

Actually remember it

Skip what you know

One subscription. All learning paths included.

Our content is best on a larger screen

Interpreting the SVD

From the unit circle to the ellipse

Explanation

To see the geometry of the singular value decomposition, we start with the unit circle in $\mathbb{R}^2$ : the set of all vectors of length one. The reason for focusing on the unit circle is simple. If we want to understand how a matrix acts on all directions at once, we can apply it to every point on the circle. The resulting shape captures the global effect of the transformation.

When we apply a matrix to the unit circle, the resulting set of points is always an ellipse (or a degenerate ellipse if the matrix has rank one or zero). This ellipse is the geometric fingerprint of the matrix: it shows exactly how the transformation stretches and rotates space.

Why does the unit circle become an ellipse?

Details

Take a $2 \times 2$ matrix $A$ and a unit vector $\vec{x}$ on the circle. The resulting vector is $\vec{y} = A \vec{x}$ . As $\vec{x}$ traces out the circle, $\vec{y}$ traces out a closed curve. Because $A$ is linear, this curve is always an ellipse centred at the origin.

If $A$ is a pure scaling, the circle becomes a larger or smaller circle.
If $A$ is diagonal with unequal entries, the circle becomes an axis‑aligned ellipse.
If $A$ is a rotation, the circle stays a circle (a special case of an ellipse).
If $A$ is a shear (a transformation that slants shapes without changing their area), the circle is tilted, but the boundary curve is still an ellipse.

So the ellipse is not a special case - it is the universal outcome of applying a linear transformation to the unit circle.

This circle‑to‑ellipse picture is the stage on which the SVD plays out. The singular values will turn out to be the lengths of the ellipse’s axes, and the singular vectors will be the directions of those axes.

Visualisation

Have a play with the interactive visualisation below to get a feel for how $A$ turns a circle into an ellipse. The visualisation shows the unit circle and the result of $A\vec{v}$ for all vectors $\vec{v}$ that lie on the unit circle for any $2 \times 2$ matrix $A$ .

0,0

Adjust the entries of the matrix $A$ . The unit circle (red) is mapped by $A$ into the ellipse (blue) with axes shown by the dashed blue lines.

Why visualise the unit circle and not a grid?

Details

It might seem natural to use a grid of points to see how a matrix acts, since we are interested in how $A$ affects every point. But because $A$ is linear, we do not need to test every point. Once we know how $A$ acts on all directions, we know how it acts everywhere. The unit circle is exactly the set of all directions, each normalised to length $1$ .

A grid, by contrast, is not rotation‑invariant. If we draw the grid aligned with the axes, apply $A$ , and look at the result, we get one picture. If we had drawn the grid rotated by $45^\circ$ before applying $A$ , the deformed pattern would look different, even though the matrix $A$ is the same. The visualisation would depend on how we chose to draw the grid, not just on $A$ .

The unit circle avoids this problem. It looks the same in every orientation, so when we apply $A$ the result is always a well‑defined ellipse. That ellipse is uniquely determined by $A$ , and its axes and lengths correspond exactly to the singular vectors and singular values. This makes the unit circle the natural probe for the geometry of the SVD.

Geometry of the SVD

Explanation

The singular value decomposition tells us that any matrix $A$ can be written as

A = U \Sigma V^T,

where $U$ and $V$ are orthogonal matrices and $\Sigma$ is diagonal with non‑negative entries.

Geometrically, this means that applying $A$ to the unit circle always happens in three clean steps.

$V^T$ rotates the circle. Since rotations preserve circles, the unit circle remains a circle, but its coordinate axes are re‑oriented.
$\Sigma$ stretches the circle into an ellipse, with the singular values giving the lengths of the ellipse’s axes.
$U$ rotates the ellipse into its final position in the output space.

So the unit circle always becomes an ellipse, and the SVD gives the universal recipe: rotate → stretch → rotate.

This geometric view explains why the SVD is so powerful. The singular values are the lengths of the ellipse’s axes. The right singular vectors (the columns of $V$ ) are the input directions that get stretched. The left singular vectors (the columns of $U$ ) are the output directions of the ellipse’s axes.

Unlike eigendecomposition, which only works in special cases, the SVD always produces this clean orthogonal picture.

What about higher dimensions?

Details

In two dimensions, we saw that the unit circle always becomes an ellipse. The same idea works in higher dimensions:

In 3D, the unit sphere becomes an ellipsoid.
In 4D and beyond, the unit hypersphere becomes a hyper‑ellipsoid.

The story is unchanged:

$V^T$ rotates the hypersphere so that the right singular vectors line up with the axes.
$\Sigma$ stretches along those axes by the singular values.
$U$ rotates the resulting ellipsoid into its final orientation.

If some singular values are zero, the ellipsoid collapses into a lower‑dimensional shape (for example, a flat ellipse inside a plane, or even a line segment).

So the geometry of the SVD is always rotate → stretch → rotate, no matter how many dimensions we are in.

Visualisation

With an animation, we can make this process vivid and interactive: it allows us to modify $U$ , $\Sigma$ and $V$ (which compose $A$ ) and see the impact of the linear transformation $A\vec{v} = U \Sigma V^T \vec{v}$ for all vectors $\vec{v}$ which lie on the unit circle. At each stage, the geometry is simple and transparent.

0,0

{\vec{e}}_{1}

{\vec{e}}_{2}

Visualising the SVD

Matrices are updated live as you move the sliders.

U = [\begin{matrix} 1.00 & 0.00 \\ 0.00 & 1.00 \end{matrix}], Σ = [\begin{matrix} 2.00 & 0 \\ 0 & 1.00 \end{matrix}], V = [\begin{matrix} 1.00 & 0.00 \\ 0.00 & 1.00 \end{matrix}]

A = U Σ V^{T} = [\begin{matrix} 2.00 & 0.00 \\ 0.00 & 1.00 \end{matrix}]

Step 1: Unit circle

We begin with the unit circle (red) and the standard basis vectors, ${\vec{e}}_{1}$ and ${\vec{e}}_{2}$ . The unit circle allows us to visualise all unit directions at once.

Use the slider below to rotate input space - this is an angle of rotation. It corresponds exactly to changing the entries in

V

whilst keeping it an orthogonal matrix.

0^{\circ}

The stretch factors are the non-negative singular values

σ_{1}, σ_{2}

- they control how much we stretch along the purple and green axes.

σ_{1}

: 2.0

σ_{2}

: 1.0

Use the slider below to rotate output space - this is an angle of rotation. It corresponds exactly to changing the entries in

U

whilst keeping it an orthogonal matrix.

0^{\circ}

How this compares to diagonalisation

Details

In diagonalisation we write

A = P D P^{-1}.

Which we can understand as:

First, $P^{-1}$ changes into the eigenbasis.
Then $D$ stretches along the eigendirections.
Finally, $P$ changes back into the original basis.

So we return to the same basis we started in.

In the SVD we write

A = U \Sigma V^T.

Which we can understand as:

First, $V^T$ changes into the right singular vector basis.
Then $\Sigma$ stretches along those axes.
Finally, $U$ changes into the left singular vector basis.

Here we don’t “change back” - we move from one orthonormal basis (the right singular vectors) to another (the left singular vectors).

This is why the SVD works even when the input and output spaces have different dimensions: the bases at the start and end don’t have to be the same.

Practice questions

4 questions

Suppose $\vec{x}$ is a unit vector on the unit circle in $\mathbb{R}^2$ . What happens when we apply $V^T$ to $\vec{x}$ in the SVD $A = U \Sigma V^T$ ?

Select the correct answer:

$V^T \vec{x} = \vec{x}$ , since orthogonal matrices leave unit vectors unchanged.

$V^T \vec{x}$ scales $\vec{x}$ by the singular values.

$V^T \vec{x} = \vec{0}.$

$V^T \vec{x}$ is another unit vector, rotated (or reflected) on the unit circle.

+ 3 more questions

Zero singular values

Explanation

So far we have seen the geometry of the SVD when both singular values are positive: the unit circle becomes a full ellipse. But what if one of the singular values is zero, or if the matrix is rectangular? In those cases, the ellipse collapses in one or more directions. This is exactly what zero singular values record, and it is the next piece of the story.

Rank deficiency (true zero singular values)

Even within the $\min(m,n)$ singular values on $\Sigma$ 's main diagonal, some may be zero if the matrix is not full rank. This means that the transformation actively collapses certain directions completely.

For example, a $3 \times 2$ matrix of rank $1$ has only one linearly independent column. This means the second column is a multiple of the first, so the matrix cannot span two independent directions. In the SVD, this shows up as one positive singular value $\sigma_1$ and one zero singular value $\sigma_2 = 0$ . On top of that, there is the extra zero row in $\Sigma$ to make it $3 \times 2$ .

So the single non‑zero singular value reflects the one independent direction in the columns of $A$ , while the zero singular value reflects the redundancy.

Geometrically, the ellipse collapses into a line segment: only one axis survives, the other is squashed flat.

Rank and zero singular values

Tip

The number of non‑zero singular values of a matrix is equal to its rank.

So if a matrix has rank $r$ , then it has exactly $r$ positive singular values, and the remaining $\min(m,n) - r$ singular values are zero.

Visualisation

Have a play with the interactive visualisation below. Note that here we’ve fixed $U = V = I$ , so $A = \Sigma$ . That means when a singular value is zero, we see a whole row or column of zeros in $A$ .

In a typical matrix $A$ , rank deficiency doesn’t usually appear as literal zeros - instead, one row (or one column) is a linear combination of the others. The SVD makes this structure explicit by rotating into a basis where the dependency shows up as a zero in $\Sigma$ . Either way, the geometry is the same: a zero singular value means one direction is completely collapsed.

0,0

Watching the ellipse collapse

Drag the sliders to change the singular values. Watch how the ellipse collapses when a singular value goes to zero.

U = [\begin{matrix} 1.00 & 0.00 \\ 0.00 & 1.00 \end{matrix}], Σ = [\begin{matrix} 2.00 & 0 \\ 0 & 1.00 \end{matrix}], V = [\begin{matrix} 1.00 & 0.00 \\ 0.00 & 1.00 \end{matrix}]

A = U Σ V^{T} = [\begin{matrix} 2.00 & 0.00 \\ 0.00 & 1.00 \end{matrix}]

Singular values

σ_{1}, σ_{2}

control the axis lengths of the ellipse.

σ_{1}

: 2.0

σ_{2}

: 1.0

Aside: rectangular matrices

Even when a matrix is full rank, if it is rectangular then $\Sigma$ must be padded with zeros. This is not just padding for shape - it reflects a real limitation: a matrix can only stretch space in at most $\min(m,n)$ independent directions.

That is why, when we computed the SVD of a rectangular matrix $A$ , we had to add extra orthonormal vectors to complete $U$ and $V$ , and pad $\Sigma$ with zeros.

The extra vectors in $U$ or $V$ point in directions where the transformation does nothing, and the matching zeros in $\Sigma$ record this fact.

Example: transforming a vector

Details

Think about it in terms of the shape of the transformation for a vector $\vec{v}$ from applying $A$ :

A \vec{v} = U \Sigma V^T \vec{v}.

A $3 \times 2$ matrix takes $2$ ‑dimensional input vectors and maps them into $3$ ‑dimensional space. But the input only has two independent directions to begin with. That means there can only be two non‑zero singular values, so $\Sigma$ is $3 \times 2$ with one extra row of zeros. The third column of $U$ is still needed to make $U$ a full $3 \times 3$ orthogonal matrix, but its singular value is zero - meaning that in that direction, the matrix does nothing.

Geometrically: the column space of $A$ is at most $2$ -dimensional in $\mathbb{R}^3$ .
A $2 \times 3$ matrix takes $3$ ‑dimensional input vectors and maps them into $2$ ‑dimensional space. The output can only span two independent directions. This means there can only be two non‑zero singular values, so $\Sigma$ is $2 \times 3$ with one extra column of zeros. The third column of $V$ is still needed to make $V$ a full $3 \times 3$ orthogonal matrix, but its singular value is zero - meaning that any component of the input in that direction is “lost” when we apply $A$ .

Geometrically: any input component along the direction of $\vec{v}_3$ is lost, as $\vec{v}_3$ lies in the null space of $A$ .

The same reasoning applies to any rectangular matrix: the number of non‑zero singular values is limited by the smaller of the input and output dimensions.

Key Point

The SVD always maps the unit circle (or hypersphere) to an ellipse (or ellipsoid).

The positive singular values give the lengths of its axes.
The zero singular values show exactly which axes collapse.

This is why the SVD is universal: it makes both the stretching and the limitations of any matrix completely explicit.

Practice questions

5 questions

Consider the matrix

A = \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}.

From the SVD we find that

V =\dfrac{1}{\sqrt{5}} \begin{bmatrix} 1 & -2 \\ 2 & 1 \end{bmatrix}, \quad \Sigma = \begin{bmatrix} 5 & 0 \\ 0 & 0 \end{bmatrix}.

Now take an arbitrary input vector

\vec{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}.

When we compute $\vec{y} = V^T \vec{x} = \begin{bmatrix} y_1 \\ y_2 \end{bmatrix}$ , the coordinates $y_1$ and $y_2$ represent how much of $x$ lies along the right singular vectors $\vec{v}_1$ and $\vec{v}_2$ . Multiplying by $\Sigma$ gives

\vec{z} = \Sigma \vec{y} = \begin{bmatrix} 5 y_1 \\ 0 \end{bmatrix}.

Notice that the second entry of $\vec{z}$ is always zero, regardless of the input vector $\vec{x}$ .

What is the correct interpretation of this fact?

Select the correct answer:

The second raw coordinate of the input vector, $x_2$ , is always discarded by the matrix.

The component of the input along the second right singular vector $\vec{v}_2$ is lost, so only the projection of $\vec{x}$ onto $\vec{v}_1$ survives.

The second column of $A$ is a multiple of the first, so the SVD forces the second column of $V$ to be zero, which explains the zero in $\vec{z}$ .

The zero appears because $\Sigma$ must always have one zero row in a $2 \times 2$ decomposition.

+ 4 more questions