Free preview

191 lessons

Essential Calculus for ML

Free preview

Essential Calculus for ML · 191 lessons

Master the calculus behind how ML models learn

No surprise gaps

Actually remember it

Skip what you know

One subscription. All learning paths included.

Our content is best on a larger screen

Second-order partial derivatives

The four second-order partials

Explanation

For a function of one variable, differentiating f′f' again gives the second derivative f′′f''. A function of two variables, f(x,y)f(x, y), has two first partials, fxf_x and fyf_y. Each is itself a function of xx and yy, so each can be differentiated by xx or by yy. That gives four second-order partial derivatives, each defined only where both of its steps exist.

Second-order partial derivatives

Definition

A second-order partial derivative of ff is a partial derivative of one of the first partials fxf_x and fyf_y. In subscript notation,

fxy=(fx)y,f_{xy} = \left(f_x\right)_y,

so ff is differentiated by xx, and the result is then differentiated by yy. The two that use both variables, fxyf_{xy} and fyxf_{yx}, are the mixed partial derivatives.

Each of the four has a subscript form and a ∂\partial form:

Subscript form∂\partial formOrder of differentiation
fxx=(fx)xf_{xx} = \left(f_x\right)_x∂2f∂x2\dfrac{\partial^2 f}{\partial x^2}by xx, then by xx
fyy=(fy)yf_{yy} = \left(f_y\right)_y∂2f∂y2\dfrac{\partial^2 f}{\partial y^2}by yy, then by yy
fxy=(fx)yf_{xy} = \left(f_x\right)_y∂2f∂y ∂x\dfrac{\partial^2 f}{\partial y\,\partial x}by xx, then by yy
fyx=(fy)xf_{yx} = \left(f_y\right)_x∂2f∂x ∂y\dfrac{\partial^2 f}{\partial x\,\partial y}by yy, then by xx

Key Point

Subscripts are read left to right, so in fyxf_{yx} we differentiate by yy first. The denominator of the ∂\partial form is read right to left, so in ∂2f∂x ∂y\dfrac{\partial^2 f}{\partial x\,\partial y} we also differentiate by yy first.

The notation extends to more variables. A function of three variables has first partials fxf_x, fyf_y and fzf_z, and fxz=(fx)zf_{xz} = \left(f_x\right)_z is the partial of fxf_x with respect to zz.

Practice questions

4 questions

For a function f(x,y)f(x,y), which of these describes

∂2f∂y ∂x?\dfrac{\partial^2 f}{\partial y\,\partial x}?

Select the correct answer:

+ 3 more questions

What fxxf_{xx} and fyyf_{yy} measure

Explanation

For a function f(x,y)f(x,y) of two variables, the graph z=f(x,y)z=f(x,y) is a surface. The two second-order partials that repeat a variable, fxxf_{xx} and fyyf_{yy}, each describe one cross-section of that surface.

Key Point

Holding y=by=b fixed cuts the surface along the cross-section z=f(x,b)z=f(x,b), which is a function of xx alone. fxx(a,b)f_{xx}(a,b) is the ordinary second derivative of that cross-section at x=ax=a.

Similarly, fyy(a,b)f_{yy}(a,b) is the ordinary second derivative of the cross-section z=f(a,y)z=f(a,y) at y=by=b.

Each is the second derivative of a curve, so its sign gives the concavity of that cross-section.

Toggle the two views below to see the two cross-sections of one surface through the same point, first with yy held fixed and then with xx held fixed.

x
y
z
Holding y=0 fixed picks out the grey plane. Where it cuts the surface z=x2−y2 is the red cross-section z=x2, a curve in x alone. Its second derivative at the origin is 2, so fxx(0,0)=2 and the curve is concave up there. Drag anywhere in the scene to rotate it.

Nothing here is new. Each of fxxf_{xx} and fyyf_{yy} is the second derivative we already know, applied to a cross-section of the surface.

Example

For f(x,y)=x3+xy−y4f(x,y) = x^3 + xy - y^4, a calculation at the point (1,2)(1,2) gives

fxx(1,2)=6,fyy(1,2)=−48.f_{xx}(1,2) = 6, \qquad f_{yy}(1,2) = -48.

Using these values, and without differentiating:

  1. Name the cross-section that fxx(1,2)f_{xx}(1,2) describes, and state its concavity there.
  2. Name the cross-section that fyy(1,2)f_{yy}(1,2) describes, and state its concavity there.

Solution

  1. Both subscripts of fxxf_{xx} are xx, so we hold y=2y = 2 fixed at both steps. That leaves the cross-section z=f(x,2)z = f(x,2), a function of xx alone, and fxx(1,2)f_{xx}(1,2) is its ordinary second derivative at x=1x = 1. That value is positive, so this cross-section is concave up at x=1x = 1.

  2. For fyyf_{yy} we hold x=1x = 1 fixed instead, which leaves the cross-section z=f(1,y)z = f(1,y), a function of yy alone, and fyy(1,2)f_{yy}(1,2) is its ordinary second derivative at y=2y = 2. That value is negative, so this cross-section is concave down at y=2y = 2.

Practice questions

4 questions

A calculation for a function f(x,y)f(x,y) gives fxx(3,−2)=−5f_{xx}(3,-2) = -5. Which statement follows?

Select the correct answer:

+ 3 more questions

What fxyf_{xy} and fyxf_{yx} measure

Explanation

The two remaining second-order partials are the mixed partials, fxyf_{xy} and fyxf_{yx}. Neither is the second derivative of a single cross-section, because the two steps use different variables. Instead, each compares one cross-section with its neighbours.

Intuition

Tip

Imagine the surface z=f(x,y)z=f(x,y) as a mountainside. You are standing on it at the point (a,b)(a,b), facing north, the +y+y direction. The ground in front of you has some steepness, and that steepness is fy(a,b)f_y(a,b), the slope of the cross-section z=f(a,y)z=f(a,y) at y=by=b.

Now take one step to your right, east, the +x+x direction, and face north again. The ground in front of you now belongs to a different cross-section, the one at a slightly larger value of xx, and its steepness can be different.

fyx(a,b)f_{yx}(a,b) measures that change. Since fyx=(fy)xf_{yx} = \left(f_y\right)_x, it differentiates the slope in the yy direction, fyf_y, with respect to xx.

Key Point

fyx(a,b)f_{yx}(a,b) is the rate at which fyf_y, the slope in the yy direction, changes as xx increases. In the same way, fxy(a,b)f_{xy}(a,b) is the rate at which fxf_x, the slope in the xx direction, changes as yy increases.

Slide the cross-section below to watch the slope in the yy direction change with xx.

x
y
z

Move the slider to slide the red cross-section of the surface z=xy along the x axis.

x0.5
Holding x=0.5 fixed, the red cross-section is the line z=0.5y.
Its slope in the y direction is fy(0.5,y)=0.5.

Increase x with the slider and the slope in the y direction increases with it. Decrease x and it decreases.

The rate of that change is the mixed partial fyx. Here fy=x, so fyx=1.

Drag anywhere in the scene to rotate it.

As with any derivative, the sign tells us the direction of the change. A positive fyx(a,b)f_{yx}(a,b) means the slope in the yy direction is increasing as xx increases, and a negative one means it is decreasing. A value of 00 means that, at (a,b)(a,b), moving in the xx direction is not changing the slope in the yy direction.

Example

For a function f(x,y)f(x,y), a calculation at the point (3,1)(3,1) gives

fy(3,1)=2,fyx(3,1)=−5.f_y(3,1) = 2, \qquad f_{yx}(3,1) = -5.

Using these values, and without differentiating:

  1. State which first partial fyx(3,1)f_{yx}(3,1) tracks, and in which direction it tracks the change.
  2. Describe what the two values say about the surface z=f(x,y)z = f(x,y) at (3,1)(3,1).

Solution

  1. Subscripts are read left to right, so fyx=(fy)xf_{yx} = \left(f_y\right)_x: the first partial is fyf_y, and it is differentiated by xx. So fyx(3,1)f_{yx}(3,1) is the rate at which fyf_y changes as xx increases, with yy held fixed at 11.

  2. At (3,1)(3,1) the slope in the yy direction is fy(3,1)=2f_y(3,1) = 2, so the cross-section z=f(3,y)z = f(3,y) is rising at y=1y = 1. The value fyx(3,1)=−5f_{yx}(3,1) = -5 is negative, so that slope is decreasing as xx increases. A small increase in xx brings us to a cross-section that is less steep in the yy direction at y=1y = 1.

Practice questions

4 questions

For a function f(x,y)f(x,y), a calculation gives fxy(4,−1)=3f_{xy}(4,-1) = 3. Which statement follows?

Select the correct answer:

+ 3 more questions