What you are looking at
The ovals are contour lines of a loss with two parameters, x and y, and its lowest point at the cross in the middle: L = ½(x² + 10y²). It is ten times steeper up and down than left and right, the way a real model's loss is far steeper along some parameters than others.
The gradient points uphill, so each step goes the other way: p ← p − lr · ∇L. The panel beside the map shows the position, the gradient there and the size of the last step, and the chart below it shows the loss after every step.
Choosing the learning rate
- Too small (try 0.010): every step is safe and there are hundreds of them. The shallow left-right direction barely moves, and the run stops at 400 steps short of the bottom.
- Larger (try 0.170): it gets there in about thirty steps, but the steep direction overshoots every time, so the path zig-zags across the valley.
- Past 0.2 (try 0.300): each overshoot is bigger than the last and the loss explodes within a few steps. In real training this is usually what “the loss went to NaN” means.
The limit is 2 divided by the steepest curvature, here 10, while the speed of progress is set by the shallowest, here 1. Their ratio is the condition number, and the larger it is, the worse plain gradient descent does: the rate has to be small enough for the steep direction, which leaves the shallow one crawling. Scaling input features to similar ranges helps for this reason: it makes the bowl rounder, so one rate suits every direction.
Momentum
Momentum keeps a running velocity: v ← βv − lr · ∇L, then p ← p + v. On a zig-zag the steep component flips sign every step, so it cancels out in v, while the steady pull along the valley adds up. At a learning rate of 0.17, set momentum to 0.50 and the run takes about half the steps. At 0.90 it overshoots the minimum and circles for a long time before settling: too much momentum is its own problem.
Momentum also moves the limit, to 2(1 + β) divided by the steepest curvature. The red mark on the learning-rate slider follows it.
In real training
You never see the surface: a model has millions of parameters, not two. You see the loss curve, the chart under the map. A curve that falls and then flattens early can usually take a larger rate; one that jumps about or turns into NaN needs a smaller one.
Optimisers such as Adam scale each parameter's step by the size of its recent gradients, which copes with uneven steepness far better than one shared rate. Schedules warm the rate up over the first steps and lower it towards the end. And stochastic gradient descent takes each step from a small batch of examples rather than all of them: noisier steps, but far cheaper ones.
In an interview
Questions this lab prepares you for:
- What does the learning rate do, and how would you choose one?
- Why normalise features before training?
- How do batch, mini-batch and stochastic gradient descent differ?
- What do momentum and Adam add to plain gradient descent?
- Your loss turned into NaN after a few hundred steps. What do you check? (The learning rate first, then exploding gradients, where gradient clipping helps, then bad inputs such as a division by zero or a log of zero.)
The code
The same update the lab runs, in NumPy.
import numpy as np STEEP = 10 def gradient(p): x, y = p return np.array([x, STEEP * y]) def descend(start, lr=0.17, momentum=0.0, steps=400): p = np.array(start, dtype=float) v = np.zeros(2) path = [p.copy()] for _ in range(steps): v = momentum * v - lr * gradient(p) p = p + v path.append(p.copy()) return path