Gradient descent and backpropagation

A prediction error is a single number, whereas a network may contain many adjustable coefficients. Training needs a way to determine how each coefficient contributes to that error. The gradient supplies the local sensitivities, backpropagation computes them efficiently through shared intermediate calculations, and an optimizer turns them into parameter updates.

Gradient descent updates parameters using a loss derivative. Backpropagation computes derivatives by applying the chain rule through a computation graph. These are distinct operations: one determines gradients, while the other uses them in an update rule.

1. A scalar objective

Consider $\hat y=wx$ with fixed $x=2$ and target $y=6$. Using squared loss without a factor of $1/2$ gives:

$$\mathcal L(w)=(2w-6)^2=4(w-3)^2$$

The derivative follows from the outer square and the inner affine function:

$$\mathcal L'(w)=2(2w-6)\cdot2=8w-24, \qquad \mathcal L''(w)=8$$

At $w=1$, the loss is 16 and the derivative is $-16$. Increasing $w$ by a sufficiently small amount therefore lowers the loss; the update calculation below determines how the chosen step size affects the result.

2. Updates and convergence conditions

Gradient descent with fixed learning rate $\eta$ uses:

$$w_{t+1}=w_t-\eta \mathcal L'(w_t)$$

For $w_0=1$ and $\eta=0.1$, this gives $w_1=2.6$, prediction $5.2$ and loss $0.64$. With $\eta=1$, the update instead gives $w_1=17$ and loss 784.

The convergence condition can be derived exactly for this quadratic. Defining the error relative to the minimizer as $e_t=w_t-3$ yields:

$$e_{t+1}=(1-8\eta)e_t$$

For a nonzero initial error, convergence requires $|1-8\eta|<1$, equivalent to:

$$0<\eta<\frac14$$

At $\eta=1/8$, the error becomes zero after one update. The stability interval follows from the curvature eight and the specified loss scaling. [1]

3. The local descent argument

For a differentiable scalar objective $J(\theta)$ and a small displacement $\Delta$:

$$J(\theta+\Delta) =J(\theta)+\nabla J(\theta)^\top\Delta +o(\|\Delta\|)$$

Substituting $\Delta=-\eta\nabla J(\theta)$ makes the linear term $-\eta\|\nabla J(\theta)\|_2^2$. At a nonstationary point this term is negative for $\eta>0$. The remainder is negligible relative to the displacement only in the local limit, so finite-step behavior still depends on step size. At a stationary point, the first-order argument alone establishes no strict decrease.

4. Batch linear-regression gradients

Let $X\in\mathbb R^{n\times d}$, $w\in\mathbb R^d$, $b\in\mathbb R$ and $y\in\mathbb R^n$. Define:

$$r=Xw+b\mathbf1_n-y,\qquad J(w,b)=\frac{1}{2n}r^\top r$$

Unlike the scalar example, this objective includes $1/2$. Expanding the sum of squared residuals and differentiating each term gives:

$$\frac{\partial J}{\partial w_j} =\frac1n\sum_i r_iX_{ij},\qquad \nabla_wJ=\frac1nX^\top r,\qquad \frac{\partial J}{\partial b}=\frac1n\mathbf1_n^\top r$$

Adding $\rho\|w\|_2^2/2$ adds $\rho w$ to the weight gradient. A minibatch version replaces the full-data average with a selected batch average. The differentiation is unchanged in form, while the statistical properties of the estimated gradient depend on the sampling procedure.

5. Backpropagation through a hidden layer

A scalar-output network with $m$ hidden units can be specified by:

$$z=\mathbf{W}x+b,\quad a=\operatorname{ReLU}(z),\quad \hat y=v^\top a+c,\quad \ell=\frac12(\hat y-y)^2$$

Here $x\in\mathbb R^d$, $\mathbf{W}\in\mathbb R^{m\times d}$, $b,v\in\mathbb R^m$, and $c,y\in\mathbb R$. Let $r=\hat y-y$. Differentiation from the output gives:

$$\nabla_v\ell=ra,\qquad \partial_c\ell=r$$

Propagating through the activation and first affine layer gives:

$$\delta=(rv)\odot\mathbf1[z>0],\qquad \nabla_\mathbf{W}\ell=\delta x^\top,\quad \nabla_b\ell=\delta,\quad \nabla_x\ell=\mathbf{W}^\top\delta$$

Here $\odot$ denotes coordinatewise multiplication. ReLU has no classical derivative at zero; assigning zero to its backward multiplier there is an implementation convention. Away from the kink, the mask is the ordinary derivative. The weight gradient has shape $m\times d$, as required for an update of $\mathbf{W}$.

This reverse accumulation is backpropagation. A rule such as $\mathbf{W}\leftarrow \mathbf{W}-\eta\nabla_\mathbf{W}\ell$ is the subsequent optimization step. During ordinary inference, parameters remain fixed and only the forward computation is required. [2]

References

• 1: Google ML Crash Course, Gradient Descent
• 2: PyTorch, Automatic Differentiation