Features, parameters and hyperparameters

When a prediction changes, the cause may be a different input, an updated coefficient, or a different training configuration. Features, parameters and hyperparameters identify these three levels. A small linear model is sufficient to track each level separately before applying the same distinction to a neural network.

Features, parameters and hyperparameters describe different roles in a learning procedure. Features represent an input; parameters determine the model's mapping; hyperparameters configure the training process or model family. Their distinction depends on the specified optimization procedure, rather than on whether a quantity happens to be stored as a number.

1. Features and prediction parameters

For a feature vector $x\in\mathbb R^d$, weights $w\in\mathbb R^d$ and scalar bias $b$, linear regression predicts:

$$\hat y=w^\top x+b=\sum_{j=1}^{d}w_jx_j+b$$

The entries of $x$ describe the current sample. The $d+1$ quantities in $(w,b)$ specify the shared prediction rule. Changing a sample changes $x$ without necessarily changing the model; fitting changes $(w,b)$ using a training objective. [1]

For $x=(2,-1)^\top$, $w=(3,4)^\top$ and $b=0.5$, the prediction is:

$$\hat y=3\cdot2+4\cdot(-1)+0.5=2.5$$

Changing the second weight from four to five changes the prediction to $1.5$. This example isolates a parameter change while leaving the input fixed. A different input under the original weights would instead illustrate a change in activation or output.

2. Batch representation and activations

A dataset containing $n$ feature vectors can be arranged as the rows of $X\in\mathbb R^{n\times d}$. Batch prediction is then:

$$\hat y=Xw+b\mathbf1_n\in\mathbb R^n$$

Here $\mathbf1_n$ is an all-ones vector. Consider the following batch and weights:

$$X=\begin{bmatrix}2&-1\\0&1\end{bmatrix},\qquad w=\begin{bmatrix}3\\4\end{bmatrix}$$

With $b=0.5$, the result is $(2.5,4.5)^\top$. There are three model parameters and two output values. Increasing the number of samples increases the number of outputs, but does not increase the number of shared parameters.

The same distinction applies inside a neural network. An activation is an intermediate value produced for a particular input. A weight is a coefficient reused across computations. Both may be arrays, but their mathematical roles differ. [1]

3. Fixed and trainable features

A fixed feature transformation might be $\phi(x)=(x_1,x_2,x_1x_2)^\top$. Its third coordinate introduces a nonlinear interaction between the original coordinates. A linear predictor on $\phi(x)$ can therefore represent functions that are not affine in the original $x$.

The transformation is nevertheless fixed if its construction does not change during fitting. By contrast, $\phi_\alpha(x)$ has adjustable parameters $\alpha$. End-to-end learning can optimize both the representation and the final prediction function. Feature construction and parameter learning are consequently related but distinct decisions.

4. Hyperparameters and model selection

Let $\lambda$ collect a configuration such as the learning rate, hidden width, regularization coefficient or training duration. The output of a finite training procedure can be represented as:

$$\theta_\lambda=\operatorname{Train}(\mathcal D_{\mathrm{train}};\lambda)$$

This notation includes the procedure and its configuration; it does not assume exact minimization. Given a finite candidate set $\Lambda$, validation-based selection has the form:

$$\lambda^*\in\arg\min_{\lambda\in\Lambda} \widehat R_{\mathrm{val}}(\theta_\lambda)$$

Hyperparameters are outside the ordinary parameter updates of the specified training run. They need not be selected manually: a separate search or optimization procedure may choose them. The distinction concerns the level of optimization being described. [2]

5. Regularization as an explicit example

For batch targets $y\in\mathbb R^n$, a regularized linear-regression objective is:

$$J(w,b)=\frac{1}{2n}\|Xw+b\mathbf1_n-y\|_2^2 +\frac{\rho}{2}\|w\|_2^2,\qquad \rho\ge0$$

The first term measures mean squared residual with a factor of $1/2$. The second penalizes weight magnitude; the bias is not penalized in this definition. The model parameters are $w,b$, while $\rho$ controls the relative strength of the penalty.

Differentiating the penalty gives $\rho w$, which is added to the unregularized weight gradient. Thus a hyperparameter can directly affect parameter updates without itself being updated by that same rule. A complete specification must identify the adjustable variables, fixed quantities and selection procedure; labels alone are insufficient.

References

• 1: Google ML Crash Course, Linear Regression
• 2: Google ML Crash Course, Hyperparameters