AI, machine learning and deep learning

A learning system must connect three decisions: what to predict, how to represent the input, and how to improve its predictions from examples. Separating these decisions makes the relationship between AI, machine learning and deep learning concrete. The following formulation starts with a prediction task and shows which parts become trainable.

Artificial intelligence (AI), machine learning and deep learning refer to different levels of a technical framework. AI is the broader research field; machine learning comprises methods that infer useful structure from data; deep learning is a branch of machine learning based on multilayer neural networks. A task, a model family and an optimization method are separate concepts: classification specifies an output requirement, a neural network specifies a function family, and gradient descent specifies a parameter-update procedure. [1]

1. Learning as an optimization problem

A supervised dataset consists of input–target pairs:

$$\mathcal D=\{(x_i,y_i)\}_{i=1}^{n},\qquad x_i\in\mathbb R^d$$

Here $n$ is the number of samples and $d$ is the input dimension. A model $f_\theta$ produces the prediction $\hat y_i=f_\theta(x_i)$, with adjustable parameters collected in $\theta$. Given a task-specific loss $\ell(\hat y,y)$, the empirical risk is:

$$\widehat R_{\mathcal D}(\theta) =\frac1n\sum_{i=1}^{n}\ell(f_\theta(x_i),y_i)$$

Training seeks parameters that reduce this objective, possibly with an additional regularization term. Empirical risk measures fit to the observed dataset; generalization evaluates the resulting model on new samples. [2]

2. Classification, regression and generation

The distinction between tasks depends on the meaning and structure of the target. [2]

Task	Target	Example formulation
Regression	A real-valued quantity $y$	$\hat y=w^\top x+b$, with squared loss
Classification	A category $y\in\{1,\ldots,K\}$	Class probabilities $p_\theta(k\mid x)$
Sequence generation	A token sequence	Conditional distributions over successive tokens

For regression, a common loss is $(\hat y-y)^2$. For classification, the negative log-likelihood of the target category is $-\log p_\theta(y\mid x)$. For an autoregressive sequence model, the objective sums negative log-probabilities over target tokens. Although each implementation operates on numbers, these objectives encode different prediction requirements.

Training setup is another distinction. Supervised learning uses supplied targets, whereas self-supervised learning constructs targets from the data itself, such as the next token of an observed sequence. Generation is therefore compatible with self-supervised training; output type and target-construction method are not competing categories.

3. Fixed and learned representations

A model based on fixed features can be written as:

$$\hat y=g_\beta(\phi(x))$$

The representation $\phi$ is specified in advance and the prediction parameters $\beta$ are fitted. When the representation is trainable, the model becomes:

$$\hat y=g_\beta(\phi_\alpha(x)),\qquad \theta=(\alpha,\beta)$$

Optimization can update both $\alpha$ and $\beta$. A fixed-feature classifier already learns $\beta$; representation learning adds $\alpha$ to the optimized variables. This comparison isolates one change in the learning pipeline.

A multilayer representation has the form:

$$h^{(0)}=x,\qquad h^{(l)}=\sigma_l(\mathbf{W}_lh^{(l-1)}+b_l),\quad l=1,\ldots,L_{\mathrm{net}}$$

Here $L_{\mathrm{net}}$ counts successive transformations. The matrices and biases have compatible shapes, and $\sigma_l$ denotes an activation function. Depth describes function composition. Its effect on expressive capacity depends on the transformations involved: a stack of affine maps without intervening nonlinearities remains affine. [1]

4. A two-sample regression example

Consider the constructed samples $(x,y)=(1,2)$ and $(2,4)$ with $f_w(x)=wx$. Their average squared error is:

$$\widehat R(w) =\frac{(w-2)^2+(2w-4)^2}{2} =\frac52(w-2)^2$$

Thus $\widehat R(1)=2.5$ and $\widehat R(2)=0$, with the unique minimizer $w=2$. The next question is whether the fitted relation also describes new samples, which motivates the evaluation procedures in article 5.

5. Model and application system

A trained function $f_\theta$ is typically one component of an application. Preprocessing, retrieval, decoding and output handling can alter system behavior without changing its stored parameters. Consequently, a statement about the model architecture does not fully specify how an application produces an answer.

The relevant analytical objects are the data, representation, model, objective, optimization procedure and evaluation distribution. Keeping these objects distinct provides a consistent basis for comparing traditional learning methods, deep networks and language models.

References

• 1: Deep Learning, Introduction
• 2: Google, What is Machine Learning?