Mathematical notation

The articles use elementary algebra, finite-dimensional linear algebra, derivatives and discrete probability. This appendix defines the shared operations and conventions. Article-specific symbols are defined at their point of use. The following recurring symbols are reserved across the series.

Symbol	Meaning
$L_{\mathrm{net}}$	Network depth, measured in successive layers
$\mathcal L$, $\ell$	Loss objective and single-example or single-position loss
$J$	Optimization objective, including regularization when specified
$H,W$	Scalar image height and width
$\mathbf{W}$, $\mathbf{W}_l$	Weight matrix, including a layer index where needed
$\mathbf{W}_Q,\mathbf{W}_K,\mathbf{W}_V$	Attention projection matrices
$w$	Weight vector or scalar, according to the stated shape
$V$, $V_h$	Vocabulary size and attention value matrix

Scalar image width $W$ and matrix weights $\mathbf{W}$ differ typographically as well as by shape. Other matrices retain their conventional letters, such as $A,B,E$. Shapes are stated locally whenever an object is introduced.

Displayed equations carry no terminal sentence punctuation. The preceding prose ends with a colon, and the following prose starts a new sentence. Internal commas separating mathematical declarations remain part of the notation.

1. Vector and matrix shapes

A single vector is a column $x\in\mathbb R^d$, and $x^\top$ denotes its transpose. For $A\in\mathbb R^{m\times d}$, the product $y=Ax\in\mathbb R^m$ is defined coordinatewise by:

$$y_i=\sum_{j=1}^d A_{ij}x_j$$

For $A\in\mathbb R^{m\times d}$ and $B\in\mathbb R^{d\times n}$, matrix multiplication follows the same summation rule:

$$(AB)_{ik}=\sum_{j=1}^d A_{ij}B_{jk},\qquad AB\in\mathbb R^{m\times n}$$

Inner dimensions must match. Matrix multiplication is generally not commutative. The map $Ax+b$ is affine and becomes linear when $b=0$. The notation $I_d$ denotes the $d\times d$ identity matrix, $\mathbf1_n$ an all-ones vector, and $u\odot v$ coordinatewise multiplication.

Batch feature vectors are arranged as rows of $X\in\mathbb R^{n\times d}$, giving predictions $Xw+b\mathbf1_n$. Image tensors and sequence tensors require separately declared axis conventions.

2. Norm, rank and null space

The Euclidean norm and its square are:

$$\|x\|_2=\sqrt{\sum_jx_j^2},\qquad \|x\|_2^2=x^\top x$$

Matrix rank is the maximum number of linearly independent columns. The null space is $\ker A=\{v:Av=0\}$. For a matrix with $d$ columns, rank–nullity states $d=\operatorname{rank}(A)+\dim\ker A$.

A brief justification starts with a basis of the null space and extends it to a basis of the input space. The images of the added basis vectors span the image of $A$ and are independent: a dependence among those images would place a nontrivial combination of the added vectors in the null space, contradicting independence of the extended basis. Their count is therefore the rank, giving the stated identity.

Orthogonality means zero dot product. For orthogonal $u,v$, expansion of $(u+v)^\top(u+v)$ eliminates the cross terms and gives $\|u+v\|_2^2=\|u\|_2^2+\|v\|_2^2$.

3. Derivatives, gradients and the chain rule

A scalar derivative is defined by the difference-quotient limit:

$$f'(t)=\lim_{h\to0}\frac{f(t+h)-f(t)}h$$

For the square function, $((t+h)^2-t^2)/h=2t+h$, which tends to $2t$. A partial derivative $\partial J/\partial\theta_j$ varies one coordinate while holding the others fixed. The column of these derivatives is the gradient $\nabla J$.

For $y=f(x)$, the Jacobian entry in row $i$, column $j$ is $\partial y_i/\partial x_j$. A scalar loss obeys:

$$\frac{\partial \mathcal L}{\partial x_j}=\sum_i\frac{\partial \mathcal L}{\partial y_i}\frac{\partial y_i}{\partial x_j},\qquad \nabla_xL=J_f(x)^\top\nabla_yL$$

Backpropagation applies this rule in reverse computational order. A derivative describes local sensitivity; a finite parameter update additionally specifies a step size.

4. Exponentials and logarithms

The natural exponential is $e^t$, with inverse $\log t$ for $t>0$. The identity $\log(ab)=\log a+\log b$ converts probability products to sums. Derivatives $(e^t)'=e^t$ and $(\log t)'=1/t$, combined with the chain rule, give the softmax loss derivative.

Specifically, for $\ell=-z_y+\log\sum_j e^{z_j}$, differentiation of the second term gives $e^{z_k}/\sum_j e^{z_j}$, while the first contributes $-\mathbf1[k=y]$. Their sum is $p_k-\mathbf1[k=y]$.

5. Probability, expectation and variance

A finite distribution satisfies $p_i\ge0$ and $\sum_i p_i=1$. For values $a_i$, its expectation is $\mathbb E[a]=\sum_i p_i a_i$. This probability-weighted quantity differs from an empirical average of observed samples.

For $p(b)>0$, conditional probability is $p(a\mid b)=p(a,b)/p(b)$. Rearranging gives $p(a,b)=p(a\mid b)p(b)$; repeated application factorizes a sequence probability without requiring independence between positions.

Variance is $\operatorname{Var}(Z)=\mathbb E[(Z-\mathbb EZ)^2]=\mathbb E[Z^2]-(\mathbb EZ)^2$. Independence is an additional assumption used, for example, to remove covariance terms in the variance of a sample average.

6. Indicators, indexing and approximations

The indicator $\mathbf1[\text{condition}]$ is one when its condition holds and zero otherwise. The floor $\lfloor t\rfloor$ is the greatest integer no larger than $t$. Array positions and token IDs use zero-based indexing where stated; sample and layer indices may start at one.

The notation $r(\Delta)=o(\|\Delta\|)$ means $r(\Delta)/\|\Delta\|\to0$ as a nonzero displacement approaches zero. It specifies a local remainder and does not justify ignoring that term for a large step. Rounded decimals are approximations; exact fractions and expressions are retained where relevant to a derivation.