Compression, reconstruction and channel bottlenecks

A narrow intermediate representation can serve two different goals: force a reconstruction model to use a limited code, or reduce the cost of an expensive operation. The later expansion restores an output shape required by the task or surrounding layers. Whether it can also restore the original values depends on what the earlier reduction retained.

Compression followed by expansion serves different purposes in different architectures. A smaller spatial grid, fewer channels and a lower-dimensional vector impose different constraints. The relevant question is which representation is reduced, what the task requires it to preserve, and what information remains available to the subsequent expansion.

1. Encoding and reconstruction

An encoder maps $x\in\mathbb R^d$ to $z=E_\theta(x)\in\mathbb R^m$, and a decoder produces $\hat x=D_\phi(z)$. When $m<d$, the code forms a vector bottleneck. For $n$ samples, a squared reconstruction objective is:

$$J(\theta,\phi)=\frac1n\sum_{i=1}^n\|D_\phi(E_\theta(x_i))-x_i\|_2^2$$

The objective rewards codes that preserve what the decoder needs for reconstruction. Which distinctions survive depends on the code capacity, model family and data distribution. [1]

2. Non-invertibility of a linear bottleneck

For example, $A=(1/2,1/2)$ maps both $(0,2)^\top$ and $(1,1)^\top$ to one. Decoding with $D(z)=(z,z)^\top$ returns $(1,1)^\top$ for either input, giving squared errors two and zero respectively.

For a linear encoder $E(x)=Ax$ with $A\in\mathbb R^{m\times d}$ and $m<d$, rank–nullity gives:

$$\dim\ker A=d-\operatorname{rank}(A)\ge d-m>0$$

A nonzero vector $v$ therefore exists with $Av=0$, implying $A(x+v)=Ax$. Different inputs in $\mathbb R^d$ can have the same code. No decoder receiving only that code can distinguish all such inputs. For a linear decoder $B\in\mathbb R^{d\times m}$, the additional inequality $\operatorname{rank}(BA)\le m<d$ rules out $BA=I_d$.

The proof concerns a linear encoder on the entire input space. A restricted data set may lie in a subspace small enough for exact reconstruction; orthogonal projection provides an explicit example. [1]

3. Orthogonal projection and reconstruction error

Let $U\in\mathbb R^{d\times m}$ have orthonormal columns, so $U^\top U=I_m$. Encoding with $U^\top$ and decoding with $U$ gives:

$$\hat x=UU^\top x,\qquad e=x-\hat x,\qquad U^\top e=0$$

Since $\hat x^\top e=0$, expansion of the squared norm yields:

$$\|x\|_2^2=\|\hat x\|_2^2+\|e\|_2^2$$

The residual is the component outside the selected subspace. Inputs in the subspace are reconstructed exactly, while the squared norm of the discarded component gives the reconstruction error.

4. Spatial downsampling and upsampling

Reducing spatial resolution can lower the cost of subsequent spatial operations and enlarge the spacing between their input dependencies. It can also remove fine positional detail. These effects concern the spatial axes and are not equivalent to reducing channel width at every position.

Upsampling increases spatial resolution through replication, interpolation or a learned mapping. It is an inverse only if composition with the preceding operation recovers every input in the stated domain. The averaging example shows that increasing output size does not resolve an ambiguity already introduced by compression.

Encoder-to-decoder skip connections provide representations that bypass a bottleneck. Concatenation adds channels, whereas addition combines corresponding entries of matching shapes. These paths make earlier spatial detail available to the decoder. [2]

5. Channel bottlenecks and computational cost

Compare a $3\times3$ convolution from 64 to 64 channels with three layers: $1\times1:64\to16$, $3\times3:16\to16$, and $1\times1:16\to64$. Keeping spatial sizes fixed and excluding biases gives:

$$P_{\mathrm{direct}}=9\cdot64^2=36864$$ $$P_{\mathrm{bottleneck}}=64\cdot16+9\cdot16^2+16\cdot64=4352$$

Under these assumptions, the same counts describe MACs per output position. The bottleneck reduces the width of the expensive spatial operation before restoring the external channel size. The complete block is assessed together with its nonlinearities and residual branches. [3]

Reconstruction constraints, spatial resolution and channel-operation cost therefore provide distinct explanations for compression followed by expansion. None establishes a universal requirement that networks must use this pattern.

References

• 1: Deep Learning: Autoencoders
• 2: U-Net
• 3: Deep Residual Learning for Image Recognition