The Little Book of Deep Learning

1. Deep Learning Learns from Data by Minimizing a Loss

Then, training the model consists of computing a value w∗ that minimizes ℒ(w∗).

Learning from data. Deep learning, a subset of machine learning, focuses on models that learn representations directly from data. Instead of hand-coding rules, we collect a dataset of inputs and desired outputs, then train a parametric model to approximate the relationship between them. The model's behavior is modulated by trainable parameters, often called weights.

Formalizing goodness. The goal is to find parameter values that make the model a "good" predictor on unseen data. This is formalized using a loss function, ℒ(w), which measures how poorly the model performs on the training data for a given set of parameters w. Common losses include mean squared error for regression and cross-entropy for classification.

Training is optimization. The core task of training is to find the optimal parameters w* that minimize this loss function. This optimization process is central to deep learning, and the choice of model architecture and training techniques are heavily influenced by the need to make this minimization efficient and effective, especially for complex, high-dimensional data.

2. Efficient Computation on Specialized Hardware is Crucial

The Graphical Processing Units (GPUs) have been instrumental in the success of the field by allowing such computations to be run on affordable hardware.

Hardware acceleration. Deep learning involves massive computations, primarily linear algebra operations on large datasets. The parallel architecture of GPUs, originally designed for graphics, proved exceptionally well-suited for these tasks, making large-scale deep learning feasible on accessible hardware. Specialized chips like TPUs have further optimized this.

Memory hierarchy matters. Efficient computation on GPUs requires careful data management. The bottleneck is often data transfer between CPU and GPU memory, and within the GPU's own memory hierarchy. Processing data in batches that fit into fast GPU memory minimizes these transfers, allowing parallel computation across samples.

Tensors are key. Data, model parameters, and intermediate results are organized as tensors, multi-dimensional arrays. Deep learning frameworks manipulate tensors efficiently, abstracting away low-level memory details and enabling complex operations like reshaping and extraction without costly data copying. This tensor-based approach is fundamental to achieving high computational throughput.

3. Gradient Descent and Backpropagation Drive Training

The combination of this computation with the procedure of gradient descent is called backpropagation.

Minimizing the loss. Since the loss function for deep models is usually complex without a simple closed-form solution, gradient descent is the primary optimization algorithm. It starts with random parameters and iteratively updates them by moving a small step in the direction opposite to the gradient of the loss, which is the direction of steepest decrease.

Stochastic updates. Computing the exact gradient over the entire dataset is computationally prohibitive. Stochastic Gradient Descent (SGD) uses mini-batches of data to compute a noisy but unbiased estimate of the gradient, allowing for many more parameter updates for the same computational cost. This mini-batch approach is standard practice, often enhanced by optimizers like Adam.

Backpropagation computes gradients. Backpropagation is the algorithm that efficiently computes the gradient of the loss with respect to all model parameters. It works by applying the chain rule of calculus backward through the layers of the network, computing gradients layer by layer. This backward pass, combined with the forward pass that computes the model's output, forms the core computational loop of deep learning training.

4. Depth and Scale Unlock Powerful Capabilities

There is an accumulation of empirical results showing that performance... improves with the amount of data according to remarkable scaling laws...

The value of depth. Deep models, composed of many layers, can learn more complex and hierarchical representations than shallow ones. While theoretically a single-layer network can approximate any function, deep architectures are empirically shown to achieve state-of-the-art performance across diverse domains, requiring tens to hundreds of layers.

Scaling laws. A key finding is that model performance often improves predictably with increased scale: more data, more parameters, and more computation. This has driven the trend towards increasingly massive models, trained on enormous datasets, leading to breakthroughs like Large Language Models.

Benefits of scale. Large models, despite their immense capacity, often generalize well, challenging traditional notions of overfitting. Their scale, combined with distributed training techniques like SGD on massive datasets, allows them to capture intricate patterns and knowledge that smaller models cannot, albeit at significant computational and financial cost.

5. Deep Models are Built from Reusable Layers

We call layers standard complex compounded tensor operations that have been designed and empirically identified as being generic and efficient.

Modular components. Deep models are constructed by stacking or connecting various types of layers, which are reusable, parameterized tensor operations. This modularity simplifies model design and allows for the creation of complex architectures from well-understood building blocks.

Core layer types:

• Linear/Fully Connected: Perform affine transformations (matrix multiplication + bias).
• Convolutional: Apply localized, shared affine filters across spatial or temporal dimensions, capturing local patterns and enabling translation invariance.
• Activation Functions: Introduce non-linearity (e.g., ReLU, GELU) essential for learning complex mappings.
• Pooling: Reduce spatial size by summarizing local regions (e.g., max pooling).
• Normalizing Layers: Stabilize training by normalizing activation statistics (e.g., Batch Norm, Layer Norm).
• Dropout: Regularize by randomly setting activations to zero during training.
• Skip Connections: Allow signals to bypass layers, aiding gradient flow and training of very deep networks.

Engineering for optimizability. Many layer designs, like skip connections and normalization layers, were developed specifically to mitigate training challenges like the vanishing gradient problem, shifting focus from generic optimization to designing models that are inherently easier to optimize.

6. Attention Mechanisms Connect Distant Information

Attention layers specifically address this problem by computing an attention score for each component of the resulting tensor to each component of the input tensor, without locality constraints...

Beyond locality. While convolutional layers excel at processing local information, many tasks require integrating information from distant parts of a signal, such as understanding dependencies between words far apart in a sentence or relating objects in different parts of an image. Attention layers provide a mechanism for this global interaction.

Query, Key, Value. The core attention operator computes scores representing the relevance of each "query" element to every "key" element, typically using dot products. These scores are then used to compute a weighted average of "value" elements, effectively allowing each query to "attend" to relevant information across the entire input sequence.

Multi-Head Attention. The Multi-Head Attention layer enhances this by performing multiple attention computations in parallel ("heads") with different learned linear transformations for queries, keys, and values. The results from these heads are concatenated and linearly combined, allowing the model to jointly attend to information from different representation subspaces at different positions. This mechanism is a cornerstone of modern architectures like the Transformer.

7. Key Architectures Tackle Different Data Structures

The architecture of choice for such tasks, which has been instrumental in recent advances in deep learning, is the Transformer...

MLPs for simple data. The Multi-Layer Perceptron (MLP), a stack of fully connected layers with activations, is the simplest deep architecture. While theoretically universal approximators, they are impractical for high-dimensional structured data due to excessive parameters and lack of inductive bias.

ConvNets for grids. Convolutional Networks (ConvNets) are the standard for grid-like data such as images. They use convolutional and pooling layers to build hierarchical, translation-invariant feature representations, often culminating in fully connected layers for tasks like classification. Architectures like LeNet and ResNet (which incorporates skip connections for depth) are prime examples.

Transformers for sequences. Transformers, built primarily on attention layers, have become dominant for sequence data like text and increasingly for images. Their ability to model long-range dependencies globally, combined with positional encodings to retain sequence order, makes them highly effective. The encoder-decoder structure for translation and the decoder-only GPT for generation are key variants.

8. Deep Learning Excels at Prediction Tasks

A first category of applications... requires predicting an unknown value from an available signal.

Mapping input to output. Prediction tasks involve using a deep model to estimate a target value or category based on an input signal. This is the classic supervised learning setup, where the model is trained on pairs of inputs and corresponding ground truths.

Diverse applications:

• Image Classification: Assigning a single label to an image (e.g., ResNets, ViT).
• Object Detection: Identifying objects and their bounding boxes in an image (e.g., SSD, using ConvNet backbones).
• Semantic Segmentation: Classifying every pixel in an image (often uses ConvNets with downscaling/upscaling and skip connections).
• Speech Recognition: Converting audio signals into text sequences (e.g., Transformer-based models like Whisper).
• Reinforcement Learning: Learning optimal actions in an environment to maximize reward (e.g., DQN using ConvNets to estimate state-action values).

Leveraging pre-training. For tasks with limited labeled data, models pre-trained on large, related datasets (like image classification or language modeling) can be fine-tuned, significantly improving performance by leveraging learned general representations.

9. Deep Learning Enables Complex Synthesis

A second category of applications distinct from prediction is synthesis.

Modeling data distributions. Synthesis tasks involve generating new data samples that resemble a training dataset. This requires the model to learn the underlying probability distribution of the data, rather than just mapping inputs to outputs.

Text generation. Autoregressive models, particularly large Transformer-based models like GPT, are highly successful at generating human-like text. Trained to predict the next token in a sequence, they learn complex linguistic structures and world knowledge, enabling coherent and contextually relevant text generation, including few-shot learning capabilities.

Image generation. Diffusion models are a powerful recent approach to image synthesis. They learn to reverse a gradual degradation process (like adding noise) that transforms data into a simple distribution. By starting with random noise and iteratively applying the learned denoising steps, they can generate high-quality, diverse images, which can often be conditioned on text descriptions or other inputs.

10. The Field Extends Beyond Core Models and Supervised Learning

Such models constitute one category of a larger class of methods that fall under the name of self-supervised learning, and try to take advantage of unlabeled datasets.

Beyond standard architectures. While MLPs, ConvNets, and Transformers are prominent, other architectures exist for different data types, such as Recurrent Neural Networks (RNNs) for sequences (historically important) and Graph Neural Networks (GNNs) for non-grid data like social networks or molecules.

Learning representations. Autoencoders, including Variational Autoencoders (VAEs), focus on learning compressed, meaningful latent representations of data, useful for dimensionality reduction or generative modeling. Generative Adversarial Networks (GANs) use a competitive process between a generator and discriminator to produce realistic samples.

Self-supervised learning. A major trend is leveraging vast amounts of unlabeled data through self-supervised learning. Models are trained on auxiliary tasks where the "label" is derived automatically from the data itself (e.g., predicting masked parts of an input). This pre-training learns powerful general representations that can then be fine-tuned on smaller labeled datasets for specific downstream tasks, reducing reliance on expensive human annotation.

Last updated: May 17, 2025