Deep Learning

1. Deep Learning: Data-Driven Decision Making

Deep learning enables data-driven decisions by identifying and extracting patterns from large datasets that accurately map from sets of complex inputs to good decision outcomes.

Data-driven decisions. Deep learning excels at extracting patterns from vast datasets, enabling accurate mappings from complex inputs to desired outcomes. This makes it ideal for applications where intuition falls short and data reigns supreme. Examples include:

• Facebook's text analysis in online conversations
• Google, Baidu, and Microsoft's image search and machine translation
• Self-driving cars' environment perception and motion planning

AlphaGo's success. DeepMind's AlphaGo, a program that defeated world champion Go players, exemplifies deep learning's power. Go's immense search space made it computationally challenging, but deep learning algorithms allowed AlphaGo to evaluate board configurations and make strategic decisions.

Decision-making is key. The ability to make data-driven decisions is crucial in many domains. Deep learning provides a means to identify and extract patterns from large datasets, enabling accurate mappings from complex inputs to good decision outcomes.

2. AI, ML, and DL: A Hierarchy of Intelligence

The modern field of machine learning draws on the last two topics: computers that could learn from examples, and neural network research.

Nested fields. Artificial intelligence (AI) is the overarching field, encompassing machine learning (ML), which in turn encompasses deep learning (DL). AI aims to create intelligent systems, ML focuses on algorithms that learn from data, and DL utilizes deep neural networks.

AI's origins. The field of AI was born at a workshop at Dartmouth College in 1956. Research presented at the workshop included mathematical theorem proving, natural language processing, planning for games, computer programs that could learn from examples, and neural networks.

ML's focus. Machine learning involves developing algorithms that enable a computer to extract (or learn) functions from a dataset (sets of examples). To understand what machine learning means we need to understand three terms: dataset, algorithm, and function.

3. Machine Learning: Extracting Functions from Data

A function is a deterministic mapping from a set of input values to one or more output values.

Deterministic mappings. A function is a deterministic mapping from inputs to outputs, meaning that for any specific set of inputs, it will always return the same outputs. The goal of machine learning is to learn these functions from data.

Datasets and algorithms. Machine learning algorithms analyze datasets to identify recurring patterns, which are then represented as functions. These functions can be simple arithmetic operations, if-then-else rules, or more complex representations like neural networks.

Neural networks as functions. Deep learning, a subfield of machine learning, focuses on deep neural network models. The patterns that deep learning algorithms extract from datasets are functions that are represented as neural networks.

4. The Difficulty of Machine Learning: Noise and Bias

First, most datasets will include noise in the data, so searching for a function that matches the data exactly is not necessarily the best strategy to follow, as it is equivalent to learning the noise.

Noise and ill-posed problems. Machine learning faces challenges due to noise in data and the fact that the set of possible functions is often larger than the set of examples in the dataset, making it an ill-posed problem.

Inductive bias. To overcome these challenges, machine learning algorithms supplement the information provided by the data with a set of assumptions about the characteristics of the best function, known as the inductive bias of the algorithm.

Underfitting and overfitting. Choosing the wrong inductive bias can lead to underfitting (the function is too simple) or overfitting (the function fits the noise in the data). Finding the right balance between data and inductive bias is key to successful generalization.

5. Supervised, Unsupervised, and Reinforcement Learning

In supervised machine learning, each example in the dataset is labeled with the expected output (or target) value.

Supervised learning. In supervised machine learning, each example in the dataset is labeled with the expected output (or target) value. The algorithm learns by comparing its outputs with the target outputs and adjusting its parameters accordingly.

Unsupervised learning. Unsupervised machine learning is generally used for clustering data. In unsupervised machine learning, there are no target values in the dataset. Instead, the algorithm tries to identify functions that map similar examples into clusters.

Reinforcement learning. Reinforcement learning is most relevant for online control tasks, such as robot control and game playing. In these scenarios, an agent needs to learn a policy for how it should act in an environment in order to be rewarded.

6. Mathematical Models: Equations Describing Relationships

In its simplest form, a mathematical model is an equation that describes how one or more input variables are related to an output variable.

Models as equations. A mathematical model is an equation that describes how input variables relate to an output variable. It's a simplified representation of a real-world process.

Linear models. A simple template for a model is the equation of a line: y = mx + c, where y is the output, x is the input, m is the slope, and c is the intercept. These parameters can be adjusted to fit the model to the data.

Model usefulness. For a model to be useful it must have a correspondence with the real world. This correspondence is most obvious in terms of the meaning that can be associated with a variable.

7. Linear Models: Weighted Sums and Multiple Inputs

The multiplication of inputs by weights, followed by a summation, is known as a weighted sum.

Weighted sums. The core of a linear model is that the output is calculated as the sum of the n input values multiplied by their corresponding weights. This calculation is known as a weighted sum.

Multiple inputs. The equation of a line can be scaled to models with multiple inputs by adding a new weight for each input variable. The output is then calculated as the sum of the input values multiplied by their corresponding weights.

Learning from data. Machine learning helps by finding the parameters (or weights) of a model using a dataset. The learning done by machine learning is finding the parameters (or weights) of a model using a dataset.

8. Neural Networks: Interconnected Neurons

The power of neural networks to model complex relationships is not the result of complex mathematical models, but rather emerges from the interactions between a large set of simple neurons.

Simple units, complex networks. A neural network consists of a network of simple information processing units, called neurons. The power of neural networks to model complex relationships emerges from the interactions between a large set of simple neurons.

Layers of neurons. Neurons in a neural network are organized into layers: an input layer, hidden layers, and an output layer. Deep learning networks are neural networks that have many hidden layers of neurons.

Connections and weights. Each connection in a network connects two neurons and has a weight associated with it. The weight of a connection affects how a neuron processes the information it receives along the connection.

9. Activation Functions: Introducing Non-Linearity

In fact, it is the introduction of the nonlinear mapping into the processing of a neuron that is the reason why activation functions are used.

Two-stage processing. A neuron maps inputs to an output in two stages: calculating a weighted sum of the inputs and then passing the result through an activation function.

Nonlinear mapping. Activation functions apply a nonlinear mapping to the output of the weighted sum. This nonlinearity is crucial for enabling the network to learn complex relationships.

Common activation functions. Examples of activation functions include threshold, logistic, tanh, and ReLU. The choice of activation function can significantly impact the performance of the network.

10. Backpropagation: Training Neural Networks

The learning done by machine learning is finding the parameters (or weights) of a model using a dataset.

Iterative weight updates. The standard training process for a neural network involves initializing the weights to random values and then iteratively updating them based on the network's performance on a dataset.

Gradient descent. The gradient descent algorithm is used to find the set of weights that minimizes the error of the network. It involves calculating the gradient of the error surface and updating the weights in the direction of the negative gradient.

Backpropagation algorithm. The backpropagation algorithm is used to calculate the error gradients for each weight in the network. It works in two phases: a forward pass and a backward pass.

11. CNNs and RNNs: Tailored Architectures

Tailoring the structure of a network to the specific characteristics of the data from a task domain can reduce the training time of the network, and improves the accuracy of the network.

Domain-specific architectures. Tailoring the structure of a network to the specific characteristics of the data from a task domain can reduce the training time of the network and improve its accuracy.

Convolutional Neural Networks (CNNs). CNNs are designed for image recognition tasks and use weight sharing and pooling to achieve translation invariance. They are particularly effective at extracting local visual features.

Recurrent Neural Networks (RNNs). RNNs are designed to process sequential data and have a memory buffer that stores the output of the hidden layer for one input and feeds it back into the hidden layer along with the next input from the sequence.

12. The Future: Interpretability and New Hardware

In any data-driven process the primary determinant of success is knowing what to measure and how to measure it.

Interpretability challenge. A key challenge in deep learning is the lack of interpretability. Understanding how a model makes its decisions is crucial for building trust and ensuring fairness.

New hardware. The demand for faster hardware continues to drive innovation in deep learning. Neuromorphic computing and quantum computing are two promising areas of research that could revolutionize the field.

Data-driven decisions. Deep learning is ideally suited for applications involving large datasets of high-dimensional data. Consequently, deep learning is likely to make a significant contribution to some of the major scientific challenges of our age.

Last updated: April 11, 2025