The Hundred-Page Machine Learning Book

1. Machine Learning: Algorithms from Examples

Machine learning can also be defined as the process of solving a practical problem by 1) gathering a dataset, and 2) algorithmically building a statistical model based on that dataset.

Solving Practical Problems. Machine learning (ML) is about creating algorithms that learn from data to solve real-world problems. Instead of explicitly programming a machine to perform a task, ML algorithms are trained on datasets, allowing them to identify patterns and make predictions or decisions. This approach is particularly useful when dealing with complex or dynamic systems where explicit programming is difficult or impossible.

Data-Driven Approach. The core of machine learning lies in the data. ML algorithms require a dataset of examples to learn from. These examples can come from various sources, including nature, human-generated data, or even other algorithms. The quality and quantity of the data significantly impact the performance of the ML model.

Statistical Models. At its heart, machine learning involves building statistical models based on the gathered data. These models capture the underlying relationships and patterns within the data, enabling the algorithm to make predictions or decisions on new, unseen data. The goal is to create a model that generalizes well, meaning it can accurately perform its task on data it hasn't been explicitly trained on.

2. Supervised Learning: Labeled Data for Prediction

In supervised learning, the dataset is the collection of labeled examples {(x i, yi )}.

Learning from Labeled Examples. Supervised learning is a type of machine learning where the algorithm learns from a dataset containing labeled examples. Each example consists of a feature vector (x) and a corresponding label (y). The label represents the desired output or target for that particular input.

Classification and Regression. Supervised learning can be further divided into two main categories: classification and regression. In classification, the goal is to predict a categorical label, such as "spam" or "not spam." In regression, the goal is to predict a continuous value, such as the price of a house.

Model Training and Prediction. The supervised learning algorithm uses the labeled dataset to train a model that can map input feature vectors to their corresponding labels. Once the model is trained, it can be used to predict the labels for new, unseen feature vectors. The accuracy of the model is typically evaluated using a separate test dataset.

3. Unsupervised Learning: Discovering Hidden Structures

In unsupervised learning, the dataset is a collection of unlabeled examples {x i}.

Exploring Unlabeled Data. Unsupervised learning is a type of machine learning where the algorithm learns from a dataset containing only unlabeled examples. The goal is to discover hidden structures, patterns, or relationships within the data without any prior knowledge of the desired output.

Clustering and Dimensionality Reduction. Two common tasks in unsupervised learning are clustering and dimensionality reduction. Clustering involves grouping similar examples together into clusters, while dimensionality reduction involves reducing the number of features in the dataset while preserving its essential information.

Applications in Various Fields. Unsupervised learning has applications in various fields, including customer segmentation, anomaly detection, and data visualization. For example, it can be used to identify different customer segments based on their purchasing behavior or to detect fraudulent transactions based on their unusual patterns.

4. Linear Regression: Modeling Relationships with Lines

On the other hand, the hyperplane in linear regression is chosen to be as close to all training examples as possible.

Finding the Best Fit. Linear regression is a supervised learning algorithm that models the relationship between a dependent variable (target) and one or more independent variables (features) by fitting a linear equation to the observed data. The goal is to find the line (or hyperplane in higher dimensions) that best represents the relationship between the variables.

Minimizing the Error. The "best fit" is determined by minimizing the sum of the squared differences between the predicted values and the actual values. This is known as the least squares method. The resulting linear equation can then be used to predict the value of the target variable for new, unseen feature vectors.

Simple and Interpretable. Linear regression is a relatively simple and interpretable algorithm, making it a good starting point for many regression problems. However, it may not be suitable for datasets with complex, non-linear relationships between the variables. In such cases, more advanced algorithms may be required.

5. Logistic Regression: Classification with Probabilities

They figured out that if we define a negative label as 0 and the positive label as 1, we would just need to find a simple continuous function whose codomain is (0 , 1).

Predicting Probabilities. Logistic regression is a supervised learning algorithm used for binary classification problems. Unlike linear regression, which predicts a continuous value, logistic regression predicts the probability of an example belonging to a particular class.

Sigmoid Function. Logistic regression uses the sigmoid function to map the linear combination of features to a probability value between 0 and 1. The sigmoid function is an S-shaped curve that squashes any real-valued input into this range.

Maximum Likelihood Estimation. The parameters of the logistic regression model are typically estimated using maximum likelihood estimation. This involves finding the values of the parameters that maximize the likelihood of observing the given labeled dataset. The model can then be used to classify new examples based on their predicted probabilities.

6. Decision Trees: Making Decisions Step-by-Step

As the leaf node is reached, the decision is made about the class to which the example belongs.

Hierarchical Decision-Making. A decision tree is a supervised learning algorithm that uses a tree-like structure to make decisions. Each internal node in the tree represents a test on a particular feature, and each branch represents the outcome of that test. The leaf nodes represent the final classification or prediction.

Entropy and Information Gain. Decision trees are built by recursively splitting the dataset based on the feature that provides the most information gain. Information gain is a measure of how much the entropy (uncertainty) of the dataset is reduced by splitting on a particular feature.

Easy to Interpret. Decision trees are relatively easy to interpret, making them a popular choice for problems where explainability is important. However, they can be prone to overfitting, especially if the tree is allowed to grow too deep. Techniques like pruning and regularization can be used to prevent overfitting.

7. SVM: Finding the Optimal Separating Boundary

In machine learning, the boundary separating the examples of different classes is called the decision boundary.

Maximizing the Margin. Support Vector Machines (SVMs) are supervised learning algorithms used for both classification and regression. The goal of an SVM is to find the optimal hyperplane that separates the examples of different classes with the largest possible margin.

Support Vectors. The support vectors are the examples that lie closest to the hyperplane and influence its position. The SVM algorithm focuses on these support vectors to determine the optimal separating boundary.

Kernel Trick. SVMs can also be used to solve non-linear classification problems by using the kernel trick. The kernel trick involves mapping the original feature space into a higher-dimensional space where the examples become linearly separable. Common kernel functions include the polynomial kernel and the radial basis function (RBF) kernel.

8. Neural Networks: Mimicking the Brain's Complexity

As you can see in fig. 1, in multilayer perceptron all outputs of one layer are connected to each input of the succeeding layer.

Interconnected Nodes. Neural networks are machine learning models inspired by the structure and function of the human brain. They consist of interconnected nodes (neurons) organized into layers. Each connection between nodes has a weight associated with it, which represents the strength of the connection.

Activation Functions. Each node in a neural network applies an activation function to the weighted sum of its inputs. Activation functions introduce non-linearity into the model, allowing it to learn complex relationships between the variables. Common activation functions include the sigmoid function, the ReLU function, and the tanh function.

Deep Learning. Deep learning refers to neural networks with multiple layers between the input and output layers. These deep neural networks can learn hierarchical representations of the data, enabling them to solve complex problems in areas such as image recognition, natural language processing, and speech recognition.

9. Feature Engineering: Crafting Meaningful Inputs

The problem of transforming raw data into a dataset is called feature engineering.

Transforming Raw Data. Feature engineering is the process of transforming raw data into a set of features that can be used by a machine learning algorithm. This is a crucial step in the machine learning pipeline, as the quality of the features significantly impacts the performance of the model.

Domain Knowledge. Feature engineering often requires domain knowledge to identify the most relevant and informative features. It involves selecting, transforming, and creating new features from the raw data.

Techniques for Feature Engineering. Common techniques for feature engineering include one-hot encoding, binning, normalization, and standardization. One-hot encoding is used to convert categorical features into numerical features, while binning is used to convert continuous features into categorical features. Normalization and standardization are used to scale the features to a common range.

10. Model Assessment: Evaluating Performance Metrics

The test set contains the examples that the learning algorithm has never seen before, so if our model performs well on predicting the labels of the examples from the test set, we say that our model generalizes well or, simply, that it’s good.

Measuring Generalization. Model assessment is the process of evaluating the performance of a machine learning model on a separate test dataset. The test dataset contains examples that the model has never seen before, providing an unbiased estimate of its ability to generalize to new data.

Metrics for Regression and Classification. Different metrics are used to assess the performance of regression and classification models. For regression, common metrics include mean squared error (MSE) and R-squared. For classification, common metrics include accuracy, precision, recall, and F1-score.

Confusion Matrix. A confusion matrix is a table that summarizes the performance of a classification model by showing the number of true positives, true negatives, false positives, and false negatives. It can be used to calculate various performance metrics, such as precision and recall.

11. Regularization: Preventing Overfitting

Regularization is an umbrella-term that encompasses methods that force the learning algorithm to build a less complex model.

Balancing Bias and Variance. Regularization is a technique used to prevent overfitting in machine learning models. Overfitting occurs when the model learns the training data too well, resulting in poor performance on new data. Regularization techniques add a penalty term to the cost function, encouraging the model to build a simpler, more generalizable model.

L1 and L2 Regularization. Two common types of regularization are L1 regularization and L2 regularization. L1 regularization adds a penalty proportional to the absolute value of the model's parameters, while L2 regularization adds a penalty proportional to the square of the model's parameters.

Dropout and Batch Normalization. In neural networks, dropout and batch normalization are also used as regularization techniques. Dropout randomly excludes some units from the computation during training, while batch normalization standardizes the outputs of each layer.

12. Ensemble Methods: Combining Multiple Models

Ensemble learning is a learning paradigm that, instead of trying to learn one super-accurate model, focuses on training a large number of low-accuracy models and then combining the predictions given by those weak models to obtain a high-accuracy meta-model.

Wisdom of the Crowd. Ensemble methods combine the predictions of multiple individual models to improve overall performance. The idea is that by combining the strengths of different models, the ensemble can achieve higher accuracy and robustness than any single model.

Bagging and Boosting. Two common ensemble methods are bagging and boosting. Bagging involves training multiple models on different subsets of the training data, while boosting involves training models sequentially, with each model focusing on correcting the errors of the previous models.

Random Forest and Gradient Boosting. Random forest and gradient boosting are two popular ensemble algorithms that use decision trees as their base models. Random forest uses bagging to create multiple decision trees, while gradient boosting uses boosting to create a sequence of decision trees.

Last updated: February 22, 2025