Why Machines Learn

1. Early AI Dreams Faced Fundamental Limits

The perceptron never lived up to the hype.

Initial excitement. Early artificial intelligence research, like Frank Rosenblatt's perceptron in the late 1950s, sparked immense hype, promising machines that could learn, see, and even be conscious. Inspired by simplified models of biological neurons (McCulloch-Pitts neurons), these early devices aimed to mimic brain function.

Simple learning. The perceptron introduced the idea of learning from data by adjusting internal weights and a bias term to find a linear boundary (hyperplane) separating data points into categories. A key theoretical result proved the perceptron could always find this boundary if the data were linearly separable.

Inherent limitations. Despite initial promise, single-layer perceptrons were mathematically proven to be incapable of solving simple non-linear problems like the XOR gate. This limitation, highlighted by Minsky and Papert in 1969, contributed significantly to the first "AI winter," halting research progress for years.

2. Mathematics Provides the Language for Machine Learning

It’s central to the plot.

Vectors as data. Machine learning fundamentally relies on representing data as mathematical objects, primarily vectors and matrices. A vector, possessing magnitude and direction, can represent anything from a person's height and weight to the pixel values of an image, allowing data points to exist in multi-dimensional spaces.

Operations reveal relationships. Linear algebra provides the tools to manipulate these data representations.

• Vector addition/subtraction: Combining or comparing data points.
• Scalar multiplication: Scaling data features.
• Dot product: Measuring similarity or projection, crucial for understanding distances and hyperplanes.

Matrices transform data. Matrices, rectangular arrays of numbers, are used to transform vectors. Multiplying a vector by a matrix can change its magnitude, direction, or even its dimensionality, forming the basis for how neural networks process information across layers.

3. Learning Algorithms Minimize Error Through Descent

When I wrote the LMS algorithm on the blackboard for the first time, somehow I just knew intuitively that this is a profound thing.

Quantifying error. Machine learning algorithms learn by minimizing the difference between their output and the desired output, often measured by a "loss function" like the mean squared error (MSE). The goal is to find the model parameters (weights, biases) that result in the lowest possible loss.

Gradient descent. Calculus provides the method to find this minimum. Gradient descent involves calculating the "gradient" (the direction of steepest increase) of the loss function with respect to the model parameters and taking small steps in the opposite direction (steepest decrease
[ERROR: Incomplete response]

Last updated: May 1, 2025