Introduction to Linear Algebra

1. Vectors and Matrices: The Foundation of Linear Algebra

Vectors are the key actors, matrices tell the story.

Building Blocks. Vectors and matrices are the fundamental building blocks of linear algebra. Vectors represent points in space or ordered lists of numbers, while matrices are rectangular arrays of numbers that can represent linear transformations or systems of equations.

Linear Combinations. A linear combination of vectors is a sum of scalar multiples of those vectors. Understanding linear combinations is crucial for grasping concepts like span, linear independence, and basis. For example, the vector (5, 7) can be expressed as a linear combination of (1, 0) and (0, 1) as 5(1, 0) + 7(0, 1).

Matrix Operations. Matrices can be added, subtracted, and multiplied under certain conditions. Matrix multiplication, in particular, is a powerful operation that combines linear transformations. The product of two matrices A and B, denoted AB, is only defined if the number of columns in A equals the number of rows in B.

2. Solving Linear Equations: Mastering the Art of Elimination

The goal is to understand the steps that solve Ax = b.

System of Equations. Solving linear equations is a central problem in linear algebra. A system of linear equations can be represented in matrix form as Ax = b, where A is the coefficient matrix, x is the vector of unknowns, and b is the constant vector.

Gaussian Elimination. Gaussian elimination is a systematic method for solving linear equations by transforming the augmented matrix [A | b] into row-echelon form. This involves performing elementary row operations, such as swapping rows, multiplying a row by a scalar, and adding a multiple of one row to another.

Back Substitution. Once the matrix is in row-echelon form, the solution can be found by back substitution. Starting from the last equation, solve for the last unknown, and then substitute that value into the previous equation to solve for the next unknown, and so on.

3. The Four Fundamental Subspaces: Unveiling the Structure of Matrices

The column space contains all vectors b that allow the equation Ax = b to be solved.

Subspace Definition. The four fundamental subspaces associated with a matrix A are the column space, the nullspace, the row space, and the left nullspace. These subspaces provide a complete picture of the matrix's structure and its action on vectors.

Column Space and Nullspace. The column space of A, denoted C(A), is the span of the columns of A. It represents all possible linear combinations of the columns of A. The nullspace of A, denoted N(A), is the set of all vectors x such that Ax = 0. It represents the set of solutions to the homogeneous equation Ax = 0.

Row Space and Left Nullspace. The row space of A, denoted C(AT), is the span of the rows of A. It is the column space of the transpose of A. The left nullspace of A, denoted N(AT), is the set of all vectors y such that ATy = 0. It is the nullspace of the transpose of A.

4. Orthogonality: Perpendicularity and Projections in Vector Spaces

Orthogonal vectors are at right angles (90°).

Orthogonal Vectors. Two vectors are orthogonal if their dot product is zero. Orthogonality is a fundamental concept in linear algebra, with applications in areas such as signal processing, data compression, and machine learning.

Projections. The projection of a vector b onto a vector a is the component of b that lies in the direction of a. The projection is given by the formula p = (a.b / a.a) * a. Projections are used to find the closest approximation of a vector in a given subspace.

Gram-Schmidt Process. The Gram-Schmidt process is a method for constructing an orthonormal basis from a set of linearly independent vectors. It involves projecting each vector onto the subspace spanned by the previous vectors and subtracting the projection to obtain an orthogonal vector.

5. Determinants: Measuring Areas, Volumes, and Matrix Invertibility

The determinant tells instantly whether a matrix is invertible.

Determinant Calculation. The determinant of a square matrix is a scalar value that provides information about the matrix's properties. For a 2x2 matrix, the determinant is calculated as ad - bc. For larger matrices, the determinant can be calculated using cofactor expansion or other methods.

Geometric Interpretation. The absolute value of the determinant of a matrix represents the area (in 2D) or volume (in 3D) scaling factor of the linear transformation represented by the matrix. A determinant of zero indicates that the matrix is singular (non-invertible) and that the transformation collapses space.

Properties of Determinants. Determinants have several important properties, including:

• det(A) = det(AT)
• det(AB) = det(A) * det(B)
• If A has a row or column of zeros, then det(A) = 0
• If A has two identical rows or columns, then det(A) = 0

6. Eigenvalues and Eigenvectors: Unlocking the Secrets of Linear Transformations

Eigenvectors keep the same direction when multiplied by A.

Eigenvalue Definition. An eigenvector of a matrix A is a non-zero vector v such that Av = λv, where λ is a scalar called the eigenvalue. Eigenvectors remain in the same direction when transformed by A, only scaled by the eigenvalue.

Finding Eigenvalues. To find the eigenvalues of a matrix A, solve the characteristic equation det(A - λI) = 0, where I is the identity matrix. The solutions to this equation are the eigenvalues of A.

Diagonalization. A matrix A is diagonalizable if it has n linearly independent eigenvectors, where n is the size of the matrix. In this case, A can be written as A = PDP-1, where D is a diagonal matrix with the eigenvalues of A on the diagonal and P is a matrix whose columns are the eigenvectors of A.

7. Singular Value Decomposition (SVD): A Powerful Tool for Data Analysis

The SVD separates any matrix A into well-chosen vectors and singular values.

SVD Definition. The singular value decomposition (SVD) is a factorization of a matrix A into three matrices: A = UΣVT, where U and V are orthogonal matrices and Σ is a diagonal matrix with non-negative entries called singular values.

Singular Values. The singular values of A are the square roots of the eigenvalues of AT A. They represent the "strengths" of the linear transformation represented by A. The singular values are typically ordered from largest to smallest.

Applications of SVD. SVD has numerous applications in areas such as:

• Image compression
• Recommender systems
• Principal component analysis (PCA)
• Data analysis

8. Linear Transformations: Mapping Vector Spaces

A linear transformation obeys the rules T(v + w) = T(v) + T(w) and T(cv) = cT(v).

Transformation Definition. A linear transformation is a function that maps vectors from one vector space to another, preserving vector addition and scalar multiplication. Linear transformations can be represented by matrices.

Matrix Representation. Any linear transformation T: V -> W can be represented by a matrix A, such that T(v) = Av for all vectors v in V. The matrix A depends on the choice of bases for V and W.

Examples of Linear Transformations:

• Rotation
• Scaling
• Shear
• Projection

9. Linear Algebra in Optimization: Finding the Best Solutions

Optimization searches for the minimum (or maximum) of a function.

Optimization Problems. Linear algebra plays a crucial role in optimization problems, which involve finding the best solution to a problem subject to certain constraints. Many optimization problems can be formulated as linear or quadratic programs.

Gradient Descent. Gradient descent is an iterative optimization algorithm that finds the minimum of a function by repeatedly moving in the direction of the negative gradient. The gradient is a vector that points in the direction of the steepest ascent of the function.

Lagrange Multipliers. Lagrange multipliers are used to find the maximum or minimum of a function subject to equality constraints. The method involves introducing a new variable (the Lagrange multiplier) for each constraint and solving a system of equations.

10. Learning from Data: Extracting Insights from Information

Linear algebra is at the heart of machine learning.

Data Representation. Linear algebra provides the tools for representing and manipulating data in machine learning. Data points are often represented as vectors, and datasets are represented as matrices.

Principal Component Analysis (PCA). PCA is a dimensionality reduction technique that uses SVD to find the principal components of a dataset. The principal components are the directions of maximum variance in the data.

Applications in Machine Learning:

• Linear regression
• Logistic regression
• Support vector machines (SVMs)
• Neural networks

Last updated: February 21, 2025