Chat

Background: Hessians and their eigenvectors

What Does a Matrix Do Geometrically?

A matrix transforms vectors in space. Think of it as a function that takes in a vector and outputs a transformed vector. For a 2D example:

• It can stretch vectors (making them longer or shorter)
• It can rotate vectors
• It can shear vectors (like pushing a rectangle into a parallelogram)
• Or any combination of these

Eigenvectors and Eigenvalues: The "Special Directions"

Eigenvectors are special directions where the matrix only stretches (or shrinks) without changing direction. If $v$ is an eigenvector of matrix $A$ with eigenvalue $\lambda$: $$Av = \lambda v$$

Geometrically:

• Eigenvector: A direction that doesn't rotate when the matrix is applied
• Eigenvalue: How much the matrix stretches (or shrinks) in that direction

For example, imagine stretching a rubber sheet. The eigenvectors are the directions along which the sheet stretches purely (no rotation), and eigenvalues tell you how much stretch happens in each direction.

• Details

Conditioning: The Ratio of Difficulties

The condition number is: $$\kappa = \frac{\lambda_{\max}}{\lambda_{\min}}$$

This ratio tells you how differently the function curves in different directions.

Well-Conditioned (κ ≈ 1)