Refs

• CS 229 linear algebra review

Misc

• https://stats.stackexchange.com/questions/391837/intuition-behind-gradient-of-expected-value-and-logarithm-of-probabilities

Matmuls

• Visualizing

  

Basics

• non-singular ↔ single solution ↔ non-0 determinant ↔ linear independence ↔ invertible ↔ full rank
  • non-singular transform ↔ covers whole plane (doesn’t reduce to a line or point)
• singular ↔ 0 or infinite solutions ↔ 0 determinant ↔ linear dependence ↔ non-invertible ↔ non-full rank
  • singular transform ↔ reduce to lower rank
• gaussian elimination, row echelon, reduced row echelon
• det A * det B = det AB
• det A^-1 = (det A)^-1
• basis of a space = minimal spanning set for that space
• TODO (sum of k rank-1 matrices) has rank ≤ min(d,p,k) where d,p are the sizes of the constituent vectors

quadratic form

SVD

• rotation, scale, rotation

• Get low-rank approximations by filtering out low-scaling-coefficient ranks

  

• Not just squares

  

• Serrano

Cholesky decomposition

Convolutions

• 1x1 conv = linear transformation

  # example in 1d
  >>> torch.tensor([[[1],[2],[3]],[[1],[1],[1]]]).float().squeeze(2) @ torch.tensor([[0,1,2,3],[4,5,6,7],[8,9,10,11]]).float()
  tensor([[32., 38., 44., 50.],
          [12., 15., 18., 21.]])
  
  >>> F.conv1d(torch.tensor([[0,1,2,3],[4,5,6,7],[8,9,10,11]]).float(),torch.tensor([[[1],[2],[3]],[[1],[1],[1]]]).float())
  tensor([[32., 38., 44., 50.],
          [12., 15., 18., 21.]])