====== Math ======

$$X_{(m,n)}=\begin{bmatrix}
| & | &  & | \\ 
x_1 & x_2 & \dots & x_m  \\ 
| & | &  & |
\end{bmatrix}
=U\Sigma V^T
=\begin{bmatrix}
| & | &  & | \\ 
u_1 & u_2 & \dots & u_n  \\ 
| & | &  & |
\end{bmatrix}
\begin{bmatrix}
\sigma_1 &  &    \\ 
 & \ddots &    \\ 
 &  & \sigma_m   \\ 
 &  &  
\end{bmatrix}
\begin{bmatrix}
| & | &  & | \\ 
v_1 & v_2 & \dots & v_m  \\ 
| & | &  & |
\end{bmatrix}^T
$$

where $n$ is the number of points in each $x$, and $m$ is the number of $x$ values. 

This can be further expanded into

$$ X = \sigma_1 u_1  v_1^T +  \sigma_2 u_2  v_2^T + \ldots + \sigma_m u_m  v_m^T$$ (assuming $n>>m$).  

We can therefore select the first $r$ terms of this expansion to get a reduced-order approximation of X.  I.e.
$$ X \approx \tilde{U}\tilde{\Sigma}\tilde{V}^T$$.

====== Interpretation ======

Correlation between the rows and columns of X.  


U and V are the eigenvectors of the correlation matrices,$X X^T$ and $X^T X$, respectively.  Because calculating $U$ and $V$ this way is slow, more optimized methods are available.    Which is the inner product of $x_i ^T x_j$ where $i$ and $j$ are the $i$th and $j$th column of $X$.  If the inner product is large, it means that $x_i$ and $x_j$ are similar. If small, it means that  $x_i$ and $x_j$  are different.  


====== Principal component analysis (PCA) ======
This is the statistical interpretation of SVD.

PCA finds elipsoids (depending on dimensionality) that account for the largest, second largest, etc. variance in your data.  

Note that X is defined differently as a matrix of horizontally aligned matrix (instead of vertical).

https://www.youtube.com/watch?v=fkf4IBRSeEc

Steps.
1.  Compute mean row, X'
2.  Subtract mean from X.  $B=X-X'$ 
3.  Calc covariance matrix, $C=B^T B$
4.  Computer eigs of C.  Eigenvectors, V, and eigvenvalues, D.
5.  Principal components, T=BV.  (V also called "loadings")
If $B=U \Sigma V^T$, then $T=U \Sigma$.  

Takeaway, PCA can be found by saving the SVD of (X minus its row-mean).  


====== Least squares, regression, and pseudo-inverse ======

SVD allows us to generalized Ax=b to non-square A matrices.  (pseudo-inverse)