Ch 07 Symmetric Matrices and Quadratic Forms

7.4 The Singular Value Decomposition (2)

Example 3

Use the results of Examples 1 and 2 to construct a singular value decomposition of $A= \begin{bmatrix} 4 & 11 & 14 \\ 8 & 7 & -2 \\ \end{bmatrix}$

Solution of Example 3

A construction can be divided into three steps.

Step 1. Find an orthogonal diagonalization of $A^TA$.

That is, find the eigenvalues of $A^TA$ and a corresponding orthonormal set of eigenvectors. If $A$ had only two columns, the calculations could be done by hand. Larger matrices usually require a matrix program. However, for the matrix $A$ here, the eigendata for $A^TA$ are provided in Example 2.

Step 2. Set up $V$ and $\Sigma$.

Arrange the eigenvalues of $A^TA$ in decreasing order. In Example 1, the eigenvalues are already listed in decreasing order: 360, 90, and 0. The corresponding unit eigenvectors, $\textbf{v}_1, \textbf{v}_2, \text{ and}, \textbf{v}_3$, are the right singular vectors of $A$. Using Example 1, construct

$$\begin{align*} V &= \begin{bmatrix} \textbf{v}_1 & \textbf{v}_2 & \textbf{v}_3 \end{bmatrix} \\ &=\begin{bmatrix} 1/3 & -2/3 & 2/3 \\ 2/3 & -1/3 & -2/3 \\ 2/3 & 2/3 & 1/3 \end{bmatrix} \end{align*}$$

The square roots of the eigenvalues are the singular values:

$$\sigma_1 = 6 \sqrt{10}, \\ \sigma_2 = 3 \sqrt{10}, \\ \sigma_3 = 0$$

The nonzero singular values are the diagonal entries of $D$.

The matrix $\Sigma$ is the same size as $A$, with $D$ in its upper left corner and with 0’s elsewhere.

$$\begin{align*} D &= \begin{bmatrix} 6\sqrt{10} & 0 \\ 0 & 3\sqrt{10} \\ \end{bmatrix}, \\ \Sigma &= \begin{bmatrix} D & \textbf{0} \end{bmatrix} \\ &= \begin{bmatrix} 6\sqrt{10} & 0 & 0 \\ 0 & 3\sqrt{10} & 0 \end{bmatrix} \\ \end{align*}$$

Step 3. Construct $U$.

When $A$ has rank $r$, the first $r$ columns of $U$ are the normalized vectors obtained from $A\textbf{v}_1 \cdots A\textbf{v}_r$. In this example, $A$ has two nonzero singular values, so $\text{rank }A=2$. Recall from equation (2) and the paragraph before Example 2 that $||A\textbf{v}_1||=\sigma_1$ and $||A\textbf{v}_2||=\sigma_2$. Thus,

$$\textbf{u}_1=\frac{1}{\sigma_1}A\textbf{v}_1=\frac{1}{6\sqrt{10}} \begin{bmatrix} 18\\ 6 \end{bmatrix} = \begin{bmatrix} \frac{3}{\sqrt{10}}\\ \frac{1}{\sqrt{10}} \end{bmatrix} \\ \textbf{u}_2=\frac{1}{\sigma_2}A\textbf{v}_2=\frac{1}{3\sqrt{10}} \begin{bmatrix} 3\\ -9 \end{bmatrix} = \begin{bmatrix} \frac{1}{\sqrt{10}}\\ \frac{-3}{\sqrt{10}} \end{bmatrix}$$

Note that $\left\{\textbf{u}_1,\textbf{u}_2\right\}$ is already a basis for $\mathbb{R}^2$. Thus no additional vectors are needed for $U$, and $U = \begin{bmatrix}\textbf{u}_1 & \textbf{u}_2\end{bmatrix}$.

The singular value decomposition of $A$ is

$$A = \begin{bmatrix} \frac{3}{\sqrt{10}} & \frac{1}{\sqrt{10}} \\ \frac{1}{\sqrt{10}} & \frac{-3}{\sqrt{10}} \\ \end{bmatrix} \begin{bmatrix} \frac{6}{\sqrt{10}} & 0 & 0 \\ 0 & \frac{3}{\sqrt{10}} & 0\\ \end{bmatrix} \begin{bmatrix} \frac{1}{3} & \frac{2}{3} & \frac{2}{3} \\ \frac{-2}{3} & \frac{-1}{3} & \frac{2}{3}\\ \frac{2}{3} & \frac{-2}{3} & \frac{1}{3} \end{bmatrix}$$

Theorem: The Invertible Matrix Theorem (concluded)

Let $A$ be an $m \times n$ matrix. Then the following statements are each equivalent to the statement that $A$ is an invertible matrix.

u. $\left(\text{Col }A\right)^{\perp} = \left\{\textbf{0}\right\}$

v. $\left(\text{Nul }A\right)^{\perp} = \mathbb{R}^n$

w. $\text{Row }A = \mathbb{R}^n$

x. $A$ has $n$ nonzero singular values.

Example 7

Reduced SVD and the Pseudoinverse of $A$)

When $\Sigma$ contains rows or columns of zeros, a more compact decomposition of $A$ is possible. Using the notation established above, let $r = \text{rank }A$, and partition $U$ and $V$ into submatrices whose first blocks contain $r$ columns:

$$U = \begin{bmatrix} U_r & U_{m-r} \end{bmatrix}\quad ,\text{where } U_r=\begin{bmatrix} \textbf{u}_1 & \cdots & \textbf{u}_r \end{bmatrix} \\ V = \begin{bmatrix} V_r & V_{n-r} \end{bmatrix}\quad ,\text{where } V_r=\begin{bmatrix} \textbf{v}_1 & \cdots & \textbf{v}_r \end{bmatrix} \\ $$

Then $U_r$ is $m \times r$ and $V_r$ is $n\times r$. Then partitioned matrix multiplication shows that

$$A = \begin{bmatrix} U_r & U_{m-r} \end{bmatrix} \begin{bmatrix} D & \mathbb{0} \\ \mathbb{0} & \mathbb{0} \end{bmatrix} \begin{bmatrix} V_r^T \\ V_{m-r}^T \end{bmatrix} = U_r D V_r^T$$

This factorization of $A$ is called a reduced singular value decomposition of $A$. Since the diagonal entries in $D$ are nonzero, $D$ is invertible.

The following matrix is called the pseudoinverse (also, the Moore-Penrose inverse) of $A$:

$$A^+ = V_r D^{-1} U_r^T \tag{10}$$