Orthogonality and Least Square

6.1 Inner Product, Length, and Orthogonality

Inner Prodcut

If $\textbf{u}$ and $\textbf{v}$ are vectors in $\mathbb{R}^n$ , then we regard $\textbf{u}$ and $\textbf{v}$ as $n \times 1$ matrices.
The transpose $\textbf{u}^T$ is a $1 \times n$ matrix, and the matrix product $\textbf{u}^T\textbf{v}$ is an $1 \times 1$ matrix, which we write as a single real number (a scalar) without brackets.
The number $\textbf{u}^T\textbf{v}$ is called the inner product of $\textbf{u}$ and $\textbf{v}$ , and it is written as $\textbf{u} \cdot \textbf{v}$ .
This inner product is also referred to as a dot product
If $\textbf{u}=\begin{bmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{bmatrix}$ and $\textbf{v}=\begin{bmatrix} v_1 \\ v_2 \\ \vdots \\v_n \end{bmatrix}$ ,
then the inner product of $\textbf{u}$ and $\textbf{v}$ is

$\begin{bmatrix} u_1 & u_2 & \cdots & u_n\end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n\end{bmatrix} = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n.$

Theorem 1

Let $\textbf{u},\textbf{v}$ , and $\textbf{w}$ be vectors in $\mathbb{R}^n$ , and let $c$ be a scalar. Then

$\textbf{u} \cdot \textbf{v} = \textbf{v} \cdot \textbf{u}$
$(\textbf{u} + \textbf{v})\cdot \textbf{w} = \textbf{u}\cdot \textbf{w} + \textbf{v}\cdot \textbf{w}$
$(c \textbf{u})\cdot \textbf{v} = c (\textbf{u}\cdot \textbf{v} )= \textbf{u}\cdot (c \textbf{v} )$
$\textbf{u}\cdot\textbf{u} \ge 0$ , and $\textbf{u}\cdot\textbf{u} = 0$ if and only if $\textbf{u} = 0$

Properties (2) and (3) can be combined several times to produce the following useful rule:

$\left(c_1 \textbf{u}_1 + \cdots + c_{p} \textbf{u}_p \right) \cdot \textbf{w} = c_{1} \left( \textbf{u}_1 \cdot \textbf{w} \right) + \cdots + c_{p} \left( \textbf{u}_p \cdot \textbf{w} \right)$

The Length of a Vector

If $\textbf{v}$ is in $\mathbb{R}^n$ , with entries $v_1,\cdots, v_n$ , then the square root of $\textbf{v}\cdot \textbf{v}$ is defined because $\textbf{v}\cdot \textbf{v}$ is non-nengative.

Definition of Length

The length (or norm) of $\textbf{v}$ is the nonnegative scalar $\lVert \textbf{v} \rVert$ defined by

$\lVert \textbf{v} \rVert = \sqrt{\textbf{v}\cdot \textbf{v}} = \sqrt{(v_1^2 +v_2^2 + \cdots +v_n^2)}$ , and $\lVert \textbf{v} \rVert ^2 = \textbf{v} \cdot \textbf{v}$

Suppose $\textbf{v}$ is in $\mathbb{R}^2$ , say $\textbf{v}=\begin{bmatrix} a \\ b \end{bmatrix}$
If we identify $\lVert \textbf{v} \rVert$ with a geometric point in the plane, as usual, then $\lVert \textbf{v} \rVert$ coincides with the standard notion of the length of the line segment from the origin to $\textbf{v}$ .
This follows from the Pythagorean Theorem applied to a triangle such as the one shown in the following figure 1.

For any scalar $c$ , the length $c\textbf{v}$ is $|c|$ times the length of $\textbf{v}$ . That is

$\lVert c \textbf{v} \rVert = \lvert c \rvert \lVert \textbf{v} \rVert$

A vector whose length is 1 is called a unit vector.
If we divide a nonzero vector $\textbf{v}$ by its length -that is, multiply by $\frac{1}{\lVert \textbf{v} \rVert}$ - we obtain a unit vector $\textbf{u}$ because the length of $\textbf{u}$ is $\frac{1}{\lVert \textbf{v} \rVert}\lVert \textbf{v} \rVert$ .
The process of creating $\textbf{u}$ , from $\textbf{v}$ is sometimes called normalizing $\textbf{v}$ , and we say that $\textbf{u}$ , is in the same direction as $\textbf{v}$ .

Example 2:

Let $\textbf{v}=(1,-2,2,0)$ . Find a unit vector $\textbf{u}$ in the same direction as $\textbf{v}$ .

Solution:

First, compute the length of $\textbf{v}$ :

$\lVert \textbf{v} \rVert^2 = \textbf{v} \cdot \textbf{v} = (1)^2 + (-2)^2 +(2)^2 +(0)^2 = 9 \\ \lVert \textbf{v} \rVert = \sqrt{9} = 3$

Then, multiply $\textbf{v}$ by $\frac{1}{\lVert \textbf{v} \rVert}$ to obtain

$\begin{align*} \textbf{u} &= \frac{1}{\lVert \textbf{v} \rVert} \textbf{v} \\ &= \frac{1}{3}\textbf{v} \\ &= \frac{1}{3}\begin{bmatrix}1 \\ -2 \\ 2 \\ 0 \end{bmatrix} \\ &= \begin{bmatrix} 1/3 \\ -2/3 \\ 2/3 \\ 0 \end{bmatrix} \end{align*}$

Distance in $\mathbb{R}^n$

Definintion of Distance

For $\textbf{u}$ and $\textbf{v}$ in $\mathbb{R}^n$ , the distance between $\textbf{u}$ and $\textbf{v}$ , written as $\text{dist}(\textbf{u},\textbf{v})$ , is the length of the vector $\textbf{u} - \textbf{v}$ . That is

$\text{dist}(\textbf{u},\textbf{v}) = \lVert \textbf{u}-\textbf{v} \rVert$

Example 4:

Compute the distance between the vectors $\textbf{u}=(7,1)$ and $\textbf{v}=(3,2)$ .

Solution of Example 4

Calculate

$\begin{align*} \textbf{u}-\textbf{v} &= \begin{bmatrix}7 \\ 1\end{bmatrix}-\begin{bmatrix}3 \\ 2\end{bmatrix}=\begin{bmatrix}4 \\ -1\end{bmatrix}\\ \lVert \textbf{u}-\textbf{v} \rVert &= \sqrt{4^2+(-1)^2} = \sqrt{17} \end{align*}$

The vectors $\textbf{u}$ , $\textbf{v}$ , and $\textbf{u}-\textbf{v}$ are shown in the figure below.

When the vector $\textbf{u}-\textbf{v}$ is added to $\textbf{v}$ , the result is $\textbf{u}$ .

Notice that the parallelogram in the above figure shows that the distance from $\textbf{u}$ to $\textbf{v}$ is the same as the distance from $\textbf{u}−\textbf{v}$ to $\textbf{0}$ .

Orthogonal Vectors

Consider $\mathbb{R}^2$ or $\mathbb{R}^3$ and two lines through the origin dertermined by vectors $\textbf{u}$ and $\textbf{v}$ .
See the figure below.

The two lines shown in the figure above are geomatrically perpendicular if and only if the distance from $\textbf{u}$ to $\textbf{v}$ is the same as the distance from $\textbf{u}$ to $-\textbf{v}$ .
This is the same as requiring the squares of the distances to be the same.
Now

$\begin{align*} [\text{dist}(\textbf{u},\textbf{-v})]^2 &= \lVert \textbf{u}-(-\textbf{v}) \rVert ^2 = \lVert \textbf{u}+\textbf{v} \rVert ^2 \\ &=(\textbf{u}+\textbf{v})\cdot (\textbf{u}+\textbf{v}) \\ &=\textbf{u}\cdot (\textbf{u}+\textbf{v})+ \textbf{v}\cdot (\textbf{u}+\textbf{v}) & \text{Theorem 1 (2)} \\ &=\textbf{u}\cdot \textbf{u}+ \textbf{u}\cdot \textbf{v}+ \textbf{v}\cdot \textbf{u}+\textbf{v} \cdot \textbf{v} & \text{Theorem 1 (1),(2)} \\ &=\lVert \textbf{u} \rVert ^2 + \lVert \textbf{v} \rVert ^2 + 2\textbf{u}\cdot \textbf{v} & \text{Theorem 1 (1)} \end{align*}$

The same calculations with $\textbf{v}$ and $\textbf{−v}$ interchanged show that

$\begin{align*} [\text{dist}(\textbf{u},\textbf{v})]^2 &= \lVert \textbf{u} \rVert ^2 + \lVert \textbf{-v} \rVert ^2 +2\textbf{u}\cdot (\textbf{-v}) \\ &=\lVert \textbf{u} \rVert ^2 +\lVert \textbf{-v} \rVert ^2 -2\textbf{u}\cdot \textbf{v} \end{align*}$

The two squared distances are equal if and only if $2\textbf{u}\cdot\textbf{v}=-2\textbf{u}\cdot\textbf{v}$ , which happens if and only if $\textbf{u} \cdot \textbf{v} = 0$ .

This calculation shows that when vectors $\textbf{u}$ and $\textbf{v}$ are identified with geometric points, the corresponding lines through the points and the origin are perpendicular if and only if $\textbf{u} \cdot \textbf{v} = 0$ .

Definition of Orthogonality

Two vectors $\textbf{u}$ and $\textbf{v}$ in $\mathbb{R}^n$ are orthogonal (to each other) if $\textbf{u}\cdot\textbf{v}= 0$ .

The zero vector is orthgonal to every vector in $\mathbb{R}^n$ becuase $\textbf{0}^T \textbf{v} = 0$ for all $\textbf{v}$ .

Theorem 2

Two vectors $\textbf{u}$ and $\textbf{v}$ are orthogonal if and only if $\lVert \textbf{u}+\textbf{v} \rVert^2 = \lVert \textbf{u} \rVert ^2 + \lVert \textbf{v} \rVert ^2$ .

Orthogonal Complements

If a vector $\textbf{z}$ is orthogonal to every vector in a subspace $W$ of $\mathbb{R}^n$ , then $\textbf{z}$ is said to be orthogonal to $W$ .

The set of all vectors $\textbf{z}$ that are orthogonal to $W$ is called the orthogonal complement of $W$ and is denoted by $W^{\perp}$ (and read as “W perpendicular” or simply “W perp”).

A vector $\textbf{x}$ is in $W^{\perp}$ if and only if $\textbf{x}$ is orthogonal to every vector in a set that spans $W$ .
$W^{\perp}$ is a subspace of $\mathbb{R}^n$

Theorem 3

Let $A$ be an $m \times n$ matrix. The orthogonal complement of the row space of $A$ is the null space of $A$ , and the orthogonal complement of the column space of $A$ is the null space of $A^{T}$ :

$(\text{Row }A)^{\perp} = \text{Nul }A \\ (\text{Col }A)^{\perp} = \text{Nul }A^T$

Proof:

The row-column rule for computing $A\textbf{x}$ shows that if $\textbf{x}$ is in $\text{Nul }A$ , then $\textbf{x}$ is orthogonal to each row of $A$ (with the rows treated as vectors in $\mathbb{R}^n$ ).
Since the rows of $A$ span the row space, $\textbf{x}$ is orthogonal to $\text{Row }A$ .
Conversely, if $\textbf{x}$ is orthogonal to $\text{Row }A$ , then $\textbf{x}$ is certainly orthogonal to each row of $A$ , and hence $A\textbf{x}=\textbf{0}$
This proves the first statement of the theorem.
Since this statement is true for any matrix, it is true for $A^T$ .
That is, the orthogonal complement of the row space of $A^T$ is the null space of $A^T$
This proves the second statement, because Row $A^T$ = Col $A$ . $\blacksquare$

Angles In R Squared and R Cubed (Optional)

If $\textbf{u}$ and $\textbf{v}$ are nonzero vectors in either $\mathbb{R}^2$ or $\mathbb{R}^3$ , then there is a nice connection between their inner product and the angle $\theta$ between the two line segments from the origin to the points identified with $\textbf{u}$ and $\textbf{v}$ .
The formula is

$\textbf{u} \cdot \textbf{v} = \lVert \textbf{u} \rVert \lVert \textbf{v} \rVert \cos{\theta} \tag{2}$

To verify this formula for vectors in $\mathbb{R}^2$ , consider the triangle shown in the figure below with sides of lengths, $\lVert \textbf{u} \rVert$ , $\lVert \textbf{v} \rVert$ , and $\lVert \textbf{u}-\textbf{v} \rVert$ .

By the law of cosines,

$\lVert \textbf{u}-\textbf{v} \rVert ^2 = \lVert \textbf{u} \rVert ^2 + \lVert \textbf{v} \rVert ^2 - 2 \lVert \textbf{u} \rVert \lVert \textbf{v} \rVert \cos{\theta}$

which can be rearranged to produce the equations below

$\begin{align*} \Vert \textbf{u} \Vert \Vert \textbf{u} \Vert \cos{\theta} &= \frac{1}{2} \left[ \Vert \textbf{u} \Vert^2 \Vert \textbf{v} \Vert^2 - \Vert \textbf{u}-\textbf{v} \Vert^2 \right] \\ &= \frac{1}{2} \left[ u_1^2 + u_2^2 +v_1^2 +v_2^2 - (u_1-v_1)^2 - (u_2-v_2)^2 \right] \\ &= u_1 v_1 +u_2v_2 \\ &= \textbf{u}\cdot\textbf{v} \end{align*}$