Orthonormalizing Sets

A set $S$ of linearly independent vectors can be orthonormalized into a set $U$, such that the resulting space spanned is the same, using the following techniques based on the relationship between the cardinality of the set and the dimension each vector belongs to.

$$ S \subseteq V \rightarrow \begin{cases} |S| > \dim(V) \hspace{15pt} S \text{ cannot be linearly independent} \\ |S| \leq \dim(V) \hspace{13pt} \text{ If S is linearly indepdendent, use Gram-Schmidt } \end{cases} $$


Projection

The projection of a vector $\mathbf{v}$ onto another vector $\mathbf{u}$ represents the vector in the span of $\mathbf{u}$ that is nearest to $\mathbf{v}$.

image.png

$$ \text{proj}_{\mathbf{u}} \mathbf{v} = \frac{\langle \mathbf{v} , \space \mathbf{u} \rangle}{\langle \mathbf{u} , \space \mathbf{u} \rangle} \mathbf{u} $$


Gram-Schmidt

Algorithm which takes a set of vectors and produces an orthonormal set of vectors which spans the same space.

$$ S = \{\mathbf{v}_1 \dots \mathbf{v}_k\} \xrightarrow{\text{Gram-Schmidt}} U = \{\mathbf{u}_1 \dots \mathbf{u}_k \} \hspace{0pt} \\\text{ with } \langle \mathbf{u}_i , \space \mathbf{u}_j \rangle = \begin{cases} 0 \hspace{15pt} i \neq j \\ 1 \hspace{15pt} i =j\end{cases} $$

image.png

$$ \mathbf{u_i} = \frac{\mathbf{z}_i}{||\mathbf{z}_i||} \hspace{15pt} \mathbf{z_i} = \mathbf{v}i - \text{proj}{\mathbf{u}1 \space \dots \space \mathbf{u}{i-1}} \mathbf{v}_i = \mathbf{v}i - \sum {\ell=1} ^{i-1} \text{proj}{\mathbf{u}{\ell}}\mathbf{v}_i $$

To find each orthogonal vector, we take the difference between the original vector and the closest vector in the span of the orthonormal vectors found before in the algorithm.