Orthogonal Complement

An orthogonal complement $V^{\perp} \sub W$ of a subspace $V \sub W$ is the subspace whose elements are all orthogonal to elements of $V$. It has the qualities that any element in $W$can be constructed from elements of $V$ and $V^{\perp}$ and that the only element $V$ and $V^{\perp}$ share is $0$.

$$ \forall u \in V^{\perp}, \forall v \in V : \begin{cases} u \perp v \\V ⊕ V^{\perp} = W \\ V \hspace{2pt} \cap \hspace{2pt} V^{\perp} = \{0\} \end{cases} $$


Fundamental Subspaces

Any matrix $A \in \R^{m \times n}$ has 4 fundamental subspaces associated with the vectors it transforms.

$$ A = \begin{bmatrix} a_{11} & a_{12} & \dots & a_{1n} \\ a_{21} & a_{22} & \dots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \dots & a_{mn} \end{bmatrix} = \begin{bmatrix} \mathbf{a}_1 \hspace{2pt} \mathbf{a}_2 \hspace{2pt} \dots \hspace{2pt} \mathbf{a}_n\end{bmatrix} = \begin{bmatrix} \boldsymbol{\alpha}_1 ^{\text{T}}\\ \boldsymbol{\alpha}_2^{\text{T}} \\ \vdots \\ \boldsymbol{\alpha}_m^{\text{T}}\end{bmatrix} $$

$$ A:\R^{n} \rightarrow \R^{m} \hspace{15pt} [A]{m \times n} \hspace{3pt}[\mathbf{x}]{n \times 1} = [\mathbf{y}]_{m \times 1} \\

A^\text{T} : \R^m \rightarrow \R^n \hspace{15pt}[A^\text{T}]{n \times m} \hspace{3pt}[\boldsymbol{\chi}]{m \times 1} = [\boldsymbol{\gamma}]_{n \times 1} $$

Fundamental Relationship

The fundamental subspaces of a matrix $A \in \R^{m \times n}$ are related in the following way

$$ \mathcal{N}(A) = \text{C}(A^\text{T})^\perp \hspace{5pt} \text{ and } \hspace{5pt}

\mathcal{N}(A^T) = \text{C}(A)^{\perp} $$

The intuition behind this considers the fact that any vector $\mathbf{x} \in \mathcal{N}(A)$ is orthogonal to the rows $\{ \boldsymbol{\alpha}_1 \dots \boldsymbol{\alpha}_m \}$ of $A$ and thus is orthogonal to any linear combination of these rows.

The complement quality comes from supposing $\dim \text{C}(A^T) < n$. To form a basis for the input space, we combine the bases of $\text{C}(A^T)$ and $\mathcal{N}(A)$ to span the entire domain (as we know the basis vectors of each basis would be orthogonal to one another, and thus linearly independent).

image.png