MAT 102 Linear Algebra · DSE · Intake 2/1 · Dr. Ronobir Chandra Sarker

Orthogonality — Part 1: Orthogonal vectors and subspaces

Lecture notes on section 4.1 of Strang, Introduction to Linear Algebra. Read with pen and paper beside you, work each example yourself before reading its solution, and do every exercise before opening the answer.

The chapter in three parts: Part 1 · Orthogonal vectors and subspaces  |  Part 2 · Projections  |  Part 3 · Least squares and orthonormal bases

⬇ Download Part 1 as PDF 🎯 Practice page
Contents
  1. Before you start
  2. Orthogonal vectors
  3. Orthogonal subspaces
  4. Question types and model answers
  5. Practice test
  6. Formula sheet

0Before you start

Why we study this

Perpendicular directions are the most convenient directions in mathematics: moving along one of them changes nothing along the other. This part gives the algebraic test for “perpendicular” in any dimension, and then applies it to the four subspaces of a matrix, which turn out to come in two perpendicular pairs. Everything in Parts 2 and 3 (projections and least squares) rests on these two ideas.

What you need from Chapter 3

Notation

SymbolMeaning
\(\vb v\T\vb w\)  or  \(\vb v\cdot\vb w\)dot product \(v_1w_1 + v_2w_2 + \cdots + v_nw_n\) (a number)
\(\nrm{\vb v}\), \(\nrm{\vb v}^2\)length of \(\vb v\), and the squared length \(\vb v\T\vb v = v_1^2 + \cdots + v_n^2\)
\(\vb v \perp \vb w\)\(\vb v\) is orthogonal (perpendicular) to \(\vb w\)
\(\mathbf C(A),\ \mathbf N(A),\ \mathbf C(A\T),\ \mathbf N(A\T)\)column space, nullspace, row space, left nullspace
\(r\), \(m \times n\)the rank of \(A\); \(A\) has \(m\) rows and \(n\) columns
\(\vb x_r\), \(\vb x_n\)the row-space part and the nullspace part of \(\vb x\)

1Orthogonal vectors (4.1)

Perpendicular directions are the most convenient directions in mathematics: moving along one of them changes nothing along the other. In \(\R^2\) and \(\R^3\) we can see a right angle. In \(\R^n\) we cannot, so we need an algebraic test. The test is a dot product.

Definition 1.1. Two vectors \(\vb v\) and \(\vb w\) in \(\R^n\) are orthogonal (perpendicular), written \(\vb v \perp \vb w\), when their dot product is zero: \[ \vb v\T\vb w = v_1 w_1 + v_2 w_2 + \cdots + v_n w_n = 0. \]

The sign of the dot product tells us the angle between the vectors: positive means an angle below \(90^\circ\), negative means an angle above \(90^\circ\), and zero means exactly \(90^\circ\) (Figure 1.1).

Figure 1.1 — the sign of the dot product. The same \(\vb v = (3, 1)\) with three different \(\vb w\). A positive dot product means an angle below \(90^\circ\), zero means exactly \(90^\circ\), negative means more than \(90^\circ\). Only the middle pair is orthogonal.

Example 1.1. Is \((1, 2, 2)\) orthogonal to \((2, -1, 0)\)? Is it orthogonal to \((1, 1, 1)\)?

\[ \begin{aligned} (1,2,2)\cdot(2,-1,0) &= 1\cdot 2 + 2\cdot(-1) + 2\cdot 0 = 2 - 2 + 0 = 0 &&\Rightarrow\ \text{orthogonal} \\ (1,2,2)\cdot(1,1,1) &= 1 + 2 + 2 = 5 \ne 0 &&\Rightarrow\ \text{not orthogonal} \end{aligned} \]

Example 1.2 (book, 4.1). Which pairs among \(\vb v_1 = (1, 2, -2, 1)\), \(\vb v_2 = (4, 0, 4, 0)\), \(\vb v_3 = (1, -1, -1, -1)\) are orthogonal?

\[ \begin{aligned} \vb v_1\T\vb v_2 &= 4 + 0 - 8 + 0 = -4 &&\text{not orthogonal} \\ \vb v_1\T\vb v_3 &= 1 - 2 + 2 - 1 = 0 &&\text{orthogonal} \\ \vb v_2\T\vb v_3 &= 4 + 0 - 4 + 0 = 0 &&\text{orthogonal} \end{aligned} \] So \(\vb v_3\) is orthogonal to both of the others, but \(\vb v_1\) and \(\vb v_2\) are not orthogonal to each other.

Remarks.

Pythagoras

Theorem 1.2 (Pythagoras). For vectors \(\vb v, \vb w\) in \(\R^n\): \[ \vb v \perp \vb w \quad\Longleftrightarrow\quad \nrm{\vb v}^2 + \nrm{\vb w}^2 = \nrm{\vb v + \vb w}^2. \]
Proof. Expand the squared length of \(\vb v + \vb w\) using \(\nrm{\vb u}^2 = \vb u\T\vb u\): \[ \begin{aligned} \nrm{\vb v + \vb w}^2 &= (\vb v + \vb w)\T(\vb v + \vb w) \\ &= \vb v\T\vb v + \vb v\T\vb w + \vb w\T\vb v + \vb w\T\vb w \\ &= \nrm{\vb v}^2 + \nrm{\vb w}^2 + 2\,\vb v\T\vb w \qquad (\text{because } \vb w\T\vb v = \vb v\T\vb w). \end{aligned} \] Therefore \(\nrm{\vb v + \vb w}^2 = \nrm{\vb v}^2 + \nrm{\vb w}^2\) holds exactly when \(2\,\vb v\T\vb w = 0\), that is, when \(\vb v\T\vb w = 0\). ∎

Figure 1.2 — Pythagoras for vectors. \(\vb v = (3, 1)\) and \(\vb w = (-1, 3)\) are orthogonal (\(\vb v\T\vb w = -3 + 3 = 0\)), so the squared lengths add: \(10 + 10 = 20 = \nrm{\vb v + \vb w}^2\). The triangle with sides \(\vb v\), \(\vb w\), \(\vb v + \vb w\) has a right angle at the tip of \(\vb v\).

Example 1.3. Verify Theorem 1.2 for \(\vb v = (1, 2, 2)\) and \(\vb w = (2, -1, 0)\).

\[ \begin{aligned} \nrm{\vb v}^2 &= 1 + 4 + 4 = 9, \qquad \nrm{\vb w}^2 = 4 + 1 + 0 = 5 \\ \vb v + \vb w &= (3, 1, 2), \qquad \nrm{\vb v + \vb w}^2 = 9 + 1 + 4 = 14 = 9 + 5. \end{aligned} \] The squared lengths add, as they must, because \(\vb v\T\vb w = 0\) (Example 1.1).

Finding vectors perpendicular to given vectors

We are often asked the reverse question: not “are these perpendicular?” but “find a vector perpendicular to this one.” There are three cases.

  1. In \(\R^2\), perpendicular to \((a, b)\): swap the entries and change one sign, \((a, b) \to (-b, a)\). Check: \(a(-b) + b(a) = 0\). For example \((3, 4) \to (-4, 3)\).
  2. In \(\R^3\), perpendicular to one vector \(\vb a\): the condition \(\vb a\T\vb y = a_1y_1 + a_2y_2 + a_3y_3 = 0\) is one equation in three unknowns. We may choose two of the \(y\)'s freely and solve for the third. The solutions form a plane through the origin (two free choices means two dimensions). For \(\vb a = (1, 2, 2)\): \(y_1 + 2y_2 + 2y_3 = 0\), and \((2, -1, 0)\), \((0, 1, -1)\), \((2, 0, -1)\) are three of the infinitely many answers.
  3. In \(\R^3\), perpendicular to two vectors \(\vb u\) and \(\vb v\): now there are two equations, \(\vb u\T\vb y = 0\) and \(\vb v\T\vb y = 0\). Each equation by itself describes a plane through the origin, and two such planes meet in a line. In matrix language: put \(\vb u\) and \(\vb v\) as the rows of a \(2 \times 3\) matrix \(A\). The conditions say \(A\vb y = \vb 0\), so the answers are exactly the nullspace of \(A\). When \(\vb u, \vb v\) are independent, the rank is \(r = 2\), so the nullspace has dimension \(n - r = 3 - 2 = 1\): one free variable, one special solution, and every answer is a multiple of it. We find that special solution by Gauss–Jordan elimination, exactly as in Chapter 3.

Figure 1.3 — how many perpendicular vectors are there? Left: perpendicular to one vector \(\vb a\) in \(\R^3\) is one equation, so the answers \(\vb y\) fill a plane (two free choices). Right: perpendicular to two vectors \(\vb u, \vb v\) is two equations, so the answers fill a line (one free choice): the nullspace of the matrix with rows \(\vb u, \vb v\).

Example 1.4. Find \(c\) so that \((1, 2, c)\) is orthogonal to \((4, 1, -2)\).

\[ \begin{aligned} (1, 2, c)\cdot(4, 1, -2) &= 4 + 2 - 2c = 6 - 2c \\ \text{orthogonal} \iff 6 - 2c &= 0 \iff c = 3 \\ \text{check: } (1, 2, 3)\cdot(4, 1, -2) &= 4 + 2 - 6 = 0 \ \checkmark \end{aligned} \]

Example 1.5. Find all vectors perpendicular to both \((1, 1, 1)\) and \((1, 2, 3)\).

  1. Write the conditions as a system. \(y_1 + y_2 + y_3 = 0\) and \(y_1 + 2y_2 + 3y_3 = 0\), that is \(A\vb y = \vb 0\) with \[ A = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix}. \] We want the nullspace of \(A\).
  2. Gauss–Jordan elimination to the reduced row echelon form \(R\): \[ \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix} \xrightarrow{\ R_2 - R_1\ } \begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{bmatrix} \xrightarrow{\ R_1 - R_2\ } \begin{bmatrix} 1 & 0 & -1 \\ 0 & 1 & 2 \end{bmatrix} = R. \]
  3. Read off the special solution. Pivot columns \(1, 2\); free column \(3\). Set the free variable \(y_3 = 1\). Row 1 of \(R\): \(y_1 - y_3 = 0\), so \(y_1 = 1\). Row 2: \(y_2 + 2y_3 = 0\), so \(y_2 = -2\). \[ \vb y = \begin{bmatrix} 1 \\ -2 \\ 1 \end{bmatrix} \]
  4. Answer. Every vector perpendicular to both is a multiple \(c\,(1, -2, 1)\): a line through the origin, as in Figure 1.3.
  5. Check against both original vectors: \((1,1,1)\cdot(1,-2,1) = 1 - 2 + 1 = 0\) and \((1,2,3)\cdot(1,-2,1) = 1 - 4 + 3 = 0\). \(\checkmark\)

Exercise 1.

  1. Is \((1, 3, -2)\) orthogonal to \((4, 0, 2)\)?
  2. Find \(c\) so that \((c, 2, 1) \perp (3, -2, 1)\).
  3. For the two vectors in (a), verify Theorem 1.2.
  4. Find all vectors perpendicular to both \((1, 1, 0)\) and \((0, 1, 1)\).
Show answer
Answer
  1. \((1,3,-2)\cdot(4,0,2) = 4 + 0 - 4 = 0\). Yes, they are orthogonal.
  2. \((c,2,1)\cdot(3,-2,1) = 3c - 4 + 1 = 3c - 3\). Orthogonal means \(3c - 3 = 0\), so \(c = 1\). Check: \((1,2,1)\cdot(3,-2,1) = 3 - 4 + 1 = 0\). \(\checkmark\)
  3. \(\nrm{\vb v}^2 = 1 + 9 + 4 = 14\), \(\nrm{\vb w}^2 = 16 + 0 + 4 = 20\), \(\vb v + \vb w = (5, 3, 0)\), \(\nrm{\vb v + \vb w}^2 = 25 + 9 + 0 = 34 = 14 + 20\). \(\checkmark\)
  4. \(A = \begin{bmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix} \xrightarrow{\ R_1 - R_2\ } \begin{bmatrix} 1 & 0 & -1 \\ 0 & 1 & 1 \end{bmatrix} = R\). Free variable \(y_3 = 1\): \(y_1 = 1\), \(y_2 = -1\). Special solution \((1, -1, 1)\); the answers are all multiples \(c\,(1, -1, 1)\). Check: \(1 - 1 + 0 = 0\) and \(0 - 1 + 1 = 0\). \(\checkmark\)

2Orthogonal subspaces (4.1)

In Chapter 3 every matrix gave us four subspaces. In this section we discover that they come in two perpendicular pairs. This is the second part of the Fundamental Theorem of Linear Algebra, and it is the picture behind everything that follows.

Definition 2.1. Two subspaces \(\mathbf V\) and \(\mathbf W\) of \(\R^n\) are orthogonal when every vector of \(\mathbf V\) is orthogonal to every vector of \(\mathbf W\): \[ \vb v\T\vb w = 0 \quad\text{for all } \vb v \text{ in } \mathbf V \text{ and all } \vb w \text{ in } \mathbf W. \]

Example 2.1. The \(z\)-axis and the \(xy\)-plane in \(\R^3\) are orthogonal subspaces.

A vector on the \(z\)-axis is \((0, 0, c)\); a vector in the \(xy\)-plane is \((a, b, 0)\). Their dot product is \(0\cdot a + 0\cdot b + c\cdot 0 = 0\) for every \(a, b, c\).

Example 2.2. The \(xy\)-plane (the floor) and the \(yz\)-plane (a wall) are not orthogonal subspaces, even though they meet at a right angle.

Both planes contain the \(y\)-axis. The vector \((0, 1, 0)\) lies in both, and \((0,1,0)\cdot(0,1,0) = 1 \ne 0\), so Definition 2.1 fails. In fact two planes through \(\vb 0\) in \(\R^3\) are never orthogonal subspaces: their dimensions add to \(2 + 2 = 4 > 3\), so they must share a line.

Figure 2.1 — orthogonal subspaces or not. Left: the \(z\)-axis and the \(xy\)-plane. Every \((0, 0, c)\) is orthogonal to every \((a, b, 0)\), so they are orthogonal subspaces. Right: a floor and a wall meet at a right angle, but they share a whole line. A nonzero vector on that line lies in both planes and is not orthogonal to itself, so two planes in \(\R^3\) are never orthogonal subspaces.

Method 2.1 (testing two subspaces). We cannot check infinitely many pairs, and we need not. Let \(\vb v_1, \dots, \vb v_k\) span \(\mathbf V\) and \(\vb w_1, \dots, \vb w_l\) span \(\mathbf W\).
  1. Compute every dot product \(\vb v_i\T\vb w_j\).
  2. If all of them are \(0\), the subspaces are orthogonal. If one of them is not \(0\), they are not.

Why this is enough: any \(\vb v = c_1\vb v_1 + \cdots + c_k\vb v_k\) and \(\vb w = d_1\vb w_1 + \cdots + d_l\vb w_l\) have \(\vb v\T\vb w = \sum_{i,j} c_i d_j\,\vb v_i\T\vb w_j\), which is \(0\) when every \(\vb v_i\T\vb w_j\) is \(0\).

Example 2.3. Let \(\mathbf V\) be the plane spanned by \((1,1,0)\) and \((0,1,1)\), and \(\mathbf W\) the line through \((1,-1,1)\). Are they orthogonal?

\[ \begin{aligned} (1,1,0)\cdot(1,-1,1) &= 1 - 1 + 0 = 0 \\ (0,1,1)\cdot(1,-1,1) &= 0 - 1 + 1 = 0 \end{aligned} \] Both dot products are \(0\), so by Method 2.1 the subspaces are orthogonal.

The nullspace is orthogonal to the row space

Theorem 2.2 (Fundamental Theorem of Linear Algebra, Part 2). For every \(m \times n\) matrix \(A\):
  1. The nullspace \(\mathbf N(A)\) is orthogonal to the row space \(\mathbf C(A\T)\). Both are subspaces of \(\R^n\).
  2. The left nullspace \(\mathbf N(A\T)\) is orthogonal to the column space \(\mathbf C(A)\). Both are subspaces of \(\R^m\).
Proof. (1) Let \(\vb x\) be any vector in \(\mathbf N(A)\), so \(A\vb x = \vb 0\). Let \(\vb r\) be any vector in the row space \(\mathbf C(A\T)\). Since the row space consists of all combinations of the rows, \(\vb r = A\T\vb y\) for some \(\vb y\) in \(\R^m\). Then \[ \vb r\T\vb x = (A\T\vb y)\T\vb x = \vb y\T A\vb x = \vb y\T\vb 0 = 0. \] So every vector of the row space is orthogonal to every vector of the nullspace.
(2) Apply part (1) to the matrix \(A\T\): the nullspace of \(A\T\) is orthogonal to the row space of \(A\T\), which is \(\mathbf C((A\T)\T) = \mathbf C(A)\). ∎

Example 2.4. \(A = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 7 \end{bmatrix}\) has \(\vb x = (2, -1, 0)\) in its nullspace. Check Theorem 2.2 on this vector.

\[ \begin{aligned} (\text{row } 1)\cdot\vb x &= 2 - 2 + 0 = 0 \\ (\text{row } 2)\cdot\vb x &= 4 - 4 + 0 = 0 \\ \big(3(\text{row } 1) - (\text{row } 2)\big)\cdot\vb x &= (1, 2, 2)\cdot(2, -1, 0) = 2 - 2 + 0 = 0 \end{aligned} \] Every row, and every combination of rows, is perpendicular to \(\vb x\).

Orthogonal complements

Definition 2.3. The orthogonal complement of a subspace \(\mathbf V\) of \(\R^n\), written \(\mathbf V^{\perp}\), is the set of all vectors in \(\R^n\) that are orthogonal to every vector of \(\mathbf V\).

The two subspaces in each pair of Theorem 2.2 are not just orthogonal: together they are large enough to fill the whole space. The dimensions show it.

Theorem 2.4. The nullspace is the orthogonal complement of the row space, and the left nullspace is the orthogonal complement of the column space: \[ \begin{aligned} &\text{in } \R^n: && \dim\mathbf C(A\T) + \dim\mathbf N(A) = r + (n - r) = n, \\ &\text{in } \R^m: && \dim\mathbf C(A) + \dim\mathbf N(A\T) = r + (m - r) = m. \end{aligned} \] Consequently every \(\vb x\) in \(\R^n\) splits in exactly one way as \[ \vb x = \vb x_r + \vb x_n, \qquad \vb x_r \text{ in the row space},\quad \vb x_n \text{ in the nullspace}, \] and the matrix sees only the row-space part: \[ A\vb x = A\vb x_r + A\vb x_n = A\vb x_r + \vb 0 = A\vb x_r. \]

Figure 2.2 — the four subspaces (Fundamental Theorem, Part 2). Each subspace is drawn as a line through \(\vb 0\). Left, in \(\R^n\): the row space (dimension \(r\)) and the nullspace (dimension \(n - r\)) are perpendicular, and every \(\vb x\) splits as \(\vb x_r + \vb x_n\) (dashed parallelogram). Right, in \(\R^m\): the column space and the left nullspace are perpendicular. \(A\) sends \(\vb x_r\) to \(\vb b\) and sends \(\vb x_n\) to \(\vb 0\), so \(A\vb x = A\vb x_r = \vb b\).

Example 2.5. \(A = \begin{bmatrix} 1 & 2 \end{bmatrix}\) has one row. Split \(\vb x = (3, 1)\) into \(\vb x_r + \vb x_n\).

  1. The two subspaces. The row space is the line through \((1, 2)\), so \(\vb x_r = c\,(1, 2)\). The nullspace is all \(\vb y\) with \(y_1 + 2y_2 = 0\), the line through \((2, -1)\), so \(\vb x_n = d\,(2, -1)\).
  2. Solve for \(c\) and \(d\). We need \(c\,(1, 2) + d\,(2, -1) = (3, 1)\), that is \(c + 2d = 3\) and \(2c - d = 1\): \[ \left[\begin{array}{cc|c} 1 & 2 & 3 \\ 2 & -1 & 1 \end{array}\right] \xrightarrow{\ R_2 - 2R_1\ } \left[\begin{array}{cc|c} 1 & 2 & 3 \\ 0 & -5 & -5 \end{array}\right] \quad\Rightarrow\quad d = 1,\quad c = 3 - 2(1) = 1. \]
  3. Answer. \(\vb x_r = (1, 2)\) and \(\vb x_n = (2, -1)\).
  4. Checks. \(\vb x_r + \vb x_n = (3, 1) = \vb x\). \(A\vb x_n = 2 - 2 = 0\). \(A\vb x = 3 + 2 = 5\) and \(A\vb x_r = 1 + 4 = 5\): equal, as Theorem 2.4 says. \(\checkmark\)

In Part 2 (projections) we will see a quicker way to find \(\vb x_r\): it is the projection of \(\vb x\) onto the row space.

Example 2.6. \(A\) is \(3 \times 4\) with rank \(2\). Give the dimensions of the four subspaces.

Example 2.7. \(A = \begin{bmatrix} 1 & 2 & 0 & 1 \\ 2 & 4 & 1 & 3 \\ 3 & 6 & 1 & 4 \end{bmatrix}\). Find the dimensions of the four subspaces.

Here the rank is not given, so we must find it first.

  1. Find the rank by elimination. \[ \begin{bmatrix} 1 & 2 & 0 & 1 \\ 2 & 4 & 1 & 3 \\ 3 & 6 & 1 & 4 \end{bmatrix} \xrightarrow{\ R_2 - 2R_1,\ \ R_3 - 3R_1\ } \begin{bmatrix} 1 & 2 & 0 & 1 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 1 \end{bmatrix} \xrightarrow{\ R_3 - R_2\ } \begin{bmatrix} 1 & 2 & 0 & 1 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 0 \end{bmatrix} = R. \] Two pivots (columns 1 and 3), so \(r = 2\). Also \(m = 3\) rows and \(n = 4\) columns.
  2. The four dimensions.
    • Row space \(\mathbf C(A\T)\): \(r = 2\).   Nullspace \(\mathbf N(A)\): \(n - r = 4 - 2 = 2\).   Together \(2 + 2 = 4\) in \(\R^4\).
    • Column space \(\mathbf C(A)\): \(r = 2\).   Left nullspace \(\mathbf N(A\T)\): \(m - r = 3 - 2 = 1\).   Together \(2 + 1 = 3\) in \(\R^3\).
  3. Check Theorem 2.2 on this matrix. The free variables are \(x_2\) and \(x_4\). From \(R\), the special solutions are \(\vb s_1 = (-2, 1, 0, 0)\) (set \(x_2 = 1, x_4 = 0\)) and \(\vb s_2 = (-1, 0, -1, 1)\) (set \(x_2 = 0, x_4 = 1\)). Each is orthogonal to every row of \(A\); for example \[ \begin{aligned} (2, 4, 1, 3)\cdot(-1, 0, -1, 1) &= -2 + 0 - 1 + 3 = 0, \\ (3, 6, 1, 4)\cdot(-2, 1, 0, 0) &= -6 + 6 + 0 + 0 = 0. \end{aligned} \]

Exercise 2.

  1. Are the line through \((1, 1, 1)\) and the plane spanned by \((1, -1, 0)\) and \((1, 0, -1)\) orthogonal subspaces?
  2. \(A\) is \(3 \times 5\) with rank \(2\). Give the four dimensions.
  3. \(A = \begin{bmatrix} 1 & 1 \end{bmatrix}\). Split \(\vb x = (4, 2)\) into \(\vb x_r + \vb x_n\).
  4. \(A = \begin{bmatrix} 1 & 1 & 2 \\ 2 & 2 & 4 \\ 1 & 0 & 1 \end{bmatrix}\). Find the rank, then the dimensions of the four subspaces.
Show answer
Answer
  1. Test every basis pair: \((1,1,1)\cdot(1,-1,0) = 1 - 1 + 0 = 0\) and \((1,1,1)\cdot(1,0,-1) = 1 + 0 - 1 = 0\). Both are \(0\), so yes, the line and the plane are orthogonal subspaces. (The plane is \(x + y + z = 0\) and \((1,1,1)\) is its normal vector.)
  2. \(m = 3\), \(n = 5\), \(r = 2\). Row space \(2\); nullspace \(5 - 2 = 3\); column space \(2\); left nullspace \(3 - 2 = 1\). Checks: \(2 + 3 = 5\) and \(2 + 1 = 3\).
  3. Row space: the line through \((1, 1)\), so \(\vb x_r = c\,(1,1)\). Nullspace: \(y_1 + y_2 = 0\), the line through \((1, -1)\), so \(\vb x_n = d\,(1,-1)\). Then \(c + d = 4\) and \(c - d = 2\); subtracting, \(2d = 2\), \(d = 1\), \(c = 3\). So \(\vb x_r = (3, 3)\), \(\vb x_n = (1, -1)\). Checks: \((3,3) + (1,-1) = (4,2)\); \(A\vb x_n = 1 - 1 = 0\); \(A\vb x = 6 = A\vb x_r\). \(\checkmark\)
  4. Elimination: \(R_2 - 2R_1\) gives the zero row \((0, 0, 0)\), and \(R_3 - R_1 = (0, -1, -1)\). Two pivots, so \(r = 2\); here \(m = n = 3\). Row space \(2\); nullspace \(3 - 2 = 1\); column space \(2\); left nullspace \(3 - 2 = 1\). (The nullspace is the line through \((-1, -1, 1)\): check \(A(-1, -1, 1) = (-1 - 1 + 2,\ -2 - 2 + 4,\ -1 + 0 + 1) = \vb 0\).)

3Question types and model answers

Almost every question on this part is one of the four types below. Each model answer is written out the way I expect to see it on paper: the formula or definition first, then the numbers, the arithmetic, the check, and a one-line conclusion.

Type 1 — Orthogonal or not; find \(c\)

Question. Find \(c\) so that \((2, c, 3)\) is orthogonal to \((1, 4, -2)\).

\[ \begin{aligned} (2, c, 3)\cdot(1, 4, -2) &= 2 + 4c - 6 = 4c - 4 \\ \text{orthogonal} \iff 4c - 4 &= 0 \iff c = 1 \\ \text{check: } (2, 1, 3)\cdot(1, 4, -2) &= 2 + 4 - 6 = 0 \ \checkmark \end{aligned} \]

Type 2 — All vectors perpendicular to two given vectors

Question. Find all vectors perpendicular to both \((1, 2, 1)\) and \((1, 3, 2)\).

\[ \begin{aligned} &\text{conditions } \vb y\cdot(1,2,1) = 0,\ \ \vb y\cdot(1,3,2) = 0 \quad\text{i.e.}\quad A\vb y = \vb 0,\quad A = \begin{bmatrix} 1 & 2 & 1 \\ 1 & 3 & 2 \end{bmatrix} \\ &\begin{bmatrix} 1 & 2 & 1 \\ 1 & 3 & 2 \end{bmatrix} \xrightarrow{\ R_2 - R_1\ } \begin{bmatrix} 1 & 2 & 1 \\ 0 & 1 & 1 \end{bmatrix} \xrightarrow{\ R_1 - 2R_2\ } \begin{bmatrix} 1 & 0 & -1 \\ 0 & 1 & 1 \end{bmatrix} = R \\ &\text{free variable } y_3 = 1:\quad y_1 = 1,\ \ y_2 = -1 \qquad \vb y = (1, -1, 1) \\ &\text{all such vectors: } c\,(1, -1, 1), \text{ a line through } \vb 0 \\ &\text{check: } (1,2,1)\cdot(1,-1,1) = 1 - 2 + 1 = 0,\quad (1,3,2)\cdot(1,-1,1) = 1 - 3 + 2 = 0 \ \checkmark \end{aligned} \]

Type 3 — Are two subspaces orthogonal?

Question. Are \(\mathbf V = \operatorname{span}\{(1,1,0), (0,1,1)\}\) and \(\mathbf W = \operatorname{span}\{(1,-1,1)\}\) orthogonal subspaces?

\[ \begin{aligned} &\text{test every pair of basis vectors:} \\ &(1,1,0)\cdot(1,-1,1) = 1 - 1 + 0 = 0 \\ &(0,1,1)\cdot(1,-1,1) = 0 - 1 + 1 = 0 \\ &\text{all dot products are } 0, \text{ so every } \vb v \text{ in } \mathbf V \text{ is orthogonal to every } \vb w \text{ in } \mathbf W. \\ &\text{Conclusion: } \mathbf V \perp \mathbf W. \end{aligned} \]

Type 4 — Dimensions of the four subspaces; the nullspace is orthogonal to the row space

Question. \(A\) is \(4 \times 5\) of rank \(3\). Give the dimensions of the four subspaces and say which pairs are orthogonal complements. Prove that \(\mathbf N(A)\) is orthogonal to \(\mathbf C(A\T)\).

\[ \begin{aligned} &\dim\mathbf C(A\T) = r = 3, \qquad \dim\mathbf N(A) = n - r = 5 - 3 = 2 \\ &\text{orthogonal complements in } \R^5:\ 3 + 2 = 5 \\ &\dim\mathbf C(A) = r = 3, \qquad \dim\mathbf N(A\T) = m - r = 4 - 3 = 1 \\ &\text{orthogonal complements in } \R^4:\ 3 + 1 = 4 \end{aligned} \]

Proof. Let \(\vb x\) be in \(\mathbf N(A)\), so \(A\vb x = \vb 0\), and let \(\vb r\) be in \(\mathbf C(A\T)\), so \(\vb r = A\T\vb y\) for some \(\vb y\). Then \(\vb r\T\vb x = (A\T\vb y)\T\vb x = \vb y\T A\vb x = \vb y\T\vb 0 = 0\). Hence every vector of the nullspace is orthogonal to every vector of the row space. \(\blacksquare\)

If the rank is not given, find it first by elimination (count the pivots), as in Example 2.7.

4Practice test

Six questions, about forty-five minutes of work. Write full answers on paper first and open each answer only afterwards. Question 6 is theory.

Q1. Are \((2, -1, 3)\) and \((1, 5, 1)\) orthogonal? Verify Pythagoras for this pair.

Show answer
Answer
\((2,-1,3)\cdot(1,5,1) = 2 - 5 + 3 = 0\), so yes, they are orthogonal.
\(\nrm{\vb v}^2 = 4 + 1 + 9 = 14\), \(\nrm{\vb w}^2 = 1 + 25 + 1 = 27\), \(\vb v + \vb w = (3, 4, 4)\), \(\nrm{\vb v + \vb w}^2 = 9 + 16 + 16 = 41 = 14 + 27\). \(\checkmark\)

Q2. Find \(c\) so that \((3, c, -1)\) is orthogonal to \((2, 1, 4)\).

Show answer
Answer
\((3,c,-1)\cdot(2,1,4) = 6 + c - 4 = c + 2\). Orthogonal means \(c + 2 = 0\), so \(c = -2\). Check: \((3,-2,-1)\cdot(2,1,4) = 6 - 2 - 4 = 0\). \(\checkmark\)

Q3. Find all vectors perpendicular to both \((1, 0, 1)\) and \((0, 1, 1)\). What kind of set do they form?

Show answer
Answer
The conditions \(y_1 + y_3 = 0\) and \(y_2 + y_3 = 0\) are \(A\vb y = \vb 0\) with \(A = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \end{bmatrix}\), which is already in reduced row echelon form: pivot columns \(1, 2\), free column \(3\). Set \(y_3 = 1\): row 1 gives \(y_1 = -1\), row 2 gives \(y_2 = -1\). Special solution \(\vb y = (-1, -1, 1)\). Check: \(-1 + 0 + 1 = 0\) and \(0 - 1 + 1 = 0\). \(\checkmark\) All such vectors are the multiples \(c\,(-1, -1, 1)\): the nullspace of \(A\), a line through the origin (dimension \(3 - 2 = 1\)).

Q4. \(A\) is \(4 \times 5\) with rank \(3\). Give the dimensions of the four subspaces. Which pairs are orthogonal complements, and in which spaces?

Show answer
Answer
\(m = 4\), \(n = 5\), \(r = 3\).
Row space: \(r = 3\). Nullspace: \(n - r = 5 - 3 = 2\). Orthogonal complements in \(\R^5\) \((3 + 2 = 5)\).
Column space: \(r = 3\). Left nullspace: \(m - r = 4 - 3 = 1\). Orthogonal complements in \(\R^4\) \((3 + 1 = 4)\).

Q5. \(A = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 1 & 1 \end{bmatrix}\). Find the rank and the dimensions of the four subspaces, and give a vector that spans the nullspace.

Show answer
Answer
Elimination: \(R_2 - 2R_1 = (0, 0, 0)\) and \(R_3 - R_1 = (0, -1, -2)\), so \[ A \to \begin{bmatrix} 1 & 2 & 3 \\ 0 & -1 & -2 \\ 0 & 0 & 0 \end{bmatrix}. \] Two pivots, so \(r = 2\); here \(m = n = 3\). Row space \(2\); nullspace \(3 - 2 = 1\); column space \(2\); left nullspace \(3 - 2 = 1\).
Nullspace: free variable \(x_3 = 1\); row 2 gives \(-x_2 - 2 = 0\), \(x_2 = -2\); row 1 gives \(x_1 - 4 + 3 = 0\), \(x_1 = 1\). The nullspace is spanned by \((1, -2, 1)\). Check: \(A(1,-2,1) = (1 - 4 + 3,\ 2 - 8 + 6,\ 1 - 2 + 1) = \vb 0\). \(\checkmark\) (It is orthogonal to every row, as Theorem 2.2 says.)

Q6 (theory). Prove that every vector in the nullspace of \(A\) is orthogonal to every vector in the row space of \(A\). Then explain why two planes through the origin in \(\R^3\) can never be orthogonal subspaces.

Show answer
Answer
Let \(\vb x\) be in \(\mathbf N(A)\), so \(A\vb x = \vb 0\). Let \(\vb r\) be in \(\mathbf C(A\T)\); the row space consists of all combinations of the rows, so \(\vb r = A\T\vb y\) for some \(\vb y\) in \(\R^m\). Then \(\vb r\T\vb x = (A\T\vb y)\T\vb x = \vb y\T A\vb x = \vb y\T\vb 0 = 0\). Hence \(\vb r \perp \vb x\) for every such pair, and \(\mathbf N(A) \perp \mathbf C(A\T)\). \(\blacksquare\)
Two planes through \(\vb 0\) in \(\R^3\) have dimensions \(2 + 2 = 4 > 3\), so they must share at least a line. A nonzero vector \(\vb v\) on that line lies in both planes, and \(\vb v\T\vb v = \nrm{\vb v}^2 \ne 0\). So the planes cannot be orthogonal subspaces.

5Formula sheet

TopicFormulas
Orthogonal vectors\(\vb v \perp \vb w \iff \vb v\T\vb w = v_1w_1 + \cdots + v_nw_n = 0\)
sign of \(\vb v\T\vb w\): positive means angle below \(90^\circ\), negative means above \(90^\circ\)
Pythagoras\(\vb v \perp \vb w \iff \nrm{\vb v}^2 + \nrm{\vb w}^2 = \nrm{\vb v + \vb w}^2\)
because \(\nrm{\vb v + \vb w}^2 = \nrm{\vb v}^2 + \nrm{\vb w}^2 + 2\,\vb v\T\vb w\)
Perpendicular vectorsin \(\R^2\): \((a, b) \to (-b, a)\)
to one vector in \(\R^3\): one equation, a plane of answers
to two vectors: the nullspace of the \(2 \times 3\) matrix with those rows (Gauss–Jordan), a line of answers
Orthogonal subspaces\(\vb v\T\vb w = 0\) for every \(\vb v\) in \(\mathbf V\), every \(\vb w\) in \(\mathbf W\)
test: every pair of basis vectors
two planes in \(\R^3\) are never orthogonal subspaces (they share a line)
Fundamental Theorem, Part 2\(\mathbf N(A) \perp \mathbf C(A\T)\) in \(\R^n\);   \(\mathbf N(A\T) \perp \mathbf C(A)\) in \(\R^m\)
proof: \(\vb r = A\T\vb y\), \(A\vb x = \vb 0\) give \(\vb r\T\vb x = \vb y\T A\vb x = 0\)
Dimensions\(\dim\mathbf C(A\T) = r\),   \(\dim\mathbf N(A) = n - r\),   sum \(n\)
\(\dim\mathbf C(A) = r\),   \(\dim\mathbf N(A\T) = m - r\),   sum \(m\)
\(r\) = number of pivots after elimination
Splitting \(\vb x\)\(\vb x = \vb x_r + \vb x_n\) (row-space part + nullspace part)
\(A\vb x_n = \vb 0\), so \(A\vb x = A\vb x_r\)