7.3 Method of Least Square

What is the method of least squares

• remember how in 7.2 Projection onto subspaces and Gram-Schmidt that a projection of x onto W

  • represents the point on W that is the closest to x

  • 

  • We can see this as the same way if a vector is inconsistent and we need the solution so we can just find the solution closest to the orignal vector

• Any $\vec{x}$ which is a solution to $A\vec{x}={\vec{b}}_0$ is the closest in minimizing the least squares error, we call such a solution a best fit solution or a least squares error solution

Finding the best-fit solution to $A\vec{x}=\vec{b}$

• it is possible to find the best-fit solution to an inconsistent system $A\vec{x}=\vec{b}$ without finding ${\vec{b}}_0$ by projection

  • recall that row(A) and null(A) are “orthogonal to eachother”. that is:

    • $Null(A)=(Row(A){)}^{\perp }$

  • similarly

    • $Null(A^T)=(Row(A^T){)}^{\perp }=Col(A{)}^T$

  • Since the perp part $\vec{b}-{\vec{b}}_0$ is orthogonal to Col(A), $\vec{b}-{\vec{b}}_0$ is in $Null(A^T)$ so we can write

    • $A^T(\vec{b}-{\vec{b}}_0)=\vec{0}<=>A^T{\vec{b}}_0=A^T\vec{b}$

    • And replacing ${\vec{b}}_0=A\vec{x}$

      • $A^{T}A\vec{x}=A^T\vec{b}$
  • 

Examples

• Finding best fit line

•