multiplelinearregressiontheequivalenceofleastsquares.docVIP

Multiple Linear Regression: The Equivalence of Least Squares Covariance In these notes we will address the linear model (1.0) Specifically, we will compare the least squares (LS) solution and the theoretical covariance solution for the model parameters. The method of LS typically is framed in relation to data; not random variables. In this framework, the above model becomes: . (1.1) Given the data , define the matrices (or, arrays): ; ; . Then the LS solution to (1b) is given by: . (1.2) While (2) has a very compact form, it is arrived at by solving the LS equations, and then rearranging them into matrix form. This process is very computational, and, in this author’s opinion, lacking in insight. In order to begin to gain some insight into (1.2), write it as: . (1.3) The three individual equations associated with (1.3) are: . (1.4) In (1.4) we have defined terms such as . To relate this term to the 2-D random variable , recall that the natural estimator of is simply . Recall also that . Hence, we can write . Using this notation, then (1.4) may be written as: . (1.5) The first equation in (1.5) is: . (1.6a) In words, (1.6a) shows that the estimator is an unbiased estimator. The second equation in (1.5) is: . Rearranging this equation, we have: . In view of (1.6a), this equation, in turn, becomes: . (1.6b) Carrying out the same steps in relation to the third equation in (1.5), we arrive at: . (1.6c) Hence, (1.5) can be written as the following two equations: