Determine the joint distribution of the least-squares

Question

Question 1. The least-squares solution β' for a linear regression model is β' = ( β₀', β₁')^T, where

β' = _i=1∑^N (x_i - x')(y_i-y')/_i=1∑^N (x_i - x')²

and

β' = y^- - β₁'x^-

Verify that β' solves the normal equations AT Aβ' = A^Ty for data y.

Question 2. For an outcome Y = y of a linear model Y = Aβ + ε, where ε ~ N(0, σ²I), show that the least-squares solution 13 solving the normal equations A^TA = A^Ty is the maximum likelihood estimator for β.

Question 3. Simulate N = 100 outcomes from the model

Y = 2.5 - 5X² + ε,

where X ~ U[0, 0, 7] and ε ~ N(0, 0, 1²) independently. From your simulated outcomes of (X, Y), fit a sequence of polynomial regression models to your data via least-squares, of degree n = 0, 1,2,3.

Provide plots of your regression curves with the data superimposed, plots of the residuals, and supply your code.

Question 4. Extend your solution to Q3, by calculating the mean error sum-of-squares; and calculate the regression sum of squares associated with increasing the model degree by 1. Based on your residual plots in Q3, comment on the validity of the error model in each case. If the error model is valid, compute the appropriate F statistic for testing whether the additional term in the model is needed, together with the associated p-value. For valid tests, conclude whether or not the additional term is needed. Supply your calculated output, comments, and your code.

Question 5. For a linear regression model:

(a) Determine the joint distribution of the least-squares solution β'.

(b) Using (a), determine the distribution of the linear regression line at an arbitrary explanatory point x; that is, the distribution of Y' where Y' = β₀' + β₁'x. Comment on how to go about constructing a confidence interval for the linear regression line at an arbitrary explanatory point x.

Question 6. Revisiting Q3, fit a linear regression model with transformed explanatory variables X^~ = X². Plot a 95% confidence interval and a 95% prediction interval on the sample plot as your regression curve and the data points, assuming the variance σ² = 0.1² is known. Supply your plot and code.

Where β' is refer to β^{^.}