Problem: Bias/Variance
Suppose you observe x,y pairs where y = f(x)+ and e is normally distributed with mean zero. Given a hypothesis h, the squared error of h can be decomposed into terms for noise, bias, and variance as follows:
E[(h(x} - y)'] = E[(h(x) -- h(x))2] + (h(x) - f(x))2 +E[(y - f(x))2]
Note that h(x) denotes the expected or average value of h(z).
1) Which of the terms above correspond to noise, bias, and variance?
2) Explain the meaning of each term, giving both an intuitive definition and a description of how that intuition is captured by the mathematical definitions above.
3) Describe in qualitative terms how the bias and variance of the k nearest neighbor classifier change as a function of k. Justify your answer.