Problem 1 [EM]
Write a 2-dimensional RNG for a Gaussian mixture model (GMM) pdf with 2 sub-populations. Use any function/sub-routine available in your language of choice.
Implement the expectation maximization (EM) algorithm for estimating the pdf parameters of 2-D GMMs from samples.
Compare the quality and speed your GMM-EM estimation on 300 samples of different GMM distributions (e.g. spherical vs ellipsoidal covariance, close vs well-separated subpopulations).
Problem 2 [Testing Faith]
Download the "old faithful" data set from blackboard. This has samples of a 2-D random variable: the first dimension is duration of the geyser eruption, the second is waiting time for the next eruption. Apply your GMM-EM algorithm to fit the data to a GMM pdf.
How many EM iterations are needed for convergence? Plot a contour plot of your final GMM pdf. Overlay the contour plot with a scatterplot of the data set. How would you use the GMM pdf estimates to cluster the data?
Problem 3 [Noise in GMM-EM]
Modify your GMM-EM routine by sampling and injecting Gaussian noise into the old faithful data at each iteration. Scale the noise to a fraction of the standard deviation in each dimension. And let the noise standard deviation decay at each iteration (e.g. inversely proportional to the square of the iteration counter).
Compare the average convergence time of the GMM-EM with and without noise. Plot the average convergence time for different initial noise standard deviations.
Turn in:
- A summary of your experiments including any relevant plots
- brief discussions of the results
- a print out of your code.
The following link gives a detailed tutorial about EM algorithms with close emphasis on EM for GMMs in section 3.2: https://mayagupta.org/publications/EMbookGuptaChen2010.pdf
You are not required to go through the whole paper in detail. But I recommend it.
Section 2.2 of this next paper also discusses the update equations for the GMM-EM algorithm: https://sipi.usc.edui-kosko/Noisy-Clustering-Neural-Networks.pdf
You can find more details about clustering with GMM-EM in the second paper linked above.