[SOLVED] Cs329 homework #2

Question 1
(a) [True or False] If two sets of variables are jointly Gaussian, then the conditional distribution of
one set conditioned on the other is again Gaussian. Similarly, the marginal distribution of either
set is also Gaussian
(b) Consider a partitioning of the components of into three groups , , and , with a
corresponding partitioning of the mean vector and of the covariance matrix in the form
Find an expression for the conditional distribution in which has been marginalized
out.
Question 2
Consider a joint distribution over the variable
whose mean and covariance are given by
(a) Show that the marginal distribution is given by .
(b) Show that the conditional distribution is given by .
Question 3
Show that the covariance matrix that maximizes the log likelihood function is given by the
sample covariance
Is the final result symmetric and positive definite (provided the sample covariance is nonsingular)?
Hints
(a) To find the maximum likelihood solution for the covariance matrix of a multivariate
Gaussian, we need to maximize the log likelihood function with respect to . The log
likelihood function is given by
(b) The derivative of the inverse of a matrix can be expressed as
We have the following properties
Question 4
(a) Derive an expression for the sequential estimation of the variance of a univariate Gaussian
distribution, by starting with the maximum likelihood expression
Verify that substituting the expression for a Gaussian distribution into the Robbins-Monro
sequential estimation formula gives a result of the same form, and hence obtain an expression
for the corresponding coefficients .
(b) Derive an expression for the sequential estimation of the covariance of a multivariate Gaussian
distribution, by starting with the maximum likelihood expression
Verify that substituting the expression for a Gaussian distribution into the Robbins-Monro
sequential estimation formula gives a result of the same form, and hence obtain an expression
for the corresponding coefficients .
Hints
(a) Consider the result for the maximum likelihood estimator of the
mean , which we will denote by when it is based on observations. If we dissect
out the contribution from the final data point , we obtain
(b) Robbins-Monro for maximum likelihood
K Norm Accuracy(%)
3 L1
3 L2
3 L-inf
5 L1
5 L2
5 L-inf
7 L1
7 L2
7 L-inf
Question 5
Consider a -dimensional Gaussian random variable with distribution in which the
covariance is known and for which we wish to infer the mean from a set of observations
. Given a prior distribution , find the
corresponding posterior distribution .
Program Question
Use online judge to solve this problem
In this coding exercise, you will implement the K-nearest Neighbors (KNN) algorithm. You are
provided with a Jupyter Notebook just for reference. The requirement on online judge will be
very different from the notebook.
This is a classification problem and we will use the Breast Cancer dataset:
Table1: Accuracy for the KNN classification problem on the validation set
A training data (X train) is provided which has several datapoints, and each datapoint is a pdimensional vector (i.e., p features). Your task is to implement the K-nearest neighbors algorithm.
Use the Euclidean distance.