Determinant and Inverse of Matrix

Given a matrix M:

• Calculate the determinant of M in terms of r. [4pts]
• For what value(s) of r does M¹ not exist? Why? What does it mean in terms of rank and singularity of M for these values of r? [3pts]
• Calculate M¹ by hand for r = 4. [5pts] (Hint 1: Please double check your answer and make sure MM¹ = I)
• Find the determinant of M¹ for r = 4. [3pts]

1.2 Characteristic Equation

Consider the eigenvalue problem:

Ax = x,x 6= 0

where x is a non-zero eigenvector and is eigenvalue of A. Prove that the determinant |A I| = 0.

1.3 Eigenvalues and Eigenvectors [10pts] Given a matrix A:

• Calculate the eigenvalues of A as a function of x [5 pts]
• Find the normalized eigenvectors of matrix A [5 pts]

2 Expectation, Co-variance and Independence [18pts]

Suppose X,Y and Z are three different random variables. Let X obey a Bernouli Distribution. The probability disbribution function is

c is a constant here. Let Y obey a standard Normal (Gaussian) distribution, which can be written as Y N(0,1). X and Y are independent. Meanwhile, let Z = XY .

• Show that Z also follows a Normal (Gaussian) distribution. Calculate the Expectation and Variance of Z. [9pts] (Hint: Sum rule and conditional probability formula)
• How should we choose c such that Y and Z are uncorrelated(which means Cov(Y,Z) = 0)? [5pts]
• Are Y and Z independent? Make use of probabilities to show your conclusion. Example: P(Y (1,0)) and P(Z (2c,3c)) [4pts]

3 Optimization

Optimization problems are related to minimizing a function (usually termed loss, cost or error function) or maximizing a function (such as the likelihood) with respect to some variable x. The Kuhn-Tucker conditions are first-order conditions that provide a unified treatment of constraint optimization. In this question, you will be solving the following optimization problem:

max f(x,y) = 2x² + 3xy x,y

s.t.

• Specify the Legrange function [2 pts]
• List the KKT conditions [2 pts]
• Solve for 4 possibilities formed by each constraint being active or inactive [5 pts]
• List all candidate points [4 pts]
• Check for maximality and sufficiency [2 pts]

4 Maximum Likelihood

4.1 Discrete Example

Suppose we have two types of coins, A and B. The probability of a Type A coin showing heads is . The probability of a Type B coin showing heads is 2. Here, we have a bunch of coins of either type A or B. Each time we choose one coin and flip it. We do this experiment 10 times and the results are shown in the chart below. (Hint: The probabilities aforementioned are for the particular sequence below.)

• What is the likelihood of the result given ? [4pts]
• What is the maximum likelihood estimation for ? [6pts]

4.2 Normal distribution [15 pts](Bonus for Undergrads)

Suppose that we observe samples of a known function g(t) = t³ with unknown amplitude at (known) arbitrary locations t₁,,t_N, and these samples are corrupted by Gaussian noise. That is, we observe the sequence of random variables

X_n = t³_n + Z_n, n = 1,,N

where the Z_n are independent and Z_n Normal

• Given X₁ = x₁,,X_N = x_N, compute the log likelihood function

`(;x₁,,x_N) = logf_X1,,XN (x₁,,x_N;) = log(f_X1 (x₁;)f_X2 (x₂;)f_XN (x_N;))

Note that the X_n are independent (as the last equality is suggesting) but not identically distributed (they have different means). [9pts]

• Compute the MLE for . [6pts]

4.3 Bonus for undergrads [10 pts]

The C.D.F of independent random variables X₁,X₂,,X_n is

, x >

where 0, 0.

• Write down the P.D.F of above independent random variables. [4pts]
• Find the MLEs of and . [6pts]

5 Information Theory

5.1 Marginal Distribution

Suppose the joint probability distribution of two binary random variables X and Y are given as follows.

• Show the marginal distribution of X and Y , respectively. [3pts]
• Find mutual information for the joint probability distribution in the previous question [3pts]

5.2 Mutual Information and Entropy

Given a dataset as below.

We want to decide whether an individual working in an essential services industry should be allowed to work or self-quarantine. Each input has four features (x₁, x₂, x₃, x₄): Age, Immunity, Travelled, Underlying Conditions. The decision (quarantine vs not) is represented as Y .

• Find entropy H(Y ). [3pts]
• Find conditional entropy H(Y |x₁), H(Y |x₄), respectively. [8pts]
• Find mutual information I(x₁,Y ) and I(x₄,Y ) and determine which one

(x₁ or x₄) is more informative. [4pts]

• Find joint entropy H(Y,x₃). [4pts]

5.3 Entropy Proofs

• Suppose X and Y are independent. Show that H(X|Y ) = H(X). [2pts]
• Suppose X and Y are independent. Show that H(X,Y ) = H(X)+H(Y ).

[2pts]

• Prove that the mutual information is symmetric, i.e., I(X,Y ) = I(Y,X) and x_i X,y_i Y [3pts]

6 Bonus for All

• If a random variable X has a Poisson distribution with mean 8, then calculate the expectation E[(X + 2)²] [2 pts]
• A person decides to toss a fair coin repeatedly until he gets a head. He will make at most 3 tosses. Let the random variable Y denote the number of heads. Find the variance of Y. [4 pts]
• Two random variables X and Y are distributed according to

, otherwise

What is the probability P(X+Y 1)? [4 pts]