1. Python and Numpy Warm-up Exercises This part of the homework is intended to help you review your linear algebra and learn (or refresh your understanding of) how to implement linear algebraic operations in Python using numpy (to which we refer in the code below as np). For each of the problems below, write a method (e.g., problem1) that returns the answer for the corresponding problem. Put all your methods in one file called homework1 WPIUSERNAME.py

(e.g., homework1 jrwhitehill.py). See the starter file homework1 template.py.

In all problems, you may assume that the the dimensions of the matrices and/or vectors are compatible for the requested mathematical operations. Note: Throughout the assignment, please use np.array, not np.matrix.

• Given matrices A and B, compute and return an expression for A + B. [ 1 pts ]

Answer (freebie!): While it is completely valid to use np.add(A, B), this is unnecessarily verbose; you really should make use of the syntactic sugar provided by Pythons/numpys operator overloading and just write: A + B. Similarly, you should use the more compact (and arguably more elegant) notation for the rest of the questions as well.

• Given matrices A, B, and C, compute and return AB C (i.e., right-multiply matrix A by matrix B, and then subtract C). Use dot or dot. [ 2 pts ]
• Given matrices A, B, and C, return A, where represents the element-wise (Hadamard) product and > represents matrix transpose. In numpy, the element-wise product is obtained simply with *. [ 2 pts ]
• Given column vectors x and y, compute the inner product of x and y (i.e., x^>y). [ 2 pts ]
• Given matrix A, return a matrix with the same dimensions as A but that contains all zeros. Use zeros. [ 2 pts ]
• Given square matrix A and column vector x, use linalg.solve to compute A¹x. Do not explicitly calculate the matrix inverse itself (e.g., np.linalg.inv, A ** -1) because this is numerically unstable (and yes, it can sometimes make a big difference!). [ 2 pts ]
• Given square matrix A and row vector x, use linalg.solve to compute xA¹. Hint: AB = (B^>A^>)^>. [ 3 pts ]
• Given square matrix A and (scalar) , compute A + I, where I is the identity matrix with the same dimensions as A. Use eye. [ 2 pts ]
• Given matrix A and integers i,j, return the jth column of the ith row of A, i.e., A_ij. [ 1 pts ]
• Given matrix A and integer i, return the sum of all the entries in the ith row whose column index is even, i.e., ^P_{j:j is even} A_ij. Do not use a loop, which in Python can be very slow. Instead use the sum function. [ 2 pts ]
• Given matrix A and scalars c,d, compute the arithmetic mean over all entries of A that are between c and d (inclusive). In other words, if S = {(i,j) : c A_ij d}, then compute . Use nonzero along with np.mean. [ 3 pts ]
• Given an (nn) matrix A and integer k, return an (nk) matrix containing the right-eigenvectors of A corresponding to the k largest eigenvalues of A. Use linalg.eig. [ 3 pts ]
• Given a n-dimensional column vector x, an integer k, and positive scalars m,s, return an (nk) matrix, each of whose columns is a sample from multidimensional Gaussian distribution N(x + mz,sI), where z is an n-dimensional column vector containing all ones and I is the identity matrix. Use either random.multivariate normal or np.random.randn. [ 3 pts ]
• Given a matrix A with n rows, return a matrix that results from randomly permuting the rows (but not the columns) in A. [ 2 pts]

1

2. Linear Regression via Analytical Solution

(a) Train an age regressor that analyzes a (48 48 = 2304)-pixel grayscale face image and outputs a real number y that estimates how old the person is (in years). Your regressor should be implemented using linear regression. The training and testing data are available here:

• https://s3.amazonaws.com/jrwprojects/age_regression_Xtr.npy
• https://s3.amazonaws.com/jrwprojects/age_regression_ytr.npy
• https://s3.amazonaws.com/jrwprojects/age_regression_Xte.npy
• https://s3.amazonaws.com/jrwprojects/age_regression_yte.npy

To get started, see the train age regressor function in homework1 template.py.

Note: you must complete this problem using only linear algebraic operations in numpy you may not use any off-the-shelf linear regression software, as that would defeat the purpose.

Compute the optimal weights w = (w₁,,w₂₃₀₄) for a linear regression model by deriving the expression for the gradient of the cost function w.r.t. w and b, setting it to 0, and then solving. Do not solve using gradient descent. The cost function is

n

i=1

where y = g(x;w) = x^>w and n is the number of examples in the training set

D_tr = {(x⁽¹⁾,y⁽¹⁾),,(x⁽ⁿ⁾,y⁽ⁿ⁾)}, each x⁽ⁱ⁾ R²³⁰⁴ and each y⁽ⁱ⁾ R. Note that this simple regression model does not include a bias term (which we will add later); correspondingly, please do not include one in your own implementation. After optimizing w only on the training set, compute and report the cost f_MSE on the training set D_tr and (separately) on the testing set D_te. [ 12 pts ]

3. Proofs For the proofs, please create a PDF (which you can generate using LaTex, or, if you prefer, a scanned copy of your legible handwriting).

• Prove that a

for any two n-dimensional column vectors x,a. Hint: differentiate w.r.t. each element of x, and then gather the partial derivatives into a column vector. [ 4 pts ]

• Prove thatx

for any n-dimensional column vector x and any n n matrix A. [ 8 pts ]

• Based on the theorem above, prove that

Ax

for any n-dimensional column vector x and any symmetric n n matrix A. [ 2 pts ]

• Based on the theorems above, prove that

_xh(Ax + b)^> (Ax + b)ⁱ = 2A^> (Ax + b)

for any n-dimensional column vector x, any symmetric n n matrix A, and any constant ndimensional column vector b. [ 4 pts ]

Submission: Create a Zip file containing both your Python and PDF files, and then submit on Canvas.

Teamwork: You may complete this homework assignment either individually or in teams up to 2 people.

2