Problem 0. The answers to these questions should be answerable without referring to external materials. Briefly justify your answers with a few words.

1. Suppose that your estimated model for predicting house prices has a large positive weight on number of bathrooms. Does this imply that if we remove the feature number of bathrooms and refit the model, the new predictions will be strictly worse than before? Why?
2. Compared to L2 norm penalty, explain why L1 norm penalty is more likely to result in a larger number of zeros in the weight vector.
3. In at most one sentence each, state one possible advantage and one possible disadvantage of using the following regularizer:
4. Briefly explain why the estimated true error from k-fold cross validation is biased. Why might k=10 be a reasonable choice for k?
5. True or False: If the step-size for gradient descent (GD) is too large, it may not converge.
6. In your own words, describe why stochastic gradient descent (SGD) works.
7. In at most one sentence each, state one possible advantage of SGD over GD and one possible disadvantage.

Convexity and Norms

Problem 1. A norm k k over Rⁿ is defined by the properties: i) non-negative: kxk 0 for all x Rⁿ with equality if and only if x = 0, ii) absolute scalability: kaxk = |a|kxk for all a R and x Rⁿ, iii) triangle inequality: kx + yk kxk + kyk for all x,y Rⁿ.

1. Show that is a norm. (Hint: You are allowed to use the Cauchy-Schwarz inequality.)
2. Show that is not a norm. (Hint: it suffices to find two points in n = 2 dimensions such that the triangle inequality does not hold.)

Context: norms are often used in regularization to encourage specific behaviors of solutions. If we define

then one can show that kxk_p is a norm for all p 1. The important cases of p = 2 and

p = 1 correspond to the penalty for ridge regression and the lasso, respectively.

Problem 2. A set A Rⁿ is convex if x + (1 )y A for all x,y A and [0,1].

For each of the grey-shaded sets above (I-III), state whether each one is convex, or state why it is not convex using any of the points a,b,c,d in your answer.

Problem 3. [4 points] We say a function f : R^d R is convex on a set A if f(x+(1)y) f(x)+(1)f(y) for all x,y A and [0,1].

For each of the grey-colored functions below (I-III), state whether each one is convex on the given interval or state why not with a counterexample using any of the points a,b,c,d in your answer.

1. Function in panel I on [a,c]
2. Function in panel II on [a,d]
3. Function in panel II on [a,b]
4. Function in panel III on [a,c]

Problem 4. [5 points] Let A Rⁿⁿ be a symmetric positive definite matrix. Show that the function f(x) = (x^TAx)^1/2 is convex. Hint: First show that f(x) is a norm, then show that all norms are convex functions. To show that f(x) is a norm, you may use the spectral theorem. The spectral theorem states that any symmetric real matrix A can be written in the form A = V V ^T, where V, Rⁿⁿ, is a diagonal matrix, and V ^TV = I (i.e. V is an orthonormal matrix). Show that f(x) = k^1/2V ^Txk₂ and invoke problem 1a above.

What does the notation ^1/2 mean and how do we know it exists?

Lasso on a real dataset

Given > 0 and data (x₁,y₁),,(x_n,y_n), the Lasso is the problem of solving

arg min

is a regularization tuning parameter. For the programming part of this homework, you are required to implement the coordinate descent method of Algorithm 1 that can solve the Lasso problem.

You may use common computing packages (such as NumPy or SciPy), but do not use an existing Lasso solver

(e.g., of scikit-learn).

Before you get started, here are some hints that you may find helpful:

• For-loops can be slow whereas vector/matrix computation in Numpy is very optimized; exploit this as much as possible.
• The pseudocode provided has many opportunities to speed up computation by precomputing quantities like a_k before the for loop. These small changes can speed things up considerably.
• As a sanity check, ensure the objective value is nonincreasing with each step.
• It is up to you to decide on a suitable stopping condition. A common criteria is to stop when no element of w changes by more than some small during an iteration. If you need your algorithm to run faster, an easy place to start is to loosen this condition.
• You will need to solve the Lasso on the same dataset for many values of . This is called a regularization path. One way to do this efficiently is to start at a large , and then for each consecutive solution, initialize the algorithm with the previous solution, decreasing by a constant ratio (e.g., by a factor of 2) until finished.
• The smallest value of for which the solution w is entirely zero is given by b

This is helpful for choosing the first in a regularization path.

Problem 5. We will first try out your solver with some synthetic data. A benefit of the Lasso is that if we believe many features are irrelevant for predicting y, the Lasso can be used to enforce a sparse solution, effectively differentiating between the relevant and irrelevant features. Suppose that x R^d,y R,k < d, and pairs of data (x_i,y_i) for i = 1,,n are generated independently according to the model where

) is some Gaussian noise (in the model above b = 0). Note that since k < d and w_j = 0 for

j > k, the features k + 1 through d are unnecessary (and potentially even harmful) for predicting y. With this model in mind, let n = 500,d = 1000,k = 100, and = 1. Generate some data by choosing x_i R^d, where each component is drawn from a N(0,1) distribution and y_i generated as specified above.

Algorithm 1: Coordinate Descent Algorithm for Lasso

while not converged do for k {1,2,d} do

n

1. [10 points] With your synthetic data, solve multiple Lasso problems on a regularization path, starting at _max where 0 features are selected and decreasing by a constant ratio (e.g., 1.5) until nearly all the features are chosen. In plot 1, plot the number of non-zeros as a function of on the x-axis (Tip: use xscale(log)).
2. [10 points] For each value of tried, record values for false discovery rate (FDR) (number of incorrect nonzeros in w ^[1] /total number of nonzeros in w) and true positive rate (TPR) (number of correct nonzeros b b in w/k). In plot 2, plot these values with the x-axis as FDR, and the y-axis as TPR and note that in an b

ideal situation we would have an (FDR,TPR) pair in the upper left corner, but that can always trivially achieve (0,0) and (

1. [5 points] Comment on the effect of in these two plots.

Problem 6. Well now put the Lasso to work on some real data. Download the training data set crimetrain.txt and the test data set crime-test.txt from the website under Homework 2. Store your data in your working directory and read in the files with:

import pandas as pd df_train = pd.read_table(crime-train.txt) df_test = pd.read_table(crime-test.txt)

This stores the data as Pandas DataFrame objects. DataFrames are similar to Numpy arrays but more flexible; unlike Numpy arrays, they store row and column indices along with the values of the data. Each column of a DataFrame can also, in principle, store data of a different type. For this assignment, however, all data are floats. Here are a few commands that will get you working with Pandas for this assignment:

df.drop(foo, axis = 1) # Return all columns except foo. df.values # Return the values as a Numpy array. df[foo].values # Grab column foo and convert to Numpy array. df.iloc[:3,:3] # Use numerical indices (like Numpy) to get 3 rows and cols.

The data consist of local crime statistics for 1,994 US communities. The response y is the rate of violent crimes reported per capita in a community. The name of the response variable is ViolentCrimesPerPop, and it is held in the first column of df train and df test. There are 95 features. These features include many variables. Some features are the consequence of complex political processes, such as the size of the police force. Others are demographic characteristics of the community, including self-reported statistics about race, age, education, and employment drawn from Census reports.

The goals of this problem are twofold: first, to encourage you to think deeply about models you might train and how they might be misused; and second, to see how Lasso encourages sparsity of linear models in settings where the feature set is very large relative to the number of training examples. We emphasize that training a model on this dataset can suggest a degree of correlation between a communitys demographics and the rate at which a community experiences and reports violent crime. We strongly encourage students to consider why these correlations may or may not hold more generally, whether correlations might result from a common cause, and what issues can result in misinterpreting what a model can explain.

We have split the dataset into a training and test set with 1,595 and 399 entries, respectively². Wed like to use this training set to fit a linear model to predict the crime rate in new communities and evaluate model performance on the test set. As there are a considerable number of input variables and fairly few training datapoints, overfitting is a serious issue. In order to avoid this, use the coordinate descent LASSO algorithm you just implemented in the previous problem.

1. [4 points] Begin by reading the documentation for the original version of this dataset: http://archive. uci.edu/ml/datasets/communities+and+crime. Report 3 features included in this dataset for which historical policy choices in the US would lead to variability in these features. As an example, the number of police in a community is often the consequence of decisions made by governing bodies, elections, and amount of tax revenue available to decisionmakers.
2. [4 points] Before you train a model for this prediction task, describe 3 features in the dataset which might, if found to have nonzero weight in model, be interpreted as reasons for higher levels of violent crime, but which might actually be a result rather than (or in addition to being) the cause of this violence.

Now, we will run the LASSO solver with = _max defined above. For the initial weights, just use 0. Then, cut down by a factor of 2 and run again, but this time pass in the values of w from your = _max solution as your initial weights. This is faster than initializing with 0 weights each time. Continue the process of cutting by a factor of 2 until the smallest value of is less than 0.01. For all plots use a log-scale for the axis (Tip: use plt.xscale(log)).

1. [4 points] Plot the number of nonzeros of each solution versus .
2. [4 points] Plot the regularization paths (in one plot) for the coefficients for input variables agePct12t29, pctWSocSec, pctUrban, agePct65up, and householdsize.
3. [4 points] On one plot, plot the squared error on the training and test data versus .
4. [4 points] Sometimes a larger value of performs nearly as well as a smaller value, but a larger value will select fewer variables and perhaps be more interpretable. Inspect the weights (on features) for = 30. Which feature variable had the largest (most positive) Lasso coefficient? What about the most negative? Discuss briefly.
5. [4 points] Suppose there was a large negative weight on agePct65up and upon seeing this result, a politician suggests policies that encourage people over the age of 65 to move to high crime areas in an effort to reduce crime. What is the (statistical) flaw in this line of reasoning? (Hint: fire trucks are often seen around burning buildings, do fire trucks cause fire?)

Logistic Regression

Binary Logistic Regression

Problem 7. Let us again consider the MNIST dataset, but now just binary classification, specifically, recognizing if a digit is a 2 or 7. Here, let Y = 1 for all the 7s digits in the dataset, and use Y = 1 for 2. We will use regularized logistic regression. Given a binary classification dataset for x_i R^d and y_i {1,1} we showed in class that the regularized negative log likelihood objective function can be written as

Note that the offset term b is not regularized. For all experiments, use = 10¹. Let.

1. [8 points] Derive the gradients _wJ(w,b), _bJ(w,b) and give your answers in terms of _i(w,b) (your answers should not contain exponentials).
2. [8 points] Implement gradient descent with an initial iterate of all zeros. Try several values of step sizes to find one that appears to make convergence on the training set as fast as possible. Run until you feel you are near to convergence.
   • For both the training set and the test, plot J(w,b) as a function of the iteration number (and show both curves on the same plot).
   • For both the training set and the test, classify the points according to the rule sign(b + x^T_i w) and plot the misclassification error as a function of the iteration number (and show both curves on the same plot).

Note that you are only optimizing on the training set. The J(w,b) and misclassification error plots should be on separate plots.

1. [7 points] Repeat (b) using stochastic gradient descent with a batch size of 1. Note, the expected gradient with respect to the random selection should be equal to the gradient found in part (a). Take careful note of how to scale the regularizer.
2. [7 points] Repeat (b) using stochastic gradient descent with batch size of 100. That is, instead of approximating the gradient with a single example, use 100. Note, the expected gradient with respect to the random selection should be equal to the gradient found in part (a).

[1] For each is an incorrect nonzero if and only if = 0 while w_j = 0 ²The features have been standardized to have mean 0 and variance 1.