1 Linear Classifier with a Margin

Show that, regardless of the dimensionality of the feature vectors, a data set that has just two data points, one from each class, is sufficient to determine the location of the maximum-margin hyperplane. Hint #1: Consider a data set of two data points, x₁ C₁ (y₁ = +1) and x₂ C₂

(y₂ = 1) and set up the minimization problem (for computing the hyperplane) with appropriate constraints on w^Tx₁ + b and w^Tx₂ + b and solve it. Hint #2: This can be formed as a constrained optimization problem.

arg min wR^p

Subject to: (some constraint)

What is w? b? Hint: What are the constraints? How did we solve the constrained optimization problem in Fishers linear discriminate (see Linear Models Lecture Notes or constrained optimization from Calculus)?

2 Linear Regression with Regularization

In class we derived and discussed linear regression in detail. Find the result of minimize the loss of sum of the squared errors; however, add in a penalty for an L₂ penalty on the weights. More

formally,

argmin w

How does this change the solution to the original linear regression solution? What is the impact of adding in this penalty?

Write your own implementation of logistic regression and implement your model on either realworld (see Github data sets: https://github.com/gditzler/UA-ECE-523-Sp2018/tree/master/ data), or synthetic data. If you simply use Scikit-learns implementation of the logistic regression classifier, then youll receive zero points. A full 10/10 will be awarded to those that implement logistic regression using the optimization of cross-entropy using stochastic gradient descent.

3 Density Estimation

The ECE523 Lecture notes has a function for generating a checkerboard data set. Generate checkerboard data from two classes and use any density estimate technique we discussed to classify new data using

where is your estimate of the posterior given you estimates of using a density estimator and pb_Y (y) using a maximum likelihood estimator. You should plot p_bX_|Y (x|y) using a pseudo color plot (see https://goo.gl/2SDJPL). Note that you must model pb_X(x), p_bY (y), and pb_X|Y (x|y). Note that p_bX(x) can be calculated using the Law of Total Probability.

4 Conceptual

The Bayes decision rule describes the approach we take to choosing a class for a data point x. This can be achieved modeling P(|x) or P(x|)P()/P(x). Compare and contrast these two approaches to modeling and discuss the advantages and disadvantages. For the latter model, why might knowing P(x) be useful?