[SOLVED] Cmpt 726: assignment 1 1. linear algebra

a) Let U be a subspace of R
5 defined by:
U =









x1
x2
x3
x4
x5






∈ R
5
: x1 = 3×2 and x4 = 2×5



Find an orthonormal basis {u1, u2, u3}.
b) Prove that the set {u1, u2, u2 + u3} is linearly independent.

2. SVD and Eigendecomposition Rotation Matrices and Practical Applications:
Consider the matrix A =


1 2
2 3
3 1

.
a) Define the matrix B = AA⊤ and compute the matrix B.
b) Singular Values from Eigenvalues:
The Eigendecomposition (EVD) of matrix B is given as:
B = UΛU
⊤

where U is the matrix of eigenvectors, and Λ is the diagonal matrix of eigenvalues.
• Explain how the eigenvalues of B relate to the singular values of A.
• Compute the singular value matrix Σ for A, given that the eigenvalues of B = AA⊤ are 25, 3, and 0.
c) Consider an SVD of a matrix D as follows:
D = UΣV
⊤ =
” √
3
2 −
1
2
1
2
√
3
2
# 
5 0
0 2
”
−
1
2 −
√
3
√
2
3
2 −
1
2
#⊤
A matrix Rθ ∈ R
2×2
is a 2D rotation matrix if it has the following form:
Rθ =

cos θ − sin θ
sin θ cos θ

where θ ∈ R. Geometrically speaking, Rθv rotates v counterclockwise by angle θ, for any v ∈ R
2
, as
shown in Figure 1.

Figure 1: In this case, x = (1, 0) and y = (0, 1) are both rotated by θ =
π
4
.

Show that U and V
⊤ are both rotation matrices and find their corresponding rotational angles θU and θV ⊤ .
3. Taylor Expansion Given #»x =

x1, x2, x3
⊤
, consider a nonlinear function f : R
3 −→ R as follows:
f(
#»x) = 5x
2
1 + 3x
2
2 + 2x
2
3 + 4x1x2 − 2x1x3 + 6x2x3
a) Compute the Gradient and Hessian matrix of f.
b) Find the second order Taylor Expansion at the point #»x0 = [0, 0, 0]⊤.
c) State whether f is convex, concave, or neither convex nor concave. Prove your claim.

4. Convexity For any #»x, #»y ∈ R
n and any t ∈ [0, 1], a function f is said to be convex if it satisfies any of these
conditions:
• f(t
#»x + (1 − t)
#»y ) ≤ tf(
#»x) + (1 − t)f(
#»y )
• If f is differentiable: f(
#»y ) ≥ f(
#»x) + (∇f(
#»x))⊤
(
#»y −
#»x)
• If f is twice differentiable: Hf(
#»x) ⪰ 0
a) Given x ∈ R and only using the definition of convex functions given above, prove that the rectified linear
unit function, ReLU(x) := max(x, 0), is convex.

5. Linear Regresssion – House Price Prediction with Polynomial Features and Ridge Regression
You are working on predicting house prices in a real estate market using a dataset that consists of 500 examples,
each with multiple features: number of rooms, house age, area size, etc. The target variable is the house price. You
decide to apply linear regression and Ridge regression (a regularized form of linear regression) to build predictive
models.

Your goal is to assess the generalization of these models using cross-validation and to experiment with
different polynomial degrees and regularization strengths to improve the model’s performance.
Data Loading: Use the following code to load the Boston Housing dataset from GitHub and initialize the
DataFrame. For the model, only use the variables rm (number of rooms) and lstat (lower status population,
percentage), and the target variable medv (median house value in $1,000s).

import pandas as pd
# Load Boston Housing Data from GitHub
url = “https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv”
df = pd.read_csv(url)
# Features: ’rm’ (number of rooms) and ’lstat’ (lower status population, percentage)
X = df[[’rm’, ’lstat’]]
y = df[’medv’] # Target: ’medv’ (median house value in $1,000s)

Implementation Hints: For managing your dataset, use pandas dataframes and for the models and training
tools, utilize scikit-learn. Plotting should be done with the built-in functions or using matplotlib.
You can refer to the official scikit-learn documentation for functions like train-test split, cross-validation, linear
regression, and Ridge regression.

For coding environments:
• You can work in a Jupyter notebook in Google Colab, and export it as a .py script for final submission.
• If you are already using VS Code, consider using #%% cell separators in your Python script, allowing you
to run parts of your script like Jupyter notebook cells.

Ensure your script includes the code for each task, such as MSE computation or the best parameter choices, and
attach the result outputs to your assignment report in PDF format.

a. Train-Test Split and Cross-Validation for Linear Regression with Polynomial Features
Split the dataset into a training set (80%) and a test set (20%). Use 5-fold cross-validation on the training set
to evaluate the performance of linear regression models with polynomial features. For each polynomial degree
(from 1 to 5), compute the average mean squared error (MSE) over the five folds and report your results.
Hint: Use PolynomialFeatures from scikit-learn to create polynomial features of different degrees. When
using the cross val score function, set the scoring parameter to neg mean squared error.

b. Cross-Validation for Ridge Regression with Polynomial Features
Ridge regression introduces a regularization term controlled by a hyperparameter α. Perform 5-fold crossvalidation on the training set with Ridge regression, using polynomial features (degrees 1 to 5) and different
values of α (e.g., 0.1, 1, 10, 100). Use grid search to find the best α and the optimal degree, and report both the
values that minimize the cross-validation error along with the corresponding average MSE.
Hint: Use GridSearchCV from scikit-learn to automate the search for the best regularization parameter α.

c. Generalization and Test Set Performance
Now that the best polynomial degree and regularization strength have been identified from part (b), train both the
linear regression model and the Ridge regression model on the training set, and evaluate their MSE on the test
set. Compare the performance of the two models, and discuss which model generalizes better to unseen data and
why.
Hint: Use mean squared error from scikit-learn to evaluate the models on the test set.

d. Theoretical Considerations
Explain why the regularization in Ridge regression helps prevent overfitting, especially when using polynomial
features. How does the choice of the regularization parameter α and the polynomial degree influence the model?
What might happen if α is too small or too large, or if the degree is too high?