5/5 - (1 vote)

Neural Networks

For this assignment, you are asked to implement neural networks. You will use this neural network to classify MNIST database of handwritten digits (0-9). The architecture of the neural network you will implement is based on the multi-layer perceptron (MLP, just another term for fully connected feedforward networks we discussed in the lecture), which is shown as following. It is designed for a K-class classification problem.

Let $(x \in R^{D}, y \in {1, 2, \dots, K})$ be a labeled instance, such an MLP performs the following computations:

For a $K$ -class classification problem, one popular loss function for training (i.e., to learn $W^{(1)}$ , $W^{(2)}$ , $b^{(1)}$ , $b^{(2)}$ ) is the cross-entropy loss. Specifically we denote the cross-entropy loss with respect to the training example $(x, y)$ by $l$ :

Note that one should look at $l$ as a function of the parameters of the network, that is, $W^{(1)}, b^{(1)}, W^{(2)}$ and $b^{(2)}$ . For ease of notation, let us define the one-hot (i.e., 1-of- $K$ ) encoding of a class $y$ as

so that

We can then perform error-backpropagation, a way to compute partial derivatives (or gradients) w.r.t the parameters of a neural network, and use gradient-based optimization to learn the parameters.

Submission: You need to submit both neural_networks.py and utils.py.

Q1. Mini batch Stochastic Gradient Descent

First, you need to implement mini-batch stochastic gradient descent which is a gradient-based optimization to learn the parameters of the neural network.You need to realize two alternatives for SGD, one without momentum and one with momentum. We will pass a variable $α$ to indicate which option. When $α \leq 0$ , the parameters are updated just by gradient. When $α > 0$ , the parameters are updated with momentum and $α$ will also represents the discount factor as following:

You can use the formula above to update the weights.Here, $α$ is the discount factor such that $α \in (0, 1)$ . It is given by us and you do not need to adjust it. $η$ is the learning rate. It is also given by us. $υ$ is the velocity update (A.K.A momentum update). $δ_{t}$ is the gradient

TODO 1 You need to complete def miniBatchStochasticGradientDescent(model, momentum, _lambda, _alpha, _learning_rate) in neural_networks.py

Notice that for a complete mini-batch SGD, you will also need to find the best size of mini-batch and number of epochs. In this assignment, we omit this step. Both size of mini-batch and number of epoch has already been given. You do not need to adjust them.

Q2. Linear Layer

Second, you need to implement the linear layer of MLP. In this part, you need to implement 3 python functions in class linear_layer.

In the function def __init__(self, input_D, output_D), you need to initialize W with random values using np.random.normal such that the mean is 0 and standard deviation is 0.1. You also need to initialize gradients to zeroes in the same function.

You can use the above formula as a reference to implement the def forward(self, X) forward pass and def backward(self, X, grad) backward pass in class linear_layer. In backward pass, you only need to return the backward_output. You also need to compute gradients of W and b in backward pass.

TODO 2 You need to complete def __init__(self, input_D, output_D) in class linear_layer of neural_networks.py
TODO 3 You need to complete def forward(self, X) in class linear_layer of neural_networks.py
TODO 4 You need to complete def backward(self, X, grad) in class linear_layer of neural_networks.py

Q3. Activation function tanh

Now, you need to implement the activation function tanh. In this part, you need to implement 2 python functions in class tanh. In def forward(self, X), you need to implement the forward pass and you need to compute the derivative and accordingly implement def backward(self, X, grad), i.e. the backward pass.

You can use the above formula for tanh as a reference.

TODO 5 You need to complete def forward(self, X) in class tanh of neural_networks.py
TODO 6 You need to complete def backward(self, X, grad) in class tanh of neural_networks.py

Q4. Activation function relu

You need to implement another activation function called relu. In this part, you need to implement 2 python functions in class relu. In def forward(self, X), you need to implement the forward pass and you need to compute the derivative and accordingly implement def backward(self, X, grad), i.e. the backward pass.

You can use the above formula for relu as a reference.

TODO 7 You need to complete def forward(self, X) in class relu of neural_networks.py
TODO 8 You need to complete def backward(self, X, grad) in class relu of neural_networks.py

Q5. Dropout (15 points)

To prevent overfitting, we usually add regularization. Dropout is another way of handling overfitting. In this part, you will initially read and understand def forward(self, X, is_train) i.e. the forward pass of class dropout. You will also derive partial derivatives accordingly to implement def backward(self, X, grad) i.e. the backward pass of class dropout.Now we take an intermediate variable $q \in R^{J}$ which is the output from one of the layers. Then we define the forward and the backward passes in dropout as follows.The forward pass obtains the output after dropout.

The backward pass computes the partial derivative of loss with respect to $q$ from the one with respect to the forward pass result, which is $\frac{\partial l}{\partial s}$ .

Note that $p_{j}, j \in {1, \dots, J}$ and $r$ are not be learned so we do not need to compute the derivatives w.r.t. to them. You do not need to find the best $r$ since we have picked it for you. Moreover, $p_{j}, j \in {1, \dots, J}$ are re-sampled every forward pass, and are kept for the corresponding backward pass.

TODO 9 You need to complete def backward(self, X, grad) in class dropout of neural_networks.py

Q6. Connecting the dots

In this part, you will combine the modules written from question Q1 to Q5 by implementing TODO snippets in the def main(main_params, optimization_type="minibatch_sgd") i.e. main function. After implementing forward and backward passes of MLP layers in Q1 to Q5,now in the main function you will call the forward methods and backward methods of every layer in the model in an appropriate order based on the architecture.

TODO 10 You need to complete main(main_params, optimization_type="minibatch_sgd") in neural_networks.py

Google Colab

Google colab is a free online jupyter notebook made available by google for researchers/students wherein the jupyter notebook is backed by GPU.

Jupyter notebook is a python notebook with executable cells accompanied by textual texts for better documentation of the code. They are a good way to document the code.

GPUs are now standard way to compute weights and gradients during training of neural networks, they are faster than CPUs due to their inherent parallelization nature.

We highly suggest trying it for computation of your forward networks and tinkering around with the num_epochs, learning rate to see how the training loss varies. You can look at it here:

https://colab.research.google.com/

Note

Do NOT change the hyperparameters and submit on vocareum even if your changes give better training loss. Changing the hyperparemeters is just for your understanding of how gradient descent is working to optimize the code.

Rate this product

$\begin{aligned} input features : & x \in R^{D} \\ {linear}^{(1)} : & u = W^{(1)} x + b^{(1)}, W^{(1)} \in R^{M \times D} and b^{(1)} \in R^{M} \\ tanh : & h = \frac{2}{1 + e^{- 2 u}} - 1 \\ relu : & h = m a x {0, u} = [\begin{matrix} max {0, u_{1}} \\ ⋮ \\ max {0, u_{M}} \end{matrix}] \\ {linear}^{(2)} : & a = W^{(2)} h + b^{(2)}, W^{(2)} \in R^{K \times M} and b^{(2)} \in R^{K} \\ softmax : & z = [\begin{matrix} \frac{e^{a_{1}}}{\sum_{k} e^{a_{k}}} \\ ⋮ \\ \frac{e^{a_{K}}}{\sum_{k} e^{a_{k}}} \end{matrix}] \\ predicted label : & \hat{y} = {a r g m a x}_{k} z_{k} . \end{aligned}$

$\begin{aligned} l = - \log (z_{y}) = \log (1 + \sum_{k \neq y} e^{a_{k} - a_{y}}) \end{aligned}$

$\begin{aligned} y \in R^{K} and y_{k} = {\begin{cases} 1, if y = k, \\ 0, otherwise . \end{cases} \end{aligned}$

$\begin{aligned} l = - \sum_{k} y_{k} \log z_{k} = - y^{T} [\begin{matrix} \log z_{1} \\ ⋮ \\ \log z_{K} \end{matrix}] = - y^{T} \log z . \end{aligned}$

$\begin{aligned} υ = α υ - η δ_{t} \\ w_{t} = w_{t - 1} + υ \end{aligned}$

$\begin{aligned} forward pass: & u = {linear}^{(1)} .forward (x) = W^{(1)} x + b^{(1)}, \\ where W^{(1)} and b^{(1)} are its parameters. \\ backward pass: & [\frac{\partial l}{\partial x}, \frac{\partial l}{\partial W^{(1)}}, \frac{\partial l}{\partial b^{(1)}}] = {linear}^{(1)} .backward (x, \frac{\partial l}{\partial u}) . \end{aligned}$

$\begin{aligned} tanh : & h = \frac{2}{1 + e^{- 2 u}} - 1 \end{aligned}$

$\begin{aligned} relu : & h = m a x {0, u} = [\begin{matrix} max {0, u_{1}} \\ ⋮ \\ max {0, u_{M}} \end{matrix}] \end{aligned}$

$\begin{aligned} forward pass: & s = dropout .forward (q \in R^{J}) = \frac{1}{1 - r} \times [\begin{matrix} 1 [p_{1} >= r] \times q_{1} \\ ⋮ \\ 1 [p_{J} >= r] \times q_{J} \end{matrix}], \\ where p_{j} is generated randomly from [0, 1), \forall j \in {1, \dots, J}, \\ and r \in [0, 1) is a pre-defined scalar named dropout rate which is given to you . \end{aligned}$

Reviews

There are no reviews yet.

Only logged in customers who have purchased this product may leave a review.

Whatsapp Us

[Solved] CSCI567 Homework3-Neural Networks

Neural Networks

Submission: You need to submit both neural_networks.py and utils.py.

Q1. Mini batch Stochastic Gradient Descent

Q2. Linear Layer

Q3. Activation function tanh

Q4. Activation function relu

Q5. Dropout (15 points)

Q6. Connecting the dots

Google Colab

Note

Reviews

Whatsapp Us

[Solved] CSCI567 Homework3-Neural Networks

Neural Networks </p> <p> <img decoding="async" src="MLP_diagram.png"></p> <p>

Submission: You need to submit both neural_networks.py and utils.py.

Q1. Mini batch Stochastic Gradient Descent

Q2. Linear Layer

Q3. Activation function tanh

Q4. Activation function relu

Q5. Dropout (15 points)

Q6. Connecting the dots

Google Colab

Note

Reviews

Related products

[Solved] CSCI567 Homework 3- Kernels

[Solved] CSCI567 Homework 2-Neural networks

[Solved] CSCI567 Homework 4- Generative models

[Solved] CSCI567 Project 4 Part1-Hidden Markov Models

[Solved] CSCI567 Homework1-K-nearest neighbor (KNN) for binary classification

[Solved] CSCI567 Homework 5- Hidden Markov Models

Neural Networks