[SOLVED] CS 499/599 HW1

CS 499/599 HW1
Homework 1: Build Your Own Models
Important Dates
[Hand-out] 1.03.2022 (Mon.) 12:00 pm PST

Copyright By Assignmentchef assignmentchef

[Deadline] 1.17.2022 (Mon.) 11:59 am PST
Homework Overview
The learning objective of this homework is for you to create a codebase to train and evaluate various deep neural network (DNN) models. You also need to analyze the impact of various factors (that you can control during training) on the final DNN models. You will use this codebase and the trained models to complete the homework assignments (HW 2, 3, and 4) throughout the term.

Initial Setup
To begin with, you can choose any deep learning framework that youre already familiar with (e.g., PyTorch, TensorFlow, or ObJAX). If you are not familiar with any of these frameworks, you can start with PyTorch or TensorFlow (> v2.0). These are popular choices, and you can find many tutorials [link] or example code [link] from the Internet.

[Note] I do NOT recommend copying and pasting the sample code found from the Internet. It will be an easy solution for this homework. However, later on, you may have difficulty understanding the attacks and defenses. For example, some attacks (and defenses) require you to know how the deep learning framework computes gradients and how you can manipulate (or control) them.

Datasets and DNN Models

We will limit our scope to two popular image classification datasets: MNIST [link] and CIFAR-10 [link]. Most deep learning frameworks support those datasets by default. We will also use three DNNs: LeNet [link], VGG16 [link] and ResNet18 [link]. I added the links to the original papers.

Recommended Code Structure

models: a dir containing your model definitions.
reports: a dir where you will include your write-up.
datasets.py: a Python script containing functions for loading datasets.
train.py: a Python script for training a model.
train.sh: a bash-shell script for training multiple models.
valid.py: a Python script for evaluating a pre-trained model.

Note that this is an example code structure; you can find many nice examples from the Internet [example].
Task I: Train and Evaluate Your Models
The first task is simple; train 6 DNN models. You will train 3 DNNs (LeNet, VGG16, and ResNet18) on 2 datasets (MNIST and CIFAR-10). You need to measure your models performance with two metrics: classification accuracy and loss. You can compute them on both the training and testing data.

Please compute those metrics in every 5 training iterations (epochs). Draw 2 plots for each model training: { epochs } vs. { training accuracy & testing accuracy } and { epochs } vs. { training loss & testing loss } [see this example plots].
Task II: Analyze the Impact of Your Training Techniques on Models
Now, lets turn our attention to how you train those 6 DNN models. You probably made various choices to train those models; for example, you may use cross-entropy to compute the loss of a model. Depending on how you train your models, they have slightly different properties. In this task, we will analyze the impact of various choices that you can make for training a model on its performance. Since this task may require training multiple DNN models, which takes some time, lets reduce our scope to two cases: (i) training LeNet on MNIST and (ii) ResNet18 on CIFAR10.

You can control the following things:
Data augmentations: transform the inputs of a neural network, e.g., cropping, resizing, flipping, shifting,
Model architectures: add additional layers to a neural network, e.g., adding Dropout before the classification head,
Optimization algorithm (or loss functions): choosing a different optimizer, e.g., SGD or Adam, or a different loss function.
Training hyper-parameters: batch-size, learning rate, total number of training iterations (epochs),
Lets compare models trained in the following 5 scenarios:
Data augmentation: Rotation: train your models with and w/o rotations and compare the plots.
Data augmentation: Horizontal flip: train your models with and w/o random horizontal flips and compare the plots.
Optimization: SGD/Adam: train your models with SGD or Adam and compare the plots.
Hyper-parameters: batch-size: train your models with two different batch-sizes and compare the plots.
Hyper-parameters: learning rate: train your models with two different learning rates and compare the plots.
You may (or may not) find a significant difference between the two models. Explain your intuitions on why you observe (or do not observe) them.
Submission Instructions
Use Canvas to submit your homework. You need to make a single compressed file (.tar.gz) that contains your code and a write-up as a PDF file. Put your write-up under the reports folder. Your PDF write-up should contain the following things:
Your experimental setup: specify your training configurations such as your hyper-parameter choices.
Your 12 plots: 2 plots for each model, and you have 6 models.
Your analysis: write-down a summary (the acc. and loss of the models); provide 2-3 sentences explaining why you see the results.
Your 20 plots: 2 plots for each model, and you have 2 models for each of the five scenarios.
Your analysis: Provide 2-3 sentences for each scenarios explaining why you observe the result.

CS: assignmentchef QQ: 1823890830 Email: [email protected]