[SOLVED] Cs405 homework 5

Question 1
Consider a regression problem involving multiple target variables in which it is assumed that the
distribution of the targets, conditioned on the input vector x, is a Gaussian of the form
where is the output of a neural network with input vector and wight vector , and is
the covariance of the assumed Gaussian noise on the targets.
(a) Given a set of independent observations of and , write down the error function that must be
minimized in order to find the maximum likelihood solution for , if we assume that is fixed and
known.
(b) Now assume that is also to be determined from the data, and write down an expression for the
maximum likelihood solution for . (Note: The optimizations of and are now coupled.)
Question 2
The error function for binary classification problems was derived for a network having a logisticsigmoid output activation function, so that , and data having target values
. Derive the corresponding error function if we consider a network having an output
and target values for class and for class . What would be the
appropriate choice of output unit activation function?
Hint. The error function is given by:
Question 3
Verify the following results for the conditional mean and variance of the mixture density network
model.
(a)
(b)
Question 4
Can you represent the following boolean function with a single logistic threshold unit (i.e., a single unit
from a neural network)? If yes, show the weights. If not, explain why not in 1-2 sentences.
A B f(A,B)
1 1 0
0 0 0
1 0 1
0 1 0
Question 5
Below is a diagram of a small convolutional neural network that converts a 13×13 image into 4 output
values. The network has the following layers/operations from input to output: convolution with 3
filters, max pooling, ReLU, and finally a fully-connected layer. For this network we will not be using any
bias/offset parameters (b). Please answer the following questions about this network.
(a) How many weights in the convolutional layer do we need to learn?
(b) How many ReLU operations are performed on the forward pass?
(c) How many weights do we need to learn for the entire network?
(d) True or false: A fully-connected neural network with the same size layers as the above network
can represent any classifier?
(e) What is the disadvantage of a fully-connected neural network compared to a convolutional neural
network with the same size layers?
Question 6
The neural networks shown in class used logistic units: that is, for a given unit , if is the vector of
activations of units that send their output to , and is the weight vector corresponding to these
outputs, then the activation of will be . However, activation functions could be
anything. In this exercise we will explore some others. Consider the following neural network,
consisting of two input units, a single hidden layer containing two units, and one output unit:
(a) Say that the network is using linear units: that is, defining and and as above, the output of a
unit is for some fixed constant . Let the weight values be fixed. Re-design the neural
network to compute the same function without using any hidden units. Express the new weights in
terms of the old weights and the constant .
(b) Is it always possible to express a neural network made up of only linear units without a hidden
layer? Give a one-sentence justification.
(c) Another common activation function is a theshold, where the activation is where is 1
if and 0 otherwise. Let the hidden units use sigmoid activation functions and let the output unit
use a threshold activation function. Find weights which cause this network to compute the XOR of
and for binary-valued and . Keep in mind that there is no bias term for these units.