[Solved] CS584-Assignment 3-Recurrent Neural Networks

1. Language Modeling (100 points) In this homework, you need to implement a Recurrent Neural Network for language modeling with word embeddings on the Penn Treebank corpus. Please follow the steps as below:
   • (10pts) Preprocess the train and validation data, build the vocabulary, tokenize, etc.
   • (10pts) Initialize parameters for the model.
   • (20pts) Implement the forward pass for the model. Use an embeddings layer as the first layer of your network (i.e. nn.embedding lookup). Use a recurrent neural network cell (GRU or LSTM) as the next layer.
   • (20pts) Calculate the loss of the model (sequence cross-entropy loss is suggested)

i.e. tf.contrib.seq2seq.sequence loss.

• (10pts) Calcuate the models perplexity (perplexity is number of predictionsexp(total loss) ).
• (10pts) Set up the training step: use a learning rate of 1e 3 and an adam optimizer.
• (20pts) Train you model and report the perplexity (set window size to be 20, batch size to be greater than 1, e.g. 50).

Checking on your model: To check how your model is doing (and to actually see the cool thing your model is doing), we recommend, but are not requiring, looking at the sentences your model is generating. To do so, you can append together the argmax of the logit predictions after each sess.run() to create a generated sentence representation. You can then convert each word index to its corresponding English word from your vocab to create readable sentences.