Assignment 4

1 Introduction

The goal of this assignment is to do experiment with deep Q network (DQN), which combines the advantage of Q-learning and the neural network. In classical Q-learning methods, the action value function Q is intractable with the increase of state space and action space. DQN introduces the success of deep learning and has achieved a super-human level of play in atari games. Your goal is to implement the DQN algorithm and its improved algorithm and play with them in some classical RL control scenarios.

2 Deep Q-learning

Algorithm 1: deep Q-learning with experience replay. Initialize replay memory D to capacity N

Initialize action-value function Q with random weights h

Initialize target action-value function Q^ with weights h²5h

For episode 5 1, M do

Initialize sequence s₁~f gx₁ and preprocessed sequence w₁~w s₁

For t5 1,T do

With probability e select a random action a_t otherwise select at~argmaxaQw st ,a;h

Execute action at in emulator and observe reward rt and image xt11

Set stz1~st,at,xtz1 and preprocess w_tz1~wstz1

Store transition w_t,a_t,r_{t,wtz1 in} D

Sample random minibatch of transitions w_j,a_j,r_j,w_jz1 from D

r_j if episode terminates at step jz1 Setyj~rjzc maxa0 Q^ wjz1,a0;h{ otherwise

2

Perform a gradient descent step on y_j{Q w_j,a_j;h with respect to the network parameters h

Every C steps reset Q^~Q

End For

End For

Figure 1: Deep Q-learning with experience replay

You can refer to the original paper for the details of the DQN. Human-level control through deep reinforcement learning. Nature 518.7540 (2015): 529.

3 Experiment Description

• Programming language: python3
• You should compare the performance of DQN and one kind of improved DQN and test them in a classical RL control environmentMountainCar.

OPENAI gym provides this environment, which is implemented with python (https://gym.openai.com/envs/MountainCar-v0/). Whats more, gym also provides other more complex environment like atari games and mujoco.

Since the state is abstracted into cars position, convolutional layer is not necessary in our experiment. You can get started with OPENAI gym refer to this link (https://gym.openai.com/docs/). Note that it is suggested to implement your neural network on the Tensorflow or Pytorch.