Qlearning_readme分析和总结分析和总结.docx

Q-learning with Look-up Table and Recurrent Neural Networks as a Controller for the Inverted Pendulum Problem ABSTRACT In reinforcement learning, Q-learning can discover the optimal policy. This paper discusses the inverted pendulum problem using Q-learning as controller agent. Two methods for Q-learning are discussed here. One is look-up table, and the other is approximation with recurrent neural networks. INTRODUCTION Q-learning is an incremental dynamic programming procedure that determines the optimal policy in a step-by-step manner. It is an on-line procedure for learning the optimal policy through experience gained solely on the basis of samples of the form: sn ? (in, an, jn, gn) (1.1) n n n+1where n denotes discrete time, and each sample s n consists of a four-tuple described by a trial action a on state in that results in a transition to state j = n n n+1 (i denote state at time n) at a cost g ? g(i , a , j ) . And it is highly suited for n n n n n solving Markovian decision problems without explicit knowledge of the transition probabilities. The requirement of using Q-learning successfully is based on the assumption that the state of the environment is fully observable, which in turn means that the environment is a fully observable Markov chain. However, if the state of the environment is partially observable, for example : the sensor device on the inverted pendulum may be imprecise, special methods are required for discovering the optimal policy. To overcome this problem, a utilization of recurrent neural networks combined with Q-learning as a learning agent had been proposed. According to Bellman’s optimality criterion combined with value iteration algorithm, the small step-size version formula of Q-learning is described by Q(i, a) :? (1 ? ?)Q(i, a) ? ??N ? ? p ? g(i, a, j) ? ? maxQ( j,b)? for all (i,a) (1.2) j?1 ij ? b?Aj ? where η is a small learning-rate parameter that lies in the range 0 η1. And the other symbols are defined “asi” :is sate at ti“me