uptableandrecurrentneuralnetworksasacontrollerfor.docVIP

Q-learning with Look-up Table and Recurrent Neural Networks as a Controller for the Inverted Pendulum Problem ABSTRACT In reinforcement learning, Q-learning can discover the optimal policy. This paper discusses the inverted pendulum problem using Q-learning as controller agent. Two methods for Q-learning are discussed here. One is look-up table, and the other is approximation with recurrent neural networks. INTRODUCTION Q-learning is an incremental dynamic programming procedure that determines the optimal policy in a step-by-step manner. It is an on-line procedure for learning the optimal policy through experience gained solely on the basis of samples of the form: (1.1) where n denotes discrete time, and each sample consists of a four-tuple described by a trial action an on state in that results in a transition to state jn=in+1 (in denote state at time n) at a cost . And it is highly suited for solving Markovian decision problems without explicit knowledge of the transition probabilities. The requirement of using Q-learning successfully is based on the assumption that the state of the environment is fully observable, which in turn means that the environment is a fully observable Markov chain. However, if the state of the environment is partially observable, for example : the sensor device on the inverted pendulum may be imprecise, special methods are required for discovering the optimal policy. To overcome this problem, a utilization of recurrent neural networks combined with Q-learning as a learning agent had been proposed. According to Bellman’s optimality criterion combined with value iteration algorithm, the small step-size version formula of Q-learning is described by for all (i,a) (1.2) where η is a small learning-rate parameter that lies in the range 0η1. And the other symbols are defined as : “ i ” is sate at time n; “ j “ is state at time n+1; “a” is the action been taken at time n; “” is the transition probability defined as ; “” is