《1992 Technical Note Q-learning》.pdf

Machine Learning, 8, 279-292 (1992) © 1992 Kluwer Academic Publishers, Boston. Manufactured in The Netherlands. Technical Note Q,-Learning CHRISTOPHER J.C.H. WATKINS 25b Framfield Road, Highbury, London N5 IUU, England PETER DAYAN Centre for Cognitive Science, University of Edinburgh, 2 Buccleuch Place, Edinburgh EH8 9EH, Scotland Abstract. Q-learning (Watkins, 1989) is a simple way for agents to learn how to act optimally in controlled Markovian domains. It amounts to an incremental method for dynamic programming which imposes limited computational demands. It works by successively improving its evaluations of the quality of particular actions at particular states. This paper presents and proves in detail a convergence theorem for Q,-learning based on that outlined in Watkins (1989). We show that Q-learning converges to the optimum action-values with probability 1 so long as all actions are repeatedly sampled in all states and the action-values are represented discretely. We also sketch extensions to the cases of non-discounted, but absorbing, Markov environments, and where many Q values can be changed each iteration, rather than just one. Keywords. Q-learning, reinforcement learning, temporal differences, asynchronous dynamic programming 1. Introduction Q-learning (Watkins, 1989) is a form of model-free reinforcement learning. It can also be viewed as a method of asynchronous dynamic programming (DP). It provides agents with the capability of learning to act optimally in Markovian domains by experiencing the con- sequences of actions, without requiring them to build maps of the domains. Learning proceeds similarly to Suttons (1984; 1988) method of temporal differences (TD): an agent tries an action at a particular state, and evaluates its consequences in ter