Qlearning with infinite stat space.pdf

$ % Convergence of Q-learning with linear function approximation Francisco S. Melo and M. Isabel Ribeiro Institute for Systems and Robotics [fmelo,mir]@isr.ist.utl.pt European Control Conference, Kos, Greece, July 2007 July 4th, 2007 Slide 1 $ % Outline of the presentation ? Motivation and problem formulation ? Background ? Related work ? Q-learning with LFA ? Some results ? Concluding remarks July 4th, 2007 Slide 2 $ % Motivation ? Markov decision processes provide useful models to address discrete-time stochastic control problems; ? Many powerful methods are available (e.g., TD(λ), Q-learning, SARSA). However... ? Many such methods rely on explicit representations of the state-space; ? Many interesting problems have a state-space unsuited for explicit representation (e.g., infinite); July 4th, 2007 Slide 3 $ % Problem formulation ? In this paper we consider Markov decision processes with infinite state-spaces; ? We propose a modified version of Q-learning that accomodates MDPs with infinite state-space; ? To this end, we make use of linear function approximation to achieve compact representation. July 4th, 2007 Slide 4 $ % Outline of the presentation ? Motivation and problem formulation ? Background ? Related work ? Q-learning with LFA ? Some results ? Concluding remarks July 4th, 2007 Slide 5 $ % Background ? We consider a controlled Markov chain {Xt}, where each r.v. Xt takes values in a set X ; ? X ? Rp is the state-space, assumed compact; ? The transitions of the chain are governed by the transition probabilities Pa(x, U) = P [Xt+1 ∈ U | Xt = x,At = a] , where Pa is a probability kernel ; ? The sequence {At} represents the control process; ? The control At takes values in a finite set A; July 4th, 2007 Slide 6 $ % Background (cont.) ? For every transition from x to y under the action a, a reward r(x, a, y) is issued; ? r is a bounded, real function, known as the reinforcement; Xt Xt+1 At rt July 4th, 2007 Slide 7 $ % Background (cont.) ? The control se