Reinforcement LearningApproximate Planning.pptVIP

Reinforcement Learning:Approximate Planning Csaba Szepesvári University of Alberta Kioloa, MLSS’08 Slides: http://www.cs.ualberta.ca/~szepesva/MLSS08/ Planning Problem The MDP .. is given (p,r can be queried) .. can be sampled from at any state Trajectories “Simulation Optimization” Goal: Find an optimal policy Constraints: Computational efficiency Polynomial complexity O(1) ′ real-time decisions Sample efficiency ~ computational efficiency Methods for planning Exact solutions (DP) Approximate solutions Rollouts (′ search) Sparse lookahead trees, UCT Approximate value functions RDM, FVI, LP Policy search Policy gradient (Likelihood Ratio Method), Pegasus [Ng Jordan ’00] Hybrid Actor-critic Bellman’s Curse of Dimensionality The state space in many problems is.. Continuous High-dimensional “Curse of Dimensionality” (Bellman, 57) Running time of algorithms scales exponentially with the dimension of the state space. Transition probabilities Kernel: P(dy|x,a) Density: p(y|x,a) ?!! e.g. p(y|x,a) ~ exp( -||y-f(x,a)||2/(2?2)) A Lower Bound Theorem (Chow, Tsitsiklis ’89) Markovian Decision Problems d dimensional state space Bounded transition probabilities, rewards Lipschitz-continuous transition probabilities and rewards ? Any algorithm computing an 2-approximation of the optimal value function needs ?(2-d) values of p and r. What’s next then?? Monte-Carlo Search Methods Problem: Can generate trajectories from an initial state Find a good action at the initial state Sparse lookahead trees [Kearns et al., ’02]: Sparse lookahead trees Effective horizon:  H(2) = Kr/(2(1-°)) Size of the tree: S = c |A|H (2) (unavoidable) Good news: S is independent of d! ..but is exponential in H(2) Still attractive: Generic, easy to implement Would you use it? Idea.. Need to propagate values from good branches as early as possible Why sample suboptimal actions at all? Breadth-first  ? Depth-first! Bandit algorithms ? Upper Confidence Bounds UCB [Auer et al. ’02] Bandit with