reinforcement learning.pptVIP

* * * * * * * * * * * * * * * * * * * * * * maybe show the pendulum? * * Temporal Difference Learning combines ideas from MC and DP like MC: learn directly from experience (don’t need a model) like DP: bootstrap works for continuous tasks, usually faster then MC constant-alpha MC: have to wait until the end of episode to update simplest TD update after every step, based on the successor target MC vs. TD observed the following 8 episodes: A – 0, B – 0 B – 1 B – 1 B - 1 B – 1 B – 1 B – 1 B – 0 MC and TD agree on V(B) = 3/4 MC: V(A) = 0 converges to values that minimize the error on training data TD: V(A) = 3/4 converges to ML estimateof the Markov process A B r = 0 100% r = 1 75% r = 0 25% Sarsa again, need Q(s,a), not just V(s) control start with a random policy update Q and ? after each step again, need ?-soft policies st st+1 at st+2 at+1 at+2 rt rt+1 Q-learning previous algorithms: on-policy algorithms start with a random policy, iteratively improve converge to optimal Q-learning: off-policy use any policy to estimate Q Q directly approximates Q* (Bellman optimality eqn) independent of the policy being followed only requirement: keep updating each (s,a) pair Sarsa Outline examples defining a Markov Decision Process solving an MDP using Dynamic Programming Reinforcement Learning Monte Carlo methods Temporal-Difference learning miscellaneous state representation function approximation rewards State representation pole-balancing move car left/right to keep the pole balanced state representation position and velocity of car angle and angular velocity of pole what about Markov property? would need more info noise in sensors, temperature, bending of pole solution coarse discretization of 4 state variables left, center, right totally non-Markov, but still works Function approximation until now, state space small and discrete represent Vt as a parameterized function linear regression, decision tree, neural net, … linear regression: update