Lecture5Model-FreeControl.PDFVIP

Lecture 5: Model-Free Control Lecture 5: Model-Free Control David Silver Lecture 5: Model-Free Control Outline 1 Introduction 2 On-Policy Monte-Carlo Control 3 On-Policy Temporal-Di?erence Learning 4 O?-Policy Learning 5 Summary Lecture 5: Model-Free Control Introduction Model-Free Reinforcement Learning Last lecture: Model-free prediction Estimate the value function of an unknown MDP This lecture: Model-free control Optimise the value function of an unknown MDP Lecture 5: Model-Free Control Introduction Uses of Model-Free Control Some example problems that can be modelled as MDPs Elevator Robocup Soccer Parallel Parking Quake Ship Steering Portfolio management Bioreactor Protein Folding Helicopter Robot walking Aeroplane Logistics Game of Go For most of these problems, either: MDP model is unknown, but experience can be sampled MDP model is known, but is too big to use, except by samples Model-free control can solve these problems Lecture 5: Model-Free Control Introduction On and O?-Policy Learning On-policy learning “Learn on the job” Learn about policy π from experience sampled from π O?-policy learning “Look over someone’s shoulder” Learn about policy π from experience sampled from μ Lecture 5: Model-Free Control On-Policy Monte-Carlo Control Generalised Policy Iteration Generalised Policy Iteration (Refresher) Policy evaluation Estimate vπ e.g. Iterative policy evaluation Policy improvement Generate π0 π e.g. Greedy policy improvement≥ Lecture 5: Model-Free Control On-Policy Monte