Deep Reinforcement Learning_iclr2015.pdf

Deep Reinforcement Learning David Silver, Google DeepMind Reinforcement Learning: AI = RL RL is a general-purpose framework for artificial intelligence I RL is for an agent with the capacity to act I Each action influences the agent’s future state I Success is measured by a scalar reward signal RL in a nutshell: I Select actions to maximise future reward We seek a single agent which can solve any human-level task I The essence of an intelligent agent Agent and Environment state reward action at rt st I At each step t the agent: I Receives state st I Receives scalar reward rt I Executes action at I The environment: I Receives action at I Emits state st I Emits scalar reward rt Examples of RL I Control physical systems: walk, fly, drive, swim, ... I Interact with users: retain customers, personalise channel, optimise user experience, ... I Solve logistical problems: scheduling, bandwidth allocation, elevator control, cognitive radio, power optimisation, .. I Play games: chess, checkers, Go, Atari games, ... I Learn sequential algorithms: attention, memory, conditional computation, activations, ... Policies and Value Functions I Policy π is a behaviour function selecting actions given states a = π(s) I Value function Qπ(s, a) is expected total reward from state s and action a under policy π Qπ(s, a) = E [ rt+1 + γrt+2 + γ 2rt+3 + ... | s, a ] “How good is action a in state s?” Approaches To Reinforcement Learning Policy-based RL I Search directly for the optimal policy π? I This is the policy achieving maximum future reward Value-based RL I Estimate the optimal value function Q?(s, a) I This is the maximum value achievable under any policy Model-based RL I Build a transition model of the environment I Plan (e.g. by lookahead) using model Deep Reinforcement Learning I Can we apply deep learning to RL? I Use deep network to represent value function / policy / model I Optimise value function / policy /model end-to-end I Using stochastic gradient descent Bellman Equation I