深度学习课件：深度强化学习复习过程.pptx

Introduction to Deep Reinforcement LearningYen-Chen Wu2015/12/11OutlineReinforcement LearningMarkov Decision ProcessHow to Solve MDPsDPMCTDQ-learning (DQN)Paper ReviewReinforcement LearningBranches of Machine LearningWhat makes different?There is no supervisor, only a reward signalFeedback is delayed, not instantaneousTime really matters (sequential, non i.i.d data)Agent’s actions affect the subsequent data it receivesGoal: Maximize Cumulative RewardActions may have long term consequencesReward may be delayedIt may be better to sacrifice immediate reward to gain more long-term rewardAgent & Enviroment→←↑↓DefenseAttackJumpFull observability vs Partial observabilityLearning and PlanningExploration and ExploitationPrediction and ControlMarkov ProcessesMarkov Reward Processes Markov Decision ProcessesMarkov Decision ProcessMarkov ProcessMarkov Reward Processes Markov Decision ProcessMarkov Decision Process(MDP)S : finite set of states (observations)A : finite set of actionsP : transition probabilityR : immediate rewardγ : discount factorGoal :Choose policy πMaximize expected return :Dynamic ProgrammingMonte-CarloTemporal-DifferenceQ-LearningHow to Solve MDPModel-basedDynamic ProgrammingEvaluate policyUpdate policyModel FreeUnknown Transition Probability & RewardMC vs TDModel Free: Q-learningBasic idea : iterative update (lack of generalization)In practical : function approximatorLinear ?Using DNN !Instead of tabularoptimal action-value function (Q-learning) =Bellman equationDeep Q-network (DQN)Video/watch?v=LJ4oCb6u7kk/watch?v=LJ4oCb6u7kkDeep Q-NetworkFull-connected 512 nodesInput : 84x84x4Output a node for each actionConvolves 32 filters of 8x8 with stride 4Convolves 64 filters of 4x4 with stride 2Convolves 64 filters of 3x3 with stride 1compute Q-values for all actionsUpdate DQNLoss functionGradientTwo Technique ExampleLearn the value of…Pros & ConsOn-policySARSApolicy being carried out by the agentFast but weakOff-policyDQNoptimal policy independently of the agent's