强化学习PPT.ppt

蒙特卡罗控制 How to select Policies: (Similar to policy evaluation) MC policy iteration: Policy evaluation using MC methods followed by policy improvement Policy improvement step: greedify with respect to value (or action-value) function 时序差分学习 Temporal-Difference target: the actual return after time t target: an estimate of the return 时序差分学习 (TD) Idea: Do ADP backups on a per move basis, not for the whole state space. Theorem: Average value of U(i) converges to the correct value. Theorem: If ? is appropriately decreased as a function of times a state is visited (?=?[N[i]]), then U(i) itself converges to the correct value 时序差分学习 TD T T T T T T T T T T T T T T T T T T T T TD(l) – A Forward View TD(l) is a method for averaging all n-step backups weight by ln-1 (time since visitation) l-return: Backup using l-return: 时序差分学习算法 TD(?) Idea: update from the whole epoch, not just on state transition. Special cases: ?=1: Least-mean-square (LMS), Mont Carlo ?=0: TD Intermediate choice of ? (between 0 and 1) is best. Interplay with ? … 时序差分学习算法 时序差分学习算法收敛性TD(?) Theorem: Converges w.p. 1 under certain boundaries conditions. Decrease ?i(t) s.t. In practice, often a fixed ? is used for all i and t. 时序差分学习 TD Q-LearningWatkins,1989 Estimate the Q-function using some approximator (for example, linear regression or neural networks or decision trees etc.). Derive the estimated policy as an argument of the maximum of the estimated Q-function. Allow different parameter vectors at different time points. Let us illustrate the algorithm with linear regression as the approximator, and of course, squared error as the appropriate loss function. Q-learning Q (a,i) Direct approach (ADP) would require learning a model . Q-learning does not: Do this update after each state transition: Exploration Tradeoff between exploitation (control) and exploration (identification) Extremes: greedy vs. random acting (n-armed bandit models) Q-learning converges to optimal Q-values i