4.1 Discretization School of Computing（4.1离散化学院的计算）.pdfVIP

7 using the new experience. Specifically, if we keep around the counts for both the numerator and denominator terms of (4), then as we observe more trials, we can simply keep accumulating those counts. Computing the ratio of these counts then given our estimate of Psa . Using a similar procedure, if R is unknown, we can also pick our estimate of the expected immediate reward R(s) in state s to be the average reward observed in state s. Having learned a model for the MDP, we can then use either value it- eration or policy iteration to solve the MDP using the estimated transition probabilities and rewards. For example, putting together model learning and value iteration, here is one possible algorithm for learning in an MDP with unknown state transition probabilities: 1. Initialize π randomly. 2. Repeat { (a) Execute π in the MDP for some number of trials. (b) Using the accumulated experience in the MDP, update our esti- mates for Psa (and R, if applicable). (c) Apply value iteration with the estimated state transition probabil- ities and rewards to get a new estimated value function V . (d) Update π to be the greedy policy with respect to V . } We note that, for this particular algorithm, there is one simple optimiza- tion that can make it run much more quickly. Specifically, in the inner loop of the algorithm where we apply value iteration, if instead of initializing value iteration with V = 0, we initialize it with the solution found during the pre- vious iteration of our algorithm, then that will provide value iteration with a much better initial starting point and make it converge more quickly. 4 Continuous state MDPs So far, we’ve focused our attention on MDPs with a finite number of states. We now discuss algorithms for MDPs