Chapter 9Dynamic Decision Processes.ppt

Equivalent discrete time average-cost MDP a discrete-time Markov chain with uniform transition rate g a stage cost given by C s, a /g whenever a state s is entered and an action a is chosen. Optimality equation : where g average cost per discretized time period gg average cost per time unit can also be obtained directly from the optimality equation with stage cost C s, a Cost function convertion for uniformized Markov chain Example continue Uniformize the Markov decision process with rate g p+d The optimality equation: Example continue From the optimality equation: If V s is convex, then there exists a K such that : V s+1 –V s 0 and the decision is not producing, for all s K and V s+1 –V s 0 and the decision is producing, for all s K Example continue Convexity proved by value iteration Proof by induction. V0 is convex. If Vn is convex with minimum at s K, then Vn+1 is convex. K-1 K s * Assumptions Assumption 1: The decision epochs T 1, 2, … Assumption 2: The state space S is finite or countable Assumption 3: The action space As is finite for each s Assumption 4: Stationary costs and transition probabilities; C s, a and p j |s, a , do not vary from decision epoch to decision epoch Assumption 5: Bounded costs: | Ct s, a | ? M for all a ? As and all s ? S to be relaxed Assumptions Criterion: where 0 l 1 is the discounting factor PHR is the set of all possible policies. Optimality equations Theorem: Under assumptions 1-5, the following optimal cost function V* s exists: and satisfies the following optimality equation: Further, V* . is the unique solution of the optimality equation. Moreover, a statonary policy p is optimal iff it gives the minimum value in the optimality equation. Computation of optimal policyValue Iteration Value iteration algorithm: Select any bounded value function V0, let n 0 For each s ?S, compute Repeat 2 until convergence. For each s ?S, compute Theorem: Under assumptions 1-5, Vn c