Classical Situation Stanford University经典情境斯坦福大学.ppt

Classical Situation MDP-Style Planning Stochastic, Partially Observable Stochastic, Partially Observable Stochastic, Partially Observable Robot Planning Frameworks MDP-Style Planning Markov Decision Process (discrete) Value Iteration Value function of policy p Bellman equation for optimal value function Value iteration: recursively estimating value function Greedy policy: Value Iteration for Motion Planning(assumes knowledge of robot’s location) Continuous Environments Approximate Cell Decomposition [Latombe 91] Parti-Game [Moore 96] Robot Planning Frameworks Stochastic, Partially Observable A Quiz Introduction to POMDPs (1 of 3) Introduction to POMDPs (2 of 3) Introduction to POMDPs (3 of 3) Value Iteration in POMDPs Value function of policy p Bellman equation for optimal value function Value iteration: recursively estimating value function Greedy policy: Missing Terms: Belief Space Expected reward: Next state density: Value Iteration in Belief Space Why is This So Complex? Augmented MDPs: Path Planning with Augmented MDPs Robot Planning Frameworks * hell heaven World deterministic State observable hell heaven World stochastic State observable [Koditschek 87, Barto et al. 89] Policy Universal Plan Navigation function sign hell? heaven? [Sondik 72] [Littman/Cassandra/Kaelbling 97] sign hell heaven sign heaven hell sign heaven hell sign ? ? sign hell heaven start 50% 50% Yes Completeness varies Computational Complexity Huge, often continuous, 6 dimensions State space size Rarely Optimality Sequences of actions Plans deterministic Environment observable State discrete continuous State/actions Classical AI/robot planning hell heaven World stochastic State observable [Koditschek 87, Barto et al. 89] Policy Universal Plan Navigation function s2 s3 s4 s5 s1 0.7 0.3 0.9 0.1 0.3 0.3 0.4 0.99 0.1 0.2 0.8 r=-10 r=20 r=0 r=1 r=0 [Bellman 57] [Howard 60] [Sutton/Barto 98] [Bellman 57] [Howard 60] [Sutton/Barto 98] From: A Moore C.G. Atkeson “