Continuous-Space.ppt

Planning and Acting in Partially Observable Stochastic DomainsLesli P. Kaelbling, Michael L.Littman, Anthony R.Cassandra CS594 Automated decision making course presentation Professor: Piotr. Hui Huang University of Illinois, Chicago Oct 30, 2002 Outline MDP POMDP Basic POMDP ? Belief Space MDP Policy Tree Piecewise Linear Value Function Tiger Problem MDP Model Policy Value Function Which action, a, should the agent take? In MDPs: Policy is a mapping from state to action, ?: S ? A Value Function V?t(S) given a policy ? The expected sum of reward gained from starting in state s executing non-stationary policy ? for t steps. Relation Value function is evaluation for policy based no the long-run value that agent expects to gain from execute the policy. Optimization (MDPs) Recursively calculate expected long-term reward for each state/belief: Find the action that maximizes the expected reward: POMDP: UNCERTAINTY A broad perspective What are POMDPs? Belief state Probability distributions over states of the underlying MDP The agent keeps an internal belief state, b, that summarizes its experience. The agent uses a state estimator, SE, for updating the belief state b’ based on the last action at-1, the current observation ot, and the previous belief state b. Belief state is a sufficient statistic (it satisfies the Markov proerty) 1D belief space for a 2 state POMDP POMDP ? Continuous-Space Belief MDP a POMDP can be seen as a continuous-space “belief MDP”, as the agent’s belief is encoded through a continuous “belief state”. We may solve this belief MDP like before using value iteration algorithm to find the optimal policy in a continuous space. However, some adaptations for this algorithm are needed. Belief MDP The policy of a POMDP maps the current belief state into an action. As the belief state holds all relevant information about the past, the optimal policy of the POMDP is the the solution of (continuous-space) belief MDP. A b