Advice Generation from Observed Execution Abstract Markov Decision Process Learning.pdf

Advice Generation from Observed Execution: Abstract Markov Decision Process Learning Patrick Riley and Manuela Veloso ? Computer Science Department Carnegie Mellon University Pittsburgh, PA 15213-3891 pfr@ and mmv@ Abstract An advising agent, a coach, provides advice to other agents about how to act. In this paper we contribute an advice generation method using observations of agents acting in an environment. Given an abstract state def- inition and partially specified abstract actions, the al- gorithm extracts a Markov Chain, infers a Markov De- cision Process, and then solves the MDP (given an ar- bitrary reward signal) to generate advice. We evaluate our work in a simulated robot soccer environment and experimental results show improved agent performance when using the advice generated from the MDP for both a sub-task and the full soccer game. Introduction A coach agent provides advice to other agent(s) to improve their performance. We focus on a coach that analyzes past performance to generate advice. The synthesis of observed executions in a manner that facilitates advice generation is a challenging problem. Observations that do not explicitly include the actions taken by the agents are an additional challenge. The in- tended actions must be inferred from observed behavior. In this paper we present algorithms to learn a model, including actions, based on such observations. The model is then used to generate executable advice for agents. The areas of advice reception (e.g. Maclin Shav- lik 1996) and advice generation, in both Intelligent Tutor- ing Systems (e.g. Paolucci, Suthers, Weiner 1996) and item recommendation (e.g. Shani, Brafman, Hecker- man 2002), have received attention in AI over many years. Our work in this paper further explores advice generation in the context of agent to agent advice. In most coaching environments, it is impractical for the coach to provide advice only at the most detailed level of states and actions because of communication b