基于状态空间划分的预测状态表示模型学习算法.docVIP

  基于状态空间划分的预测状态表示模型学 习算法# 刘云龙* 5 10 15 20 25 30 35 40 （厦门大学自动化系） 摘要：预测状态表示（Predictive State Representations, PSRs）是解决局部可观测问题的有效 方法，但现有研究往往以整个状态空间为问题背景或仅能获取系统的局部模型，限制了 PSR 方法的应用范围。本文提出了一种基于状态空间划分的 PSR 模型的学习算法，首先根 据训练数据，利用 Landmark 将状态空间划分为多个子状态空间，进而确定每个子空间 PSR 模型以及缺值数据的获取方式，最后利用子空间 PSR 模型构建整个系统的 PSR 模型，实现 对任意事件的预测。所提方案降低了获取模型的难度，提高了预测的精度。通过在 Cheese Maze 问题上的应用，验证了所提算法的准确性和有效性。 关键词：预测状态表示；局部模型；状态空间划分 中图分类号：TP282 An algorithm for Learning Predictive State Representations Based on State Space Partitioning Liu Yunlong (Department of Automation, Xiamen University, 361005, Xiamen) Abstract: Predictive State Representations (PSRs) are powerful methods of modeling dynamical systems by representing state using only observable data. Much progress has been made since PSRs were proposed. However, current techniques in PSRs focus on learning a model on the entire state space, or learning a local model of the system, which limit the application of PSRs. In this paper, an algorithm for learning the PSR model of a system based on state space partitioning is proposed. First, according to the training data, the entire state space is partitioned into sub-state spaces using landmarks of the environment, and then traditional technique is used to learn every sub-state space’s PSR model, which is easier than learning the model on the entire state space. We then show how one might combine these local models to produce the complete model of the underlying system. We demonstrate our ability to obtain the complete model on the cheese maze problem, and empirical results are given to show effectiveness of our algorithm. Key words: Predictive State Representations; Local Model; State Space Partitioning 0 引言 在人工智能领域，对于一个随机的、局部可观测的环境，智能体由于受到其感知能力的 限制，有可能会感知不到所处环境的某些重要特征，同时，智能体采取的动作也往往得不到 预期的效果。在这种局部可观测、随机的系统中如何获取智能体的最优策略——即不确定环 境下的规划问题，是人工智能领域研究的一个重要问题[1]。常用的解决这种问题的途径是首 先对系统建模，然后根据得到的系统模型对问题进行求解。 局部可观测马尔可夫决策过程(Partially Observable Markov Decision Processes, POMDPs) 为解决这种不确定环境下的规划问题提供了丰富的数学框架[1