基于拉普拉斯特征映射的启发式Q学习.pdfVIP

第29 卷 第3 期 控 制 与 决 策 2014 年 3 月 Vol. 29 No. 3 Control and Decision Mar. 2014 文章编号: 1001-0920 (2014) 03-0425-06 DOI: 10.13195/j.kzyjc.2012.1669 基于拉普拉斯特征映射的启发式Q 学习 朱美强, 李 明, 程玉虎, 张 倩, 王雪松 (中国矿业大学信息与电气工程学院，江苏徐州221116) 摘 要: 在基于目标的强化学习任务中, 欧氏距离常作为启发式函数用于策略选择, 其用于状态空间在欧氏空间内 不连续的任务效果不理想. 针对此问题, 引入流形学习中计算复杂度较低的拉普拉斯特征映射法, 提出一种基于谱图 理论的启发式策略选择方法. 所提出的方法适用于状态空间在某个内在维数易于估计的流形上连续, 且相邻状态间 的连接关系为无向图的任务. 格子世界的仿真结果验证了所提出方法的有效性. 关键词: 强化学习；启发式策略选择；Q 学习；拉普拉斯特征映射 中图分类号: TP181 文献标志码: A Heuristically accelerated Q-learning algorithm based on Laplacian Eigenmap ZHU Mei-qiang, LI Ming, CHENG Yu-hu, ZHANG Qian, WANG Xue-song (School of Information and Electrical Engineering，China University of Mining and Technology，Xuzhou 221116, China ．Correspondent ：ZHU Mei-qiang ，E-mail ：tianlianglu@163.com) Abstract: As a heuristic function, the Euclidean distance is usually used to select online action in reinforcement learning based on goal position. It is not applied to these tasks whose state spaces are not continuous in Euclidean space. For the problem, the Laplacian Eigenmap whose computational complexity is lower in manifold learning is introduced, then a method of heuristic policy selection based on the spectral graph theory is proposed. The proposed method is suitable for these tasks not only whose state spaces are continuous in some manifold that has a good estimation of intrinsic dimension, but also whose connection relation is expressed by an undirected graph. The simulation results of grid world show the effectiveness of the proposed method. Key words: reinforcement learning ；heuristic