基于批量递归最小二乘的自然Actor-Critic算法.pdfVIP

DOI：10．3785／j．issn．1008—973X．2015．07．019 基于批量递归最小二 王国芳，方 舟，李 平 (浙江大学航空航天学院，浙江杭州310027) 摘 要：为了减轻Actor-Critic结构中智能体用最小二乘法估计自然梯度时的在线运算负担，提高运算实时性，提 出新的学习算法：NAC-BRLS．该算法在Critic中利用批量递归最小二乘法估计自然梯度，根据估计得到的梯度乐 观地更新策略．批量递归最小二乘法的引入使得智能体能根据自身运算能力自由调整各批次运算的数据量．即每 次策略估计时使用的数据量，在全乐观和部分乐观之间进行权衡，大大提高了NAC-LSTD算法的灵活性．山地车 仿真实验表明，与NAC-LSTD算法相比，NAC—BRLS算法在保证一定收敛性能的前提下，能够明显降低智能体的 单步平均运算负担． 关键词：自然梯度；Actor—Critic；批次更新；递归最小二乘 18 中图分类号：TP 文献标志码：A 文章编号：1008—973X(2015)07—1335—08 NaturalActor-Criticbasedonbatchrecursive least-。squares WANG Zhou，LI Guo—fang，FANG Ping (SchoolAeronauticsand 310027，China) of Astronautics，ZhejiangUniversity，Hangzhou Abstract：Thecallednaturalactor—criticbasedonbatchrecursive algorithm was inordertoreducetheonline burdenofthe and thereal--time proposed computation agentimprove oper—- ation．The batchrecursive inCritictoevaluatethenatural algorithmemployed least—squares gradient，and inActortheestimatednatural useofbatchrecursiveleast— performedoptimisticupdate by gradient．The enablesthe to thedatesizeof batch toits squares agentadjust every accordingoperationalcapability．A trade-offbetween and was the ofNAC— fullyoptimisticpartiallyoptimisticmade，improvingflexibility resultsinmountaincarshowthatNAC—BRLSreducesthe corn— LSTD．Simulation largely computational without the withNAC—LSTD． plexity obviouslyaffectingcon