半Markov控制过程在线自适应优化算法.pdfVIP

Proceedings of the 25th Chinese Control Conference 7–11 August, 2006, Harbin, Heilongjiang Markov ,, , 230027 E-mail: jiangqi@mail.ustc.edu.cn : Markov : Markov , Markov 1 , , , Markov Online Adaptive Optimization Algorithm for Semi-Markov Control Processes Jiang Qi, Xi Hongsheng, Yin Baoqun Department of Automation, University of Science and Technology of China, Hefei 230027, China E-mail: jiangqi@mail.ustc.edu.cn Abstract: Semi-Markov control problems with unknown kernel are considered, a reinforcement learning based online adaptive optimization algorithm is proposed. First an event-driven stochastic switching model is introduced to formulate the semi-Markov control problems. Then by utilizing the features of event-driven policy an optimization algorithm that combines policy gradient estimation and stochastic approximation is derived. This algorithm can converge to global optimization without the explicit knowledge of the semi-Markov kernel. Moreover, this algorithm does not require the computation of performance potentials or other related quantities (e.g. Q-factors) and therefore saves computational cost signi?cantly. Simulation results demonstrate the effectiveness of the proposed algorithm. Key Words: Semi-Markov control processes, Performance potential, Event-based optimization, Policy gradient estimation, Stochastic approximation 1 (Introduction) Markov MCP Markov MCP [1-2] Markov Markov [5] [3] Markov Markov [4] SMCP Markov Markov Markov IEEE Catalog Number: 06EX1310 : 050420301. , SMCP MCP (CTMCP) SMCP MCP (DTMCP) [6] DTMCP SMCP QMarkov 1066 [6] SMCP 2 Markov chastic Switching Model of SMCPs) (Sto- SMCP: {xn, tn, un, n ≥ 0}[7] tn n xn = x(tn) ∈ X = {1, 2, · · · , N } x(t) = xn, tn ≤ t tn+1 n un = u(tn) ∈ U = {1, 2, · · · , K} u(t) = un, tn ≤ t tn+1 n tn n [tn, tn+1) n tn n SMCP (i, k) Qij(τ, k) = P {xn+1 = j, tn+1 ? tn ≤ τ |xn = i, un = k}, Qij (k) = lim τ →∞ Qij (τ, k) = P {xn+1 = j|xn = i, un = k}, τ i(k) = ∞ j∈X 0 τ Qij (dτ, k) = E