基于角色跟踪的再励学习算法.pdfVIP

2008 Agent MAS 0 MAS MAS Agent Agent Agent Peter Stone[4] Lagoudakis [1] Agent Agent Bowling[2-3] Agent Agent 1 Markov Decision Process (MDP) Q 1 Markov S A r p S A r S × A → R Agent p S×A→? ? S Markov Agent π ν ( s, π ) = ∞ ∑β t =0 (t E rt | π , s0 = s) s0 rt t ν (s,π ) = r (s, aπ ) + β ∑ p (s | s, aπ )ν (s,π ) s β ∈[0,1) aπ π NY207120. : 1982 zyzh@. chenjianfeng8252@163.com 198 2008 π? ν (s,π ? ) = max a ? ?r (s, a ) + β ∑ p ( s | s , a )ν ( )s , π ? ? ? ? s ? Agent Agent s∈S (s,π ? ) Bellman Q- Bellman s Q- s∈S Q? (s, a) = r (s, a) + β ∑ p (s | s, a)ν (s,π ? ) s a ∈ A Agent Q ( s, a) Q- t Agent ( )rt Qt +1 (s, a) = (1 ? α t )Q t (s, a) +α t ?? r t + β max b Q t s , b ?? Q αt ∈[0,1) αt Qt (s, a) 1 Q? ( s, a) 2 Agent RoboCup MAS 2D Agent [1] minimax-Q Agent { } { }G1 = Ag 1 i | i = 1, L , n G2 = Ag 2 i | i = 1, L , m Ri k R = {R i | i = 1, L , k} G1 2 ( )A2 = a2 1 , a2 2 , L , a2 m ( )π i = λ s, Ai i s Agent j δf r [1] Ai i Act M = S , A1 , A2 , π 1 , π 2 , T , p j , f , r i S ( )A1 = a1 1 , a1 2 , L , a1 n πi i i Ai T S × Act × Act → S pij i j = 1,L, k i = 1,L, m f S × Ag × Act × R → δ Agent R 199 2008 3 Bowling TR-Q [2-3] Q minimax-Q [1] 1 t=0 ( )s ∈ S Q s, A1, A2 = Q0 T ∈ Act ( )Q s, A1, A2 λ ( s,T ) = λ0 Q0 // 2t G2 A2 t +1 nm ∑ ( ) ∑ ( )MR S t , 1, i, a1 i + MR S t , 2, i, a2 i α = i=1 t i =1 n+m S × A1 × A2 A1 σr ij = σ0 ( )s, a1, a2 st +1 (1) p ∑ ( )σ j ik ? f st , j,i, aij , k ( )MR St, j,i, aij = k=1 p ( )∑ f st , j,i, aij , k k =1 (2) Agent i st aij p 3 ( ) ( ) ( ) ∑ ( ) ( )Q s , A1, A2 ∑ ( )t = α ? Q s , A1, A2 tt + 1?α t λ ?(R t + max(Q(s , (T , arg min(1 Ti ∈Act1 t +1 i T2 j ∈Act2 1 Tj ∈Act1 s t +1 , T1 i ?Q s , T1,T2 t ij Q s , T1,T2 t ij ))))) 1 Ti ∈Act1 ,Tj2 ∈Act2 4i ∑α r ij = α r ij + f ( st , j, ai ) ? 1 p f ( st , k, ai ) p k =1 j (3) (4) 200 2008 ( )∑ Q st , A1 , T 2 j n ∑ MR (st , i, ai ) ( )λ st , A1 = β ? T 2 j ∈ A ct2 ∑ ( ) ( )Q st , Ti1 , T 2 j β+ 1 ? i =1,ai = a1i n Ti1∈ A ct1 ,T 2 j ∈ A ct2 5 t = t +1 2 4 (5) RoboCup 2D 22 [5] 4-1 1 6 T