Convergence Analysis of Certain Reinforcement Learning Algorithms in SelfPlay.pdfVIP

Convergence Analysis of Certain Reinforcement Learning Algorithms in Self-Play Srinivas Turaga Yonatan Loewenstein I study the emergent behavior in the self-play of simple value function based reinforcement learning algorithms in an iterative two player general-sum game. The players are assumed to have limited knowledge of the game. Therefore learning algorithms only use information about their own choice and recieved reward histories to compute a value function based on which actions are chosen stochastically. Two related algorithms are studied, with fixed points that correspond to matching and Nash equilibria respectively. Linear stability analysis is used to probe the dynamics about the fixed points. Linear analysis shows that the matching equilibrium is stable, while the algorithm with the Nash equilibrium has marginal stability. Numerical simulations suggest that the Nash equilibrium is unstable for this class of algorithms and that algorithms with equilibria lying near the matching equilib- rium are progressively more stable. 2 Writeup.nb Introduction Many interactions between people can be phrased in terms of the principles of game theory. We may think of people as players choosing from a space of actions, each of which produces a different outcome or payoff. What makes the problem interesting is that the payoff that a player receives generally depends strongly on the actions of the other players in the game. Classical game theory provides a mechanism to study optimal behavior in games involving rational players. Nash equilibria, which