Combining Online and Offline Learning in UCT结合在线和离线学习法.ppt

A Contribution to Reinforcement Learning;Application to Computer Go Sylvain Gelly Advisor: Michele Sebag; Co-advisor: Nicolas Bredeche September 25th, 2007 Reinforcement Learning:General Scheme An Environment (or Markov Decision Process): State Action Transition function p(s,a) Reward function r(s,a,s’) An Agent: Selects action a in each state s Goal: Maximize the cumulative rewards Some Applications Computer games (Schaeffer et al. 01) Robotics (Kohl and Stone 04) Marketing (Abe et al 04) Power plant control (Stephan et al. 00) Bio-reactors (Kaisare 05) Vehicle Routing (Proper and Tadepalli 06) Basics of RLDynamic Programming Basics of RLDynamic Programming Basics of RLDynamic Programming Basics of RLDynamic Programming Basics of RLDynamic Programming Contents Theoretical and algorithmic contributions to Bayesian Network learning Extensive assessment of learning, sampling, optimization algorithms in Dynamic Programming Computer Go Bayesian Networks Bayesian NetworksMarriage between graph and probabilities theories Bayesian NetworksMarriage between graph and probabilities theories Bayesian NetworksMarriage between graph and probabilities theories BN Learning Parametric learning, given a structure Usually done by Maximum Likelihood = frequentist Fast and simple Non consistent when structure is not correct Structural learning (NP complete problem (Chickering 96)) Two main methods: Conditional independencies (Cheng et al. 97) Explore the space of (equivalent) structure+score (Chickering 02) BN: Contributions New criterion for parametric learning: learning in BN New criterion for structural learning: Covering numbers bounds and structural entropy New structural score Consistency and optimality Notations Sample: n examples Search space H P true distribution Q candidate distribution: Q Empirical loss Expectation of the loss (generalization error) Parametric Learning(as a regression problem) Loss function: Results Theorems: cons