Tsitsiklis, “A Neuro-Dynamic Programming Approach to Call Admission Control in Integrated.pdfVIP

A NeuroDynamic Programming Approach to Call Admi s s ion Control in Integrate d Service Networks The Single Link Cas e Peter Marbach and John N Ts its ikli s Lab oratory for Inform ation and Deci s ion Systems Mas sachus etts Institute of Technology Cambr idge MA email marbachmit e du jntmitedu Ab stract We formulate the call admi ss ion control problem for a s ingle link in an integrate d s ervice environment as a Markov Deci s ion Problem In pr in ciple an optimal admi ss ion control p olicy can b e compute d us ing metho ds of Dynamic Programming However as the numb er of p oss ible state s of the un derlying Markov Chain grows exp onentially in the numb er of customer class e s Dynamic Programming algor ithms for reali stic s ize problems are computation ally infeas ible We try to overcome thi s socalle d curs e of dimens ionality by us ing metho ds of NeuroDynamic Programming NDP for short NDP employs s imulationbas e d algor ithms and function approximation technique s to nd con trol p olicie s for largescale Markov Deci s ion Problems We apply two metho ds of NDP to the call admi ss ion control problem the TD algor ithm and Approxim ate Policy Iteration We ass e ss the p erformance of the s e metho ds by compar ing with two heur i stic p olicie s a p olicy which always accepts a new customer when the require d re source s are available and a thre shold p olicy 1This re s earch was supp orte d by a contract with Siemens AG Munich Germany Intro duction Markov Deci s ion Problems have b een a p opular paradigm for s equential de ci s ion making under uncertainty Dynamic Programming provide s a f rame work for studying