武大讲义-Dynamic optimization V.doc

V Discrete Time Optimization This chapter gives a brief introduction to discrete time dynamic optimization problems. The term dynamic refers to the fact that the problems involve systems evolving over time. Time is here measured by the number of whole periods (say weeks, quarters, or years) that have passed since time 0. So we speak of discrete time. In this case it is natural to study dynamic systems whose development is governed by difference equations. If the horizon is finite, then such dynamic problems can be solved, in principle, using classical calculus methods. There are, however, special solution techniques described in the present chapter that take advantage of the special structure of discrete dynamic optimization problems. Most of the chapter is concerned with dynamic programming. This is a general method for solving discrete time optimization problems that was formalized by R. Bellman in the late 1950s. There is also a brief introduction to discrete time control theory. The last two sections cover stochastic dynamic programming. 5.1 Dynamic Programming Consider a system that is observed at times . Suppose the state of the system at time is characterized by a real number . For example, might be the quantity of grain that is stockpiled at time . Assume that the initial state is historically given, and that from then on the system evolves through time under the influence of a sequence of controls , which can be chosen freely from a given set , called the control region. For example, might be the fraction of grain removed from the stock during period . The controls influence the evolution of the system through a difference equation , given, (1) where is a given function. Thus, we assume that the state of the system at time depends explicitly on the time , on the state in the preceding period , and on , the value chosen for the control at time . Suppose that we choose values for . Then (1) gives . Since is now known,