运筹学Ⅲ教学资料 17 Markov Chains.pdfVIP

Markov Chains Sometimes we are interested in how a random variable changes over time. For example, we may want to know how the price of a share of stock or a firm’s market share evolves. The study of how a random variable changes over time includes stochastic processes, which are ex- plained in this chapter. In particular, we focus on a type of stochastic process known as a Markov chain. Markov chains have been applied in areas such as education, marketing, health services, finance, accounting, and production. We begin by defining the concept of a sto- chastic process. In the rest of the chapter, we will discuss the basic ideas needed for an un- derstanding of Markov chains. 17.1 What Is a Stochastic Process? Suppose we observe some characteristic of a system at discrete points in time (labeled 0, 1, 2, . . .). Let X be the value of the system characteristic at time t. In most situations, X t t is not known with certainty before time t and may be viewed as a random variable. A discrete-time stochastic process is simply a description of the relation between the random variables X , X , X , . . . . Some examples of discrete-time stochastic processes follow. 0 1 2 E X A M P L E 1 Th