随机过程导论Chapter 4.pdfVIP

Stochastic Processes Ma Shixiang HIT Shenzhen Graduate School 1 Chapter 4 Discrete-time Markov chain 4.1 Introduction 4.2 Chapman-Kolmogorov equation 4.3 Classification of states 4.4 Periodic and Ergodic Markov chains 4.5 Absorbing Markov chains 2 4.1 Introduction Definition: X={X , n=0,1,2…}, where X denotes the state of the system n n at time n. Xn takes on values in the set S — state space.S={0,1,2…}. The stochastic process X is a discrete-time Markov chain with state space S if P {X j X i, X i ,..., X i X i } P {X j X i} n+1 n n−1 n−1 1 1, 0 0 n+1 n Eq.4-1-1 holds for all i,j , i , i ,…i in S and all n = 0,1, …… 0 1 n- 1 3 4.1 Introduction Transition probability In a Markov chain, the n-step transition probability is defined by p (n) P {X j X i} Eq.4-1-2 ij n+m m This is the probability that the process goes from state i to statej in n transitions. Transition probability matrix: The matrix containing all transition probabilities P ={p ij} 4 4.1 Introduction Example (4.1.1) Assume that once the mouse 1 2 3 finds either the piece of cheese mouse