next up previous
Next: Long-Run Behavior of Markov Up: Discrete Time Markov Chains Previous: State Probabilities

Transition Probabilities

The one-step transition probability is the probability of transitioning from one state to another in a single step. The Markov chain is said to be time homogeneous if the transition probabilities from one state to another are independent of time index $n$.

\begin{displaymath} p_{ij} = Pr\{X_{n}=j \vert X_{n-1}=i \} \end{displaymath}

The transition probability matrix, $P$, is the matrix consisting of the one-step transition probabilities, $p_{ij}$.

The $m$-step transition probability is the probability of transitioning from state $i$ to state $j$ in $m$ steps.

\begin{displaymath} p^{(m)}_{ij} = Pr\{X_{n+m}=j \vert X_{n}=i \} \end{displaymath}

The $m$-step transition matrix whose elements are the $m$-step transition probabilities $p^{(m)}_{ij}$ is denoted as $P^{(m)}$.

The $m$-step transition probabilities can be found from the single-step transition probabilities as follows.

To transition from $i$ to $j$ in $m$ steps, the process can first transition from $i$ to $r$ in $m-k$ steps, and then transition from $r$ to $j$ in $k$ steps, where $0<k<m$.

\begin{displaymath} p^{(m)}_{ij} = \sum_{r} p^{m-k}_{ir} p^{k}_{rj} \end{displaymath}

In matrix form, this becomes:

\begin{displaymath} P^{(m)} = P^{(m-k)}P^{(k)} \end{displaymath}

Setting $k=m-1$ yields:

\begin{displaymath} P^{(m)} = P \cdot P^{(m-1)} \end{displaymath}

From this equation we can see that:

\begin{displaymath} P^{(m-1)} = P \cdot P^{(m-2)} \end{displaymath}

Substituting this back into the previous equation yields:

\begin{displaymath} P^{(m)} = P \cdot P \cdot P^{(m-2)} \end{displaymath}

Continuing these substitutions, eventually we have:

\begin{displaymath} P^{(m)} = P \cdot P \cdot P \cdots P = P^{m} \end{displaymath}

Therefore, the $m$-step transition probability matrix can be found by multiplying the single-step probability matrix by itself $m$ times.

The state vector at time $m$ can also be found in terms of the transition probability matrix and the intial state vector $\Pi(0)$. We first observe that:

\begin{displaymath} \pi_{j}(m) = \sum_{i} \pi_{i}(m-1) p_{ij} \end{displaymath}

In vector and matrix form, this becomes:

\begin{displaymath} \Pi(m) = \Pi(m-1) P \end{displaymath}

We also find that, through substitution:

\begin{displaymath} \Pi(m-1) = \Pi(m-2) P \end{displaymath}

or,

\begin{displaymath} \Pi(m) = \Pi(m-2) P \cdot P \end{displaymath}

Continuing the substitution yields:

\begin{displaymath} \Pi(m) = \Pi(0) P^{m} \end{displaymath}

where $\Pi(0)$ is the vector containing the intial probabilities of being in each state at time 0.