As a preliminary remark, this note is not very precise. This offering of 126 was extremely unorganized and the content below is an amalgam of information from 3 different sources.


Continuous Time Markov Chains

A continuous time Markov chain is a discrete time Markov chain where the transitions occur as a continuous arrival process.

$$ \textbf{Time of $n$th Transition } Y_n \\ \textbf{Time between $n$th and $(n-1)$th Transition } T_n \hspace{99pt} $$

Moreover, we define $\upsilon _i$ as the mean rate of transitions out of state $i$ and assume that the time until the next transition from a state $i$ is exponential with parameter $\upsilon_i$.

$$ T_n \mid X_{n-1} = i \sim \text{Exp}(\upsilon _i) $$

To make analysis more precise, we define $q_{ij} := \upsilon_i p_{ij}$ as the mean rate of transitions from $i$ to $j$, where $i \neq j$, as we see later. These rate parameters are packaged into a transition-rate matrix $Q$, which replaces the stochastic matrix $P$.

$$ \textbf{Transition-Rate Matrix } Q \in \R^{m \times m} , \space p_{ii} = 0\\ \\

\sum {j=1} ^m q{ij} = 0 $$

Moreover, we have the following properties:

$$ \upsilon_i = \sum {j \neq i} ^m q{ij} = -q_{ii} \\ p_{ij} = \frac{q_{ij}}{\upsilon_i}
$$

A similar result for $n$-step probabilities in DTMCs follows for CTMCs:

$$ P(t) = e^{Qt} $$

Embedded Chain

As a final remark, we say that $\{X_n\}$ is the embedded discrete time Markov chain of the CTMC, from which the following properties emerge: