Markov models are a class of stochastic models of Markov processes, processes which obey the Markov property.
These models contain a transition model $\Pr[X_{t+1} \space | \space X_{t}]$ that dictate how the state at time $t$, denoted $X_t$, evolves along with an initial distribution $\Pr[X_1]$.
Furthermore, these transition probabilities are typically encoded in a matrix and are time-invariant.
For more on Markov models, see Markov Chains from CS70.
Algorithm for efficiently computing the distribution $\Pr[X_t]$ of the state at a timestep $t$.
$$ \Pr[X_t] = \sum {x{t-1}} \Pr[x_{t-1}, X_t] = \sum {x{t-1}} \Pr[X_{t} \space | \space x_{t-1}]\Pr[x_{t-1}] $$
As time goes on, the distribution of the state converges to a stationary distribution, which reflects the inductive biases embedded in the transition model.
$$ \Pr[X_{t+1}] = \Pr[X_t] $$
As a generalization of Markov chains, Hidden Markov Models model observations at each timestep.
Bayes Net Representation of a HMM. Each state is referred to as a “hidden” or “latent” state, because they cannot be directly observed (like whether the day will be rainy or sunny).
Like Markov models, HMMs are defined by an initial distribution $\Pr[X_1]$ and a transition model $\Pr[X_{t+1} \space | \space X_{t}]$, along with an observation model $\Pr[E_{t} \space | \space X_t]$, which represents how consistent each hidden state is with the evidence observed.
Moreover, the belief distribution $B_t(X_t) = \Pr[X_t \space | \space e_1 , \dots, e_t] = \Pr[X_t \space | \space e_{1:t}]$ describes the informed belief given all available evidence at time step $t$.