represents a discrete-time, finite-state Markov process with transition matrix m and initial state i0.
represents a Markov process with initial state probability vector p0.
represents a Markov process with transition matrix from the graph g.
- DiscreteMarkovProcess is also known as a discrete-time Markov chain.
- DiscreteMarkovProcess is a discrete-time and discrete-state random process.
- The states of DiscreteMarkovProcess are integers between 1 and , where is the length of transition matrix m.
- The transition matrix m specifies conditional transition probabilities m〚i,j〛Probability[x[k+1]jx[k]i], where x[k] is the state of the process at time k. »
- A discrete Markov process can be seen as a random walk on a graph, where the probability of transitioning from state to state is specified by m〚i,j〛.
- EstimatedProcess[data,DiscreteMarkovProcess[n]] indicates that a process with n states should be estimated.
- The transition matrix in the case of a graph g is constructed to give equal probability of transitioning to each incident vertex.
- DiscreteMarkovProcess allows m to be an × matrix with non-negative elements and rows that sum to 1, i0 is an integer between 1 and , and p0 is a vector of length of non-negative elements that sum to 1.
- DiscreteMarkovProcess can be used with such functions as MarkovProcessProperties, PDF, Probability, and RandomFunction.
Examplesopen allclose all
Basic Examples (2)
Generalizations & Extensions (2)
A biased coin with probability of getting heads being is flipped times. Find the probability that the length of the longest run of heads exceeds a given number . This can be modeled by using a discrete Markov process, where the state represents having heads in a run. The resulting transition matrix for , , and is given by:
A gambler, starting with three chips, places a one-chip bet at each step, with a winning probability of 0.4 and a goal of winning seven units before going broke. The states here are the integers 1 through 8, representing the gambler's wealth plus one:
In the game of craps, a player rolls a pair of dice and sums the numbers. On the first throw, he wins if he rolls 7 or 11 and loses if he rolls 2, 3, or 12, while any other number is called a point and the dice rolls continue. In later throws, the player wins if he rolls the point number and loses if he rolls 7. The states are: start, win, lose, p4, p5, p6, p8, p9, or p10. He never returns to start, while win and lose are absorbing states. The following distribution represents a roll of two dice:
A simple weather model is: given that it rains today, it will rain tomorrow with probability 0.7; and given that it does not rain today, it will rain tomorrow with probability 0.4. Given that it is raining today, represent this model with a discrete Markov process and find the probability that it will rain four days from now. A representation of the weather:
Build a weather model based on whether it rained on the two previous days. Suppose it will rain tomorrow with probability 0.7 if it rained today and yesterday, with probability 0.5 if it rained today but not yesterday, with probability 0.4 if it rained yesterday but not today, and with probability 0.2 if it did not rain yesterday or today. Use a product state space , where indicates whether it rained yesterday and whether it rained today:
A taxi is located either at the airport or in the city. From the city, the next trip is to the airport with probability 1/4, or to somewhere else in the city with probability 3/4. From the airport, the next trip is always to the city. Model the taxi using a discrete Markov process with state 1 representing the city and state 2 representing the airport, starting at the airport:
With balls distributed between urns A and B, at each step a ball is randomly selected and transferred to the other urn. Find the stationary distribution of balls in urn A. The count of balls in urn A can be modeled as a discrete Markov process. If there are balls in urn A, the probability of losing a ball from A is , and the probability of gaining a ball is , which gives the transition matrix:
Random Walks (3)
For a non-lazy random walk absorbing on both sides, if the odds of moving to the right are 2:1, even if the process is started right next to the left boundary there is still a better than even chance of ending up at the right boundary:
Machine Repair (1)
A work cell in a machine shop consists of three machines, where on a given day a machine fails with probability 0.1. Machine 1 feeds both machines 2 and 3, and if machine 1 fails, there is no production that day. Only one machine can be repaired on the same day so it is available for the following day. If several machines are down, they are repaired in priority order 1, 2, and 3. A machine that has been repaired is assumed to be working the next day. Find the proportion of time there can be some production. The failure and repair process can be modeled as a discrete Markov process by enumerating all possible combinations of failed and working machines:
In a four-state Markov chain model of a Bonus–Malus auto insurance system, each policyholder is given one of the four states 1, 2, 3, or 4, depending on the number of claims made in the preceding year, and this state determines the annual premium. No claims typically results in a lower premium, while one or more claims typically result in a higher premium. With premium levels for the four states and the given transition matrix of a policyholder's successive states, find the average yearly premium if the policyholder's yearly number of claims is a Poisson random variable with mean :
Assuming there are 365 days in a year and each day is equally probable for a birthday, find the minimum number of people such that there is a 50% probability of at least two of them sharing a birthday. The states are —number of people having distinct birthdays and —at least two people have the same birthday:
Consider a simple natural selection model in which the population of a species on an isolated island is fixed at by the food supply. A mutant is born with a slightly better chance of survival than a regular member of the species, and in each generation the mutants either gain a place with probability or lose a place with probability :
Suppose radio messages from Mars are written in a language that uses only one vowel A and four consonants BCDR, and interspersed in the ordinary communication are some long messages produced by a Markov chain with the following estimated transition matrix: