# Markov process

Also found in: Dictionary, Thesaurus, Medical, Acronyms, Wikipedia.

## Markov process

[′mär‚kȯf prä·səs]## Markov Process

an important special type of random processes, which are of great importance in applications of probability theory to various branches of natural science and technology.

An example of a Markov process is the decay of a radioactive substance. It is known that the probability of disintegration of a given atom over a short time interval *dt* is given by *adt*, where a is a constant that characterizes the rate of disintegration of the radioactive substance. This probability is independent of the fate of all the other atoms and of the age of the particular atom. Let *N* denote the number of atoms of a radioactive substance at some initial moment of time *l* = 0, and let *P*_{n}(*t*) denote the probability that *n* atoms will have disintegrated by time *t*. The probabilities *P*_{n}(*t*) satisfy the system of differential equations

where *n* = 1, 2, …, *N*. Solving this system of equations with the initial data

*P*_{0}(0) = 1 *P*_{n}(0) = 0 1 ≤ *n* ≤ *N*

we obtain

*P _{n}(t)* =

*C*(

_{N}^{n}*1 — e*)

^{-αt}^{n}e

^{-(N — n)αt}

In this example there exist 0, 1, 2, …, *N* disintegrated atoms at each moment of time; their number characterizes the state of the phenomenon under study.

This example may be put into the following more general scheme. Let ω_{1}, ω_{2},…,ω_{n}… be all possible states of a given system, either finite or infinite in number. At each moment the system may be in one of these states, and over the course of time it passes randomly from one state into another. This is called a Markov process if the state of a system ω_{i}, at a given moment is determined only by the probability *P _{ij}* that after time

*t*the system will be in state >

_{j}, this probability is independent of the course of the process in the preceding period. The probabilities

*P*(

_{ij}*t*) are called transition probabilities. Under very broad conditions the transition probabilities of a Markov process satisfy a finite or infinite system of linear homogeneous ordinary differential equations.

The theory of Markov processes is based on the studies of A. A. Markov (the elder), who in his works in 1907 set forth the foundations of the study of sequences of dependent tests and sums of random variables associated with them. This line of research is now known as the theory of Markov chains. The theory treats such systems that can pass from one state to another only at fully defined moments *t*_{1}, *t*_{2}, …, *t _{k}*, …. Let

*p*denote the probability that the system at moment

_{ij}*t*+ 1 will be in state oω

_{k}_{j}if it is known that at

*t*it was in state >

_{k}*i*. The investigation of Markov chains can be reduced to the study of the transition probability matrices ║

*P*║.

_{ij}A number of physicists and engineers have also demonstrated in their research the importance of processes in which a given system undergoes random variations as a function of a given number of continuously varying parameters (time, coordinates, and so on). Research along these lines lacked a solid logical foundation. The Soviet mathematician A. N. Kolmogorov gave a general theory and classification of Markov processes in 1930. His studies provided a logically faultless mathematical foundation of the general theory of Markov processes, encompassing, in addition to processes of the type described above, diffusion-type processes, in which the state of a system is characterized by a continuously varying coordinate of the diffusing particle.

In this case, it is natural to replace the transition probabilities by the corresponding probability density *f* (*t,x,y*). Then *f* (*t,x,y* is the probability that a particle at the point *x* will have a coordinate between *y* and *y* + *dy* after time *t*. Kolmogorov proved that under certain general conditions the densities *f* (*t,x,y* satisfy the partial differential equation

which had previously been introduced for the physically important special case of a diffusion process by the German physicists A. Fokker and M. Planck. In this equation, the coefficient *A* (*y*) is the mean rate of variation of the coordinate *y* and the coefficient *B* (*y*) is the rate of the random variations about the mean. This equation lies at the foundations of many investigations into the theory of Markov processes in the USSR and abroad.

### REFERENCES

Markov, A. A.*Izbr. trudy: Teoriia chisel, teoriia veroiatnostei*. Moscow, 1951.

Kolmogorov, A N. “Ob analiticheskikh metodakh v teorii veroiatnostei.”

*Uspekhi matematicheskikh nauk*, 1938, fasc. 5.

Feller, W.

*Vvedenie v teoriiu veroiatnostei i ee prilozheniia*, vols. 1-2. Moscow, 1967. (Translated from English.)

Gikhman, I. I., and A. V. Skorokhod.

*Vvedenie v teoriiu sluchainykh protsessov*. Moscow, 1965.

B. A. SEVAST’IANOV and S. KH. SIRAZHDINOV