Stochastic Systems

Detecting Markov Chain Instability: A Monte Carlo Approach

M. Mandjes, B. Patch, and N. S. Walton

Full-text: Open access


We devise a Monte Carlo based method for detecting whether a non-negative Markov chain is stable for a given set of parameter values. More precisely, for a given subset of the parameter space, we develop an algorithm that is capable of deciding whether the set has a subset of positive Lebesgue measure for which the Markov chain is unstable. The approach is based on a variant of simulated annealing, and consequently only mild assumptions are needed to obtain performance guarantees.

The theoretical underpinnings of our algorithm are based on a result stating that the stability of a set of parameters can be phrased in terms of the stability of a single Markov chain that searches the set for unstable parameters. Our framework leads to a procedure that is capable of performing statistically rigorous tests for instability, which has been extensively tested using several examples of standard and non-standard queueing networks.

Article information

Stoch. Syst., Volume 7, Number 2 (2017), 289-314.

Received: September 2016
Accepted: July 2017
First available in Project Euclid: 24 February 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 60K25: Queueing theory [See also 68M20, 90B22] 90B22: Queues and service [See also 60K25, 68M20] 68W40: Analysis of algorithms [See also 68Q25] 68M20: Performance evaluation; queueing; scheduling [See also 60K25, 90Bxx]

Markov chains stability Monte Carlo algorithm queueing networks stochastic networks

Creative Commons Attribution 4.0 International License.


Mandjes, M.; Patch, B.; Walton, N. S. Detecting Markov Chain Instability: A Monte Carlo Approach. Stoch. Syst. 7 (2017), no. 2, 289--314. doi:10.1287/stsy.2017.0003.

Export citation


  • Andrews, M., Zhang, L. (2003) Achieving stability in networks of input-queued switches. Networking, IEEE/ACM Trans. 11(5):848–857.
  • Baccelli, F., Bonald, T. (1999) Window flow control in FIFO networks with cross traffic. Queueing Systems 32(1):195–231.
  • Baskett, F., Chandy, KM., Muntz, RR., Palacios, FG. (1975) Open, closed, and mixed networks of queues with different classes of customers. J. ACM 22(2):248–260.
  • Bordenave, C., McDonald, D., Proutière, A. (2012) Asymptotic stability region of slotted ALOHA. Inform. Theory, IEEE Trans. 58(9):5841–5855.
  • Bramson, M. (1994) Instability of FIFO queueing networks. Ann. Appl. Probab. 4(2):414–431.
  • Bramson, M. (2008) Stability of Queueing Networks (Springer, New York).
  • Dai, J. (1995) On positive Harris recurrence of multiclass queueing networks: A unified approach via fluid limit models. Ann. Appl. Probab. 5(1):49–77.
  • Ghaderi, J., Borst, S., Whiting, P. (2014) Queue-based random-access algorithms: Fluid limits and stability issues. Stochastic Systems 4(1):81–156.
  • Hairer, M. (2010) Convergence of Markov processes. Lecture Notes,
  • Jackson, JR. (1963) Jobshop-like queueing systems. Management Sci. 10(1):131–142.
  • Kelly, FP. (1975) Networks of queues with customers of different types. J. Appl. Probab. 12(3):542–554.
  • Kelly, FP. (1979) Reversibility and Stochastic Networks (John Wiley & Sons, Chichester, UK).
  • Kirkpatrick, S., Vecchi, MP. (1983) Optimization by simulated annealing. Science 220(4598):671–680.
  • Kumar, PR., Seidman, TI. (1990) Dynamic instabilities and stabilization methods in distributed real-time scheduling of manufacturing systems. Automatic Control, IEEE Trans. 35(3):289–298.
  • Leahu, H., Mandjes, M. (2016) A numerical approach to stability of multiclass queueing networks. Preprint arXiv:1606.07294.
  • Lu, SH., Kumar, PR. (1991) Distributed scheduling based on due dates and buffer priorities. Automatic Control, IEEE Trans. 36(12):1406–1416.
  • MacPhee, I., Menshikov, M., Petritis, D., Popov, S. (2007) A Markov chain model of a polling system with parameter regeneration. Ann. Appl. Probab. 17(5/6):1447–1473.
  • McKeown, N., Mekkittikul, A., Anantharam, V., Walrand, J. (1999) Achieving 100% throughput in an input-queued switch. Commun., IEEE Trans. 47(8):1260–1267.
  • Meyn, S., Tweedie, R. (2012) Markov Chains and Stochastic Stability. Communications and Control Engineering (Springer, London).
  • Nazarathy, Y., Taimre, T., Asanjarani, A., Kuhn, J., Patch, B., Vuorinen, A. (2015) The challenge of stabilizing control for queueing systems with unobservable server states. Proc. 5th Australian Control Conf., AUCC ’15 (IEEE, Piscataway, NJ), 342–347.
  • Rybko, A., Stolyar, AL. (1993) Ergodicity of stochastic processes describing the operation of open queueing networks. Problemy Peredachi Informatsii 28(3):3–26.
  • Tassiulas, L., Ephremides, A. (1992) Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks. Automatic Control, IEEE Trans. 37(12):1936–1948.
  • Tassiulas, L., Ephremides, A. (1993) Dynamic server allocation to parallel queues with randomly varying connectivity. Inform. Theory, IEEE Trans. 39(2):466–478.
  • Wieland, JR., Pasupathy, R., Schmeiser, BW. (2003) Queueing network simulation analysis: queueing-network stability: Simulation-based checking. Chick, SE., Sanchez, PJ., Ferrin, DM., Morrice, DJ., eds. Proc. 35th Winter Simulation Conf.: Driving Innovation, WSC ’03 (ACM, New York), 520–527.
  • Williams, D. (1991) Probability with Martingales (Cambridge University Press, Cambridge, UK).