Electronic Journal of Statistics

Recursive construction of confidence regions

Tomasz Bielecki, Tao Chen, and Igor Cialenco

Full-text: Open access


Assuming that one-step transition kernel of a discrete time, time-homogenous Markov chain model is parameterized by a parameter $\theta\in\boldsymbol{\Theta}$, we derive a recursive (in time) construction of confidence regions for the unknown parameter of interest, say $\theta^{*}\in\boldsymbol{\Theta}$. It is supposed that the observed data used in the construction of the confidence regions is generated by a Markov chain whose transition kernel corresponds to $\theta^{*}$. The key step in our construction is the derivation of a recursive scheme for an appropriate point estimator of $\theta^{*}$. To achieve this, we start by what we call the base recursive point estimator, using which we design a quasi-asymptotically linear recursive point estimator (a concept introduced in this paper). For the latter estimator we prove its weak consistency and asymptotic normality. The recursive construction of confidence regions is needed not only for the purpose of speeding up the computation of the successive confidence regions, but, primarily, for the ability to apply the dynamic programming principle in the context of robust adaptive stochastic control methodology.

Article information

Electron. J. Statist., Volume 11, Number 2 (2017), 4674-4700.

Received: May 2017
First available in Project Euclid: 18 November 2017

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62M05: Markov processes: estimation 62F10: Point estimation 62F12: Asymptotic properties of estimators 62F25: Tolerance and confidence regions 60J05: Discrete-time Markov processes on general state spaces 60J20: Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.) [See also 90B30, 91D10, 91D35, 91E40]

Recursive confidence regions stochastic approximation recursive point estimators statistical inference for Markov chains ergodic processes quasi-asymptotically linear estimator

Creative Commons Attribution 4.0 International License.


Bielecki, Tomasz; Chen, Tao; Cialenco, Igor. Recursive construction of confidence regions. Electron. J. Statist. 11 (2017), no. 2, 4674--4700. doi:10.1214/17-EJS1362. https://projecteuclid.org/euclid.ejs/1510974130

Export citation


  • [1] D. J. Aldous and G. K. Eagleson. On Mixing and Stability of Limit Theorems., The Annals of Probability, 6(2):325–331, April 1978.
  • [2] T. R. Bielecki, T. Chen, I. Cialenco, A. Cousin, and M. Jeanblanc. Adaptive Robust Control Under Model Uncertainty. Preprint, 2017.
  • [3] R. Buche and H. J. Kushner. Rate of convergence for constrained stochastic approximation algorithms., SIAM J. Control Optim., 40(4) :1011–1041, 2001/02.
  • [4] I. Crimaldi and L. Pratelli. Convergence Result for Multivariate Martingales., Stoch. Proc. and their Applic., 115:571–577, 2005.
  • [5] V. Fabian. On Asymptotic Normality in stochastic approximation., Ann. Math. Statist., 39(4) :1327–1332, 1968.
  • [6] R. A. Fisher. Theory of statistical estimation., Proc. Cambridge Philos. Soc., 22:700–725, 1925.
  • [7] E. Häusler and H. Luschgy., Stable Convergence and Stable Limit Theorems. Springer, 2015.
  • [8] O. Hernández-Lerma and J. Lasserre. Further criteria for positive Harris recurrence of Markov chains., Proceedings of the American Mathematical Society, 129(5) :1521–1524, 2000.
  • [9] H. Kushner and D. Clark., Stochastic Approximation Methods for Constrained and Unconstrained Systems, volume 26 of Applied Mathematical Sciences. Springer-Verlag New York, 1978.
  • [10] S. Kullback and R. A. Leibler. On information and sufficiency., Annals of Mathematical Statistics, 22(1):79–86, 1951.
  • [11] J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function., The Annals of Mathematical Statistics, 23(3):462–466, 1952.
  • [12] H. Kushner and G. Yin., Stochastic Approximation and Recursive Algorithms and Applications, volume 35 of Stochastic Modelling and Applied Probability. Springer-Verlag New York, 2003.
  • [13] L. LeCam. On the asymptotic theory of estimation and testing hypothesis., Proccedings in the Third Berkely Symposium on Mathematical Statist. and Probab., 1:129–156, 1956.
  • [14] L. LeCam. Locally asymptotic normal families of distributions., Univ. California Publ. in Statist., 3(2):37–98, 1960.
  • [15] T. L. Lai and H. Robbins. Adaptive design and stochastic approximation., Ann. Statist., 7(6) :1196–1221, 1979.
  • [16] L. Ljung and Söderström., Theory and Practice of Recursive Identification, volume 4 of The MIT Press Series in Signal Processing, Optimization, and Control. MIT Press, 1987.
  • [17] S. Meyn and R. Tweedie., Markov Chains and Stochastic Stability. Cambridge Mathematical Library. Cambridge University Press, 1993.
  • [18] D. Revuz., Markov Chains, volume 11 of North-Holland Mathematical Library. Elsevier Science Publishers B.V., revised edition edition, 1984.
  • [19] H. Robbins and S. Monro. A stochastic approximation method., The Annals of Mathematical Statistics, 22(3):400–407, 1951.
  • [20] J. Sacks. Asymptotic distribution of stochastic approximation procedures., Ann. Math. Statist., 29(2):373–405, 1958.
  • [21] T. Sharia. Recursive parameter estimation: asymptotic expansion., Annals of the Institute of Statistical Mathematics, 62(2):343–362, 2010.
  • [22] A. Shiryayev., Probability, volume 95 of Graduate Texts in Mathematics. Springer New York, 1984.
  • [23] T. Sharia and L. Zhong. Rate of convergence of truncated stochastic approximation procedures with moving bounds., Mathematical Methods of Statistics, 25(4):262–280, 2016.
  • [24] G. Yin. A stopping rule for least-squares identification., IEEE Trans. Automat. Control, 34(6), 1989.
  • [25] G. Yin. A stopping rule for the robbins-monro method., Journal of Optimization Theory and Applications, 67:151–173, 1990.