The Annals of Statistics

On risk bounds in isotonic and other shape restricted regression problems

Sabyasachi Chatterjee, Adityanand Guntuboyina, and Bodhisattva Sen

Full-text: Open access

Abstract

We consider the problem of estimating an unknown $\theta\in\mathbb{R}^{n}$ from noisy observations under the constraint that $\theta$ belongs to certain convex polyhedral cones in $\mathbb{R}^{n}$. Under this setting, we prove bounds for the risk of the least squares estimator (LSE). The obtained risk bound behaves differently depending on the true sequence $\theta$ which highlights the adaptive behavior of $\theta$. As special cases of our general result, we derive risk bounds for the LSE in univariate isotonic and convex regression. We study the risk bound in isotonic regression in greater detail: we show that the isotonic LSE converges at a whole range of rates from $\log n/n$ (when $\theta$ is constant) to $n^{-2/3}$ (when $\theta$ is uniformly increasing in a certain sense). We argue that the bound presents a benchmark for the risk of any estimator in isotonic regression by proving nonasymptotic local minimax lower bounds. We prove an analogue of our bound for model misspecification where the true $\theta$ is not necessarily nondecreasing.

Article information

Source
Ann. Statist., Volume 43, Number 4 (2015), 1774-1800.

Dates
Received: May 2014
Revised: February 2015
First available in Project Euclid: 17 June 2015

Permanent link to this document
https://projecteuclid.org/euclid.aos/1434546222

Digital Object Identifier
doi:10.1214/15-AOS1324

Mathematical Reviews number (MathSciNet)
MR3357878

Zentralblatt MATH identifier
1317.62032

Subjects
Primary: 62G08: Nonparametric regression 62C20: Minimax procedures

Keywords
Adaptation convex polyhedral cones global risk bounds local minimax bounds model misspecification statistical dimension

Citation

Chatterjee, Sabyasachi; Guntuboyina, Adityanand; Sen, Bodhisattva. On risk bounds in isotonic and other shape restricted regression problems. Ann. Statist. 43 (2015), no. 4, 1774--1800. doi:10.1214/15-AOS1324. https://projecteuclid.org/euclid.aos/1434546222


Export citation

References

  • [1] Amelunxen, D., Lotz, M., McCoy, M. B. and Tropp, J. A. (2013). Living on the edge: A geometric theory of phase transitions in convex optimization. Available at arXiv:1303.6672.
  • [2] Ayer, M., Brunk, H. D., Ewing, G. M., Reid, W. T. and Silverman, E. (1955). An empirical distribution function for sampling with incomplete information. Ann. Math. Statist. 26 641–647.
  • [3] Barron, A., Birgé, L. and Massart, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301–413.
  • [4] Bertsekas, D. P. (2003). Convex Analysis and Optimization. Athena Scientific, Belmont, MA.
  • [5] Birgé, L. (1989). The Grenander estimator: A nonasymptotic approach. Ann. Statist. 17 1532–1549.
  • [6] Birgé, L. and Massart, P. (1993). Rates of convergence for minimum contrast estimators. Probab. Theory Related Fields 97 113–150.
  • [7] Brunk, H. D. (1955). Maximum likelihood estimates of monotone parameters. Ann. Math. Statist. 26 607–616.
  • [8] Brunk, H. D. (1970). Estimation of isotonic regression. In Nonparametric Techniques in Statistical Inference (Proc. Sympos., Indiana Univ., Bloomington, Ind., 1969) 177–197. Cambridge Univ. Press, London.
  • [9] Cai, T. and Low, M. (2011). A framework for estimation of convex functions. Available at http://www-stat.wharton.upenn.edu/~tcai/.
  • [10] Carolan, C. and Dykstra, R. (1999). Asymptotic behavior of the Grenander estimator at density flat regions. Canad. J. Statist. 27 557–566.
  • [11] Cator, E. (2011). Adaptivity and optimality of the monotone least-squares estimator. Bernoulli 17 714–735.
  • [12] Chandrasekaran, V., Recht, B., Parrilo, P. A. and Willsky, A. S. (2012). The convex geometry of linear inverse problems. Found. Comput. Math. 12 805–849.
  • [13] Chatterjee, S., Guntuboyina, A. and Sen, B. (2015). Supplement to “On risk bounds in isotonic and other shape restricted regression problems.” DOI:10.1214/15-AOS1324SUPP.
  • [14] Donoho, D. (1991). Gel’fand $n$-widths and the method of least squares. Technical report, Dept. Statistics, Univ. California, Berkeley, CA.
  • [15] Durrett, R. (2010). Probability: Theory and Examples, 4th ed. Cambridge Univ. Press, Cambridge.
  • [16] Grenander, U. (1956). On the theory of mortality measurement. II. Skand. Aktuarietidskr. 39 125–153.
  • [17] Groeneboom, P. (1983). The concave majorant of Brownian motion. Ann. Probab. 11 1016–1027.
  • [18] Groeneboom, P. (1985). Estimating a monotone density. In Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer, Vol. II (Berkeley, Calif., 1983). 539–555. Wadsworth, Belmont, CA.
  • [19] Guntuboyina, A. and Sen, B. (2014). Global risk bounds and adaptation in univariate convex regression. Probab. Theory Related Fields 1–33.
  • [20] Jankowski, H. (2014). Convergence of linear functionals of the Grenander estimator under misspecification. Ann. Statist. 42 625–653.
  • [21] Meyer, M. and Woodroofe, M. (2000). On the degrees of freedom in shape-restricted regression. Ann. Statist. 28 1083–1104.
  • [22] Oymak, S. and Hassibi, B. (2013). Sharp MSE bounds for proximal denoising. Available at arXiv:1305.2714.
  • [23] Rigollet, P. and Tsybakov, A. B. (2012). Sparse estimation by exponential weighting. Statist. Sci. 27 558–575.
  • [24] Robertson, T., Wright, F. T. and Dykstra, R. L. (1988). Order Restricted Statistical Inference. Wiley, Chichester.
  • [25] Rudelson, M. and Vershynin, R. (2006). Sparse reconstruction by convex relaxation: Fourier and Gaussian measurements. In 40th Annual Conference on Information Sciences and Systems 207–212. IEEE, New York.
  • [26] Schrijver, A. (1986). Theory of Linear and Integer Programming. Wiley, Chichester.
  • [27] Stojnic, M. (2009). Various thresholds for $\ell_{1}$-optimization in compressed sensing. Available at arXiv:0907.3666.
  • [28] Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation. Springer, New York.
  • [29] van de Geer, S. (1990). Estimating a regression function. Ann. Statist. 18 907–924.
  • [30] van de Geer, S. (1993). Hellinger-consistency of certain nonparametric maximum likelihood estimators. Ann. Statist. 21 14–44.
  • [31] van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics 3. Cambridge Univ. Press, Cambridge.
  • [32] Vershynin, R. (2014). Lectures in geometric functional analysis. Available at www-personal.umich.edu/~romanv.
  • [33] Wang, Y. (1996). The $L_{2}$ risk of an isotonic estimate. Comm. Statist. Theory Methods 25 281–294.
  • [34] Wright, F. T. (1981). The asymptotic behavior of monotone regression estimates. Ann. Statist. 9 443–448.
  • [35] Zhang, C.-H. (2002). Risk bounds in isotonic regression. Ann. Statist. 30 528–555.

Supplemental materials

  • Supplement to “On risk bounds in isotonic and other shape restricted regression problems”. In the supplementary paper [13] we provide the proofs of Lemmas 2.4, 2.5, 2.6, 6.3 and 6.4 and Theorems 4.1, 4.2, 4.3, 5.3 and 6.1. We also state and prove Lemma 11.1, which is used in the proof of Theorem 4.1, and Lemma 11.2, which is used in the proof of Theorem 3.1.