The Annals of Statistics

Minimax estimation via wavelet shrinkage

David L. Donoho and Iain M. Johnstone

Full-text: Open access

Abstract

We attempt to recover an unknown function from noisy, sampled data. Using orthonormal bases of compactly supported wavelets, we develop a nonlinear method which works in the wavelet domain by simple nonlinear shrinkage of the empirical wavelet coefficients. The shrinkage can be tuned to be nearly minimax over any member of a wide range of Triebel- and Besov-type smoothness constraints and asymptotically mini-max over Besov bodies with $p \leq q$. Linear estimates cannot achieve even the minimax rates over Triebel and Besov classes with $p<2$, so the method can significantly outperform every linear method (e.g., kernel, smoothing spline, sieve in a minimax sense). Variants of our method based on simple threshold nonlinear estimators are nearly minimax. Our method possesses the interpretation of spatial adaptivity; it reconstructs using a kernel which may vary in shape and bandwidth from point to point, depending on the data. Least favorable distributions for certain of the Triebel and Besov scales generate objects with sparse wavelet transforms. Many real objects have similarly sparse transforms, which suggests that these minimax results are relevant for practical problems. Sequels to this paper, which was first drafted in November 1990, discuss practical implementation, spatial adaptation properties, universal near minimaxity and applications to inverse problems.

Article information

Source
Ann. Statist. Volume 26, Number 3 (1998), 879-921.

Dates
First available in Project Euclid: 21 June 2002

Permanent link to this document
http://projecteuclid.org/euclid.aos/1024691081

Digital Object Identifier
doi:10.1214/aos/1024691081

Mathematical Reviews number (MathSciNet)
MR1635414

Zentralblatt MATH identifier
0935.62041

Subjects
Primary: 62G07 62C20: Minimax procedures
Secondary: 62G20 41A30: Approximation by other special function classes

Keywords
Minimax decision theory minimax Bayes estimation Besov Hölder Sobolev Triebel spaces nonlinear estimation white noise model nonparametric regression, orthonormal bases of compactly supported wavelets renormalization white noise approximation

Citation

Donoho, David L.; Johnstone, Iain M. Minimax estimation via wavelet shrinkage. Ann. Statist. 26 (1998), no. 3, 879--921. doi:10.1214/aos/1024691081. http://projecteuclid.org/euclid.aos/1024691081.


Export citation

References

  • BERGH, J. and LOFSTROM, J. 1976. Interpolation spaces: An Introduction. Springer, New York. ¨ ¨ Z.
  • BICKEL, P. J. 1983. Minimax estimation of a normal mean subject to doing well at a point. In Z. Recent Advances in Statistics M. H. Rizvi, J. S. Rustagi and D. Siegmund, eds. 511 528. Academic Press, New York. Z.
  • BIRGE, L. and MASSART, P. 1997. From model selection to adaptive estimation. In Festschrift for ´ Z Lucien Le Cam: Research Papers in Probability and Statistics E. T. D. Pollard and G.. Yang, eds. 55 87. Springer, New York. Z.
  • BREIMAN, L., FRIEDMAN, J., OLSHEN, R. and STONE, C. 1983. CART: Classification and Regression Trees. Wadsworth, Belmont, CA. Z.
  • BROWN, L. D. and LOW, M. G. 1996. Asy mptotic equivalence of nonparametric regression and white noise. Ann. Statist. 24 2384 2398. Z.
  • CHUI, C. K. 1992. An Introduction to Wavelets. Academic Press, San Diego. Z.
  • COHEN, A., DAUBECHIES, I., JAWERTH, B. and VIAL, P. 1993. Multiresolution analysis, wavelets, and fast algorithms on an interval. C. R. Acad. Sci. Paris Ser. I Math. 316 417 421. ´ Z.
  • COHEN, A., DAUBECHIES, I. and VIAL, P. 1993. Wavelets and fast wavelet transform on an interval. Appl. Comput. Harmon. Anal. 1 54 81. Z.
  • DAUBECHIES, I. 1988. Orthonormal bases of compactly supported wavelets. Comm. Pure Appl. Math. 41 909 996. Z.
  • DAUBECHIES, I. 1992. Ten Lectures on Wavelets. SIAM, Philadelphia. Z.
  • DESLAURIERS, G. and DUBUC, S. 1987. Interpolation dy adique. In Fractals, Dimensions non-entieres et applications. Masson, Paris. Z.
  • DESLAURIERS, G. and DUBUC, S. 1989. Sy mmetric iterative interpolation processes. Constr. Approx. 5 49 68.Z.
  • DEVORE, R. and POPOV, V. 1988. Interpolation of Besov spaces. Trans. Amer. Math. Soc. 305 397 414. Z.
  • DONOHO, D. 1992. De-noising via soft-thresholding. IEEE Trans. Inform. Theory 41 613 627. Z.
  • DONOHO, D. 1994. Asy mptotic minimax risk for sup-norm loss; solution via optimal recovery. Probab. Theory Related Fields 99 145 170. Z.
  • DONOHO, D. 1995. Nonlinear solution of linear inverse problems by wavelet-vaguelette decomposition. Appl. Comput. Harmon. Anal. 2 101 126. Z.
  • DONOHO, D. L. and JOHNSTONE, I. M. 1989. Minimax risk over l -balls. Technical Report 322, p Dept. Statistics, Stanford Univ. Z.
  • DONOHO, D. L. and JOHNSTONE, I. M. 1994a. Ideal spatial adaptation via wavelet shrinkage. Biometrika 81 425 455. Z.
  • DONOHO, D. L. and JOHNSTONE, I. M. 1994b. Minimax risk over l -balls for l -error. Probab. p q Theory Related Fields 99 277 303. Z.
  • DONOHO, D. L. and JOHNSTONE, I. M. 1995. Adapting to unknown smoothness via wavelet shrinkage. J. Amer. Statist. Assoc. 90 1200 1224. Z.
  • DONOHO, D. L. and JOHNSTONE, I. M. 1997. Asy mptotic minimaxity of wavelet estimators with sampled data. Technical Report, Dept. Statistics, Stanford Univ. Z.
  • DONOHO, D. L., JOHNSTONE, I. M., KERKy ACHARIAN, G. and PICARD, D. 1995. Wavelet shrinkage: Z. Asy mptopia? with discussion. J. Roy. Statist. Soc. Ser. B 57 301 369. Z.
  • DONOHO, D. L., JOHNSTONE, I. M., KERKy ACHARIAN, G. and PICARD, D. 1996. Density estimation by wavelet thresholding. Ann. Statist. 24 508 539.
  • DONOHO, D. L., LIU, R. C. and MACGIBBON, K. B. 1990. Minimax risk over hy perrectangles, and implications. Ann. Statist. 18 1416 1437. Z.
  • DONOHO, D. L. and NUSSBAUM, M. 1990. Minimax quadratic estimation of a quadratic functional. J. Complexity 6 290 323. Z.
  • DUBUC, S. 1986. Interpolation through an iterative scheme. J. Math. Anal. Appl. 114 185 204. Z.
  • EFROIMOVICH, S. and PINSKER, M. 1981. Estimation of square-integrable density on the basis of Z a sequence of observations. Problems Inform. Transmission 17 182 195. In Russian. in Problemy Peredatsii Informatsii 17 50 68. Z.
  • EFROIMOVICH, S. and PINSKER, M. 1982. Estimation of square-integrable probability density of a Z random variable. Problems Inform. Transmission 18 175 189. In Russian in Prob. lemy Peredatsii Informatsii 18 19 38. Z.
  • FEICHTINGER, H. and GROCHENIG, K. 1989. Banach spaces related to integrable group represen¨ tations and their atomic decompositions. I. J. Funct. Anal. 86 307 340. Z.
  • FRAZIER, M. and JAWERTH, B. 1985. Decomposition of Besov spaces. Indiana Univ. Math. J. 34 777 799. Z.
  • FRAZIER, M. and JAWERTH, B. 1986. The -transform and applications to distribution spaces. Function Spaces and Applications. Lecture Notes in Math. 1302 223 246. Springer, Berlin. Z.
  • FRAZIER, M. and JAWERTH, B. 1990. A discrete transform and decomposition of distribution spaces. J. Funct. Anal. 93 34 170. Z.
  • FRAZIER, M., JAWERTH, B. and WEISS, G. 1991. Littlewood-Paley Theory and the Study of Function Spaces. Amer. Math. Soc., Providence, RI. Z.
  • GROCHENIG, K. 1988. Unconditional bases in translationand dilation-invariant function spaces ¨ n Z. on R. In Constructive Theory of Functions, Conference Varna B. Sendov, ed. 174 183. Bulgarian Acad. Sci., Sofia. Z.
  • IBRAGIMOV, I. A. and KHAS'MINSKII, R. Z. 1981. Statistical Estimation: Asy mptotic Theory. Springer, New York. Z.
  • IBRAGIMOV, I. A. and KHAS'MINSKII, R. Z. 1982. Bounds for the risks of non-parametric regression estimates. Theory Probab. Appl. 27 84 99. Z.
  • JAFFARD, S. 1989. Estimation holderiennes ponctuelle des fonctions au moy en des coefficients ¨ d'ondelettes. C. R. Acad. Sci. Paris Ser. I Math. 308 79 81. ´ Z.
  • JOHNSTONE, I. M. 1994. Minimax Bay es, asy mptotic minimax and sparse wavelet priors. In Z. Statistical Decision Theory and Related Topics V S. Gupta and J. Berger, eds. 303 326. Springer, New York. Z.
  • JOHNSTONE, I., KERKy ACHARIAN, G. and PICARD, D. 1992. Estimation d'une densite de proba´ bilite par methode d'ondelettes. C. R. Acad. Sci. Paris Ser. A 315 211 216. ´ ´ Z.
  • KAISER, G. 1994. A Friendly Guide to Wavelets. Springer, New York. Z.
  • KERKy ACHARIAN, G. and PICARD, D. 1992. Density estimation in Besov spaces. Statist. Probab. Lett. 13 15 24. Z.
  • LE CAM, L. 1986. Asy mptotic Methods in Statistical Decision Theory. Springer, Berlin. Z.
  • LEMARIE, P. and MEy ER, Y. 1986. Ondelettes et bases Hilbertiennes. Rev. Mat. Iberoamericana ´ 2 1 18. Z.
  • LEPSKI, O., MAMMEN, E. and SPOKOINY, V. 1997. Optimal spatial adaptation to inhomogeneous smoothness; an approach based on kernel estimates with variable bandwidth selectors. Ann. Statist. 25 929 947. Z. 2 Z.
  • MALLAT, S. G. 1989a. Multiresolution approximation and wavelet orthonormal bases of L R. Trans. Amer. Math. Soc. 315 69 87. Z.
  • MALLAT, S. G. 1989b. Multifrequency channel decompositions of images and wavelet models. IEEE Trans. on Acoust. Signal Speech Process. 37 2091 2110. Z.
  • MALLAT, S. G. 1989c. A theory for multiresolution signal decomposition: the wavelet representation, IEEE Transactions on Pattern Analy sis and Machine Intelligence 11 674 693. Z. Z
  • MEy ER, Y. 1990a. Ondelettes et Operateurs I. Hermann, Paris. English translation published ´. by Cambridge Univ. Press.
  • MEy ER, Y. 1990b. Ondelettes et Operateurs II. Operateurs de Calderon-Zy gmund. Hermann, ´ ´ ´ Paris. Z.
  • MEy ER, Y. 1991. Ondelettes sur l'intervalle. Rev. Mat. Iberoamericana 7 115 133. Z.
  • MULLER, H.-G. and STADTMULLER, U. 1987. Variable bandwidth kernel estimators of regression ¨ curves. Ann. Statist. 15 182 201. Z.
  • NEMIROVSKII, A. 1985. Nonparametric estimation of smooth regression functions. Izv. Akad. Z. Nauk. SSR Teckhn. Kibernet. 3 50 60 in Russian. J. Comput. Sy stem Sci. 23 1 11 Z. Z. 1986 in English. Z.
  • NEMIROVSKII, A., POLy AK, B. and TSy BAKOV, A. 1985. Rate of convergence of nonparametric estimates of maximum-likelihood ty pe. Problems Inform. Transmission 21 258 272. Z.
  • NUSSBAUM, M. 1985. Spline smoothing in regression models and asy mptotic efficiency in l. 2 Ann. Statist. 13 984 997. Z.
  • PEETRE, J. 1975. New Thoughts on Besov Spaces I. Duke Univ. Z.
  • PINSKER, M. 1980. Optimal filtering of square integrable signals in Gaussian white noise. Z Problems Inform. Transmission 16 120 133. In Russian in Problemy Peredatsii. Informatsii 16 52 68. Z.
  • SPECKMAN, P. 1985. Spline smoothing and optimal rates of convergence in nonparametric regression models. Ann. Statist. 13 970 983. Z.
  • STONE, C. 1982. Optimal global rates of convergence for nonparametric estimators. Ann. Statist. 10 1040 1053. Z.
  • TRIEBEL, H. 1983. Theory of Function Spaces. Birkhauser, Basel. ¨ Z.
  • WALTER, G. 1994. Wavelets and Other Orthogonal Sy stems with Applications. Chemical Rubber Company, Boca Raton, FL.
  • BERKELEY, CALIFORNIA STANFORD, CALIFORNIA 94305-4065 E-MAIL: imj@donohoestat.stanford.edu