Limits to classification and regression estimation from ergodic processes
Andrew B. Nobel
Source: Ann. Statist. Volume 27, Number 1 (1999), 262-273.
Abstract
We answer two open questions concerning the existence of universal schemes for classification and regression estimation from stationary ergodic processes. It is shown that no measurable procedure can produce weakly consistent regression estimates from every bivariate stationary ergodic process, even if the covariate and response variables are restricted to take values in the unit interval. It is further shown that no measurable procedure can produce weakly consistent classification rules from every bivariate stationary ergodic process for which the response variable is binary valued. The results of the paper are derived via reduction arguments and are based in part on recent work concerning density estimaton from ergodic processes.
Primary Subjects: 62G07
Secondary Subjects: 60G10, 62M99
Keywords: Classification; regression; ergodic processes; counterexamples; reduction arguments
Full-text: Open access
Links and Identifiers
Permanent link to this document: http://projecteuclid.org/euclid.aos/1018031110
Mathematical Reviews number (MathSciNet):
MR1701110
Digital Object Identifier: doi:10.1214/aos/1018031110
Zentralblatt MATH identifier:
0933.62033
References
Adams, T. M. (1997). Families of ergodic processes without consistent density or regression estimates. Preprint.
Adams, T. M. and Nobel, A. B. (1998). On density estimation from ergodic processes. Ann. Probab. 26 794-804.
Mathematical Reviews (MathSciNet):
MR99m:62038
Zentralblatt MATH:
0932.62042
Algoet, P. (1992). Universal schemes for prediction, gambling and portfolio selection. Ann.
Mathematical Reviews (MathSciNet):
MR93d:60075
Zentralblatt MATH:
0758.90006
Probab. 20 901-941. Correction: [Ann. Probab. 23 474-478 (1995).]
Ash, R. (1972). Real Analysis and Probability. Academic Press, New York.
Mathematical Reviews (MathSciNet):
MR55:8280
Bailey, D. H. (1976). Sequential schemes for classifying and predicting ergodic processes. Ph.D. dissertation, Dept. Mathematics, Stanford Univ.
Barron, A. R. (1985). Logically smooth density estimation. Technical Report TR 56, Dept. Statistics, Stanford Univ.
Cheng, B. C. and Robinson, P. M. (1991). Density estimation in strongly dependent non-linear time series. Statist. Sinica 1 335-359.
Delecroix, M. (1987). Sur l'estimation et la pr´evision non-param´etrique des processus ergodiques. Ph.D. thesis, Univ. Lille Flandres Artois, Lille, France.
Delecroix, M. and Rosa, A. C. (1996). Nonparametric estimation of a regression function and its derivatives under an ergodic hypothesis. J. Nonparametric Statist. 6 367-382.
Mathematical Reviews (MathSciNet):
MR98k:62050
Zentralblatt MATH:
0879.62034
Devroye, L. and Wagner, T. (1980). Distribution-free consistency results in nonparametric discrimination and regression function estimation. Ann. Statist. 8 231-239.
Mathematical Reviews (MathSciNet):
MR81k:62051
Zentralblatt MATH:
0431.62025
Devroye, L., Gy ¨orfi, L. and Lugosi, G. (1996). A Probabalistic Theory of Pattern Recognition. Springer, New York.
Gy ¨orfi, L. (1981). Strongly consistent density estimate from ergodic sample. J. Multivariate Anal. 11 81-84.
Gy ¨orfi, L., H¨ardle, W., Sarda, P. and Vieu. P. (1989). Nonparametric Curve Estimation from Time Series. Springer, Berlin.
Mathematical Reviews (MathSciNet):
MR91a:62104
Gy ¨orfi, L. and Lugosi, G. (1992). Kernel density estimation from ergodic sample is not universally consistent. Comput. Statist. Data Anal. 14 437-442.
Mathematical Reviews (MathSciNet):
MR93m:62076
Gy ¨orfi, L., Morvai, G. and Yakowitz, S. (1998). Limits to consistent on-line forecasting for ergodic time series. IEEE Trans. Inform. Theory 44 886-892.
Mathematical Reviews (MathSciNet):
MR98i:62072
Hidalgo, J. (1997). Non-parametric estimation with strongly dependent time multivariate time series. J. Time Ser. Anal. 18 95-122.
Zentralblatt MATH:
0923.62045
Ho, H. C. (1995). On the strong uniform consistency of density estimation for strongly dependent sequences. Statist. Probab. Lett. 22 149-156.
Mathematical Reviews (MathSciNet):
MR96g:62070
Zentralblatt MATH:
0810.62039
Masry, E. (1996). Multivariate local polynomial regression for time series: uniform strong consistency and rates. J. Time Ser. Anal. 17 571-599.
Mathematical Reviews (MathSciNet):
MR98a:62117
Morvai, G., Yakowitz, S. and Algoet, P. (1997). Weakly convergent nonparametric forecasting of stationary time series. IEEE Trans. Inform. Theory 43 483-498.
Mathematical Reviews (MathSciNet):
MR98g:60070
Zentralblatt MATH:
0871.62082
Morvai, G., Yakowitz, S. and Gy ¨orfi, L. (1996). Nonparametric inference for ergodic, stationary time series. Ann. Statist. 24 370-379.
Mathematical Reviews (MathSciNet):
MR97g:62071
Zentralblatt MATH:
0855.62076
Morvai, G., Kulkarni, S. and Nobel, A. B. (1997). Regression estimation from an individual stationary sequence. Unpublished manuscript.
Nobel, A. B., Morvai, G. and Kulkarni, S. (1997). Density estimation from an individual numerical sequence. IEEE Trans. Inform. Theory 44 537-541.
Ornstein, D. S. (1978). Guessing the next output of a stationary process. Israel J. Math. 30 292-296.
Mathematical Reviews (MathSciNet):
MR80b:28024
Zentralblatt MATH:
0386.60032
Rosenblatt, M. (1970). Density estimates and Markov sequences. In Nonparametric Techniques in Statistical Inference (M. Puri, ed.) 199-210. Cambridge Univ. Press, London.
Mathematical Reviews (MathSciNet):
MR43:4198
Rosenblatt, M. (1991). Stochastic Curve Estimation. IMS, Hayward, CA.
Roussas, G. (1967). Nonparametric estimation in Markov processes. Ann. Inst. Statist. Math. 21 73-87.
Roussas, G. (1969). Nonparametric estimation of the transition distribution function of a Markov process. Ann. Math. Statist. 40 1386-1400.
Mathematical Reviews (MathSciNet):
MR39:7775
Zentralblatt MATH:
0188.50501
Ryabko, B. Ya. (1988). Prediction of random sequences and universal coding. Problems Inform. Transmission 24 87-96.
Mathematical Reviews (MathSciNet):
MR89k:94034
Zentralblatt MATH:
0666.94009
Spiegelman, C. and Sacks, J. (1980). Consistent window estimation in nonparametric regression. Ann. Statist. 8 240-246.
Mathematical Reviews (MathSciNet):
MR81f:62070
Zentralblatt MATH:
0432.62066
Stone, C. (1977). Consistent nonparametric regression. Ann. Statist. 5 595-620.
Mathematical Reviews (MathSciNet):
MR56:1574
Zentralblatt MATH:
0366.62051
Yakowitz, S. (1993). Nearest neighbor regression estimation for null-recurrent Markov time series. Stochastic Process. Appl. 48 311-318.
Mathematical Reviews (MathSciNet):
MR95c:62056
Zentralblatt MATH:
0791.62084
Yakowitz, S., Gy ¨orfi, L., Kieffer, J. and Morvai, G. (1997). Strongly consistent nonparametric estimation of smooth regression functions for stationary ergodic sequences. J. Multivariate Anal. To appear.
Yakowitz, S. and Heyde, C. (1997). Long range dependency effects with implications for forecasting and queueing inference. Unpublished manuscript.
The Annals of Statistics
