Statistical Science

From Unit Root to Stein’s Estimator to Fisher’s k Statistics: If You Have a Moment, I Can Tell You More

Xiao-Li Meng

Full-text: Open access

Abstract

Any general textbook that discusses moment generating functions (MGFs) shows how to obtain a moment of positive-integer order via differentiation, although usually the presented examples are only illustrative, because the corresponding moments can be calculated in more direct ways. It is thus somewhat unfortunate that very few textbooks discuss the use of MGFs when it becomes the simplest, and sometimes the only, approach for analytic calculation and manipulation of moments. Such situations arise when we need to evaluate the moments of ratios and logarithms, two of the most common transformations in statistics. Such moments can be obtained by differentiating and integrating a joint MGF of the underlying untransformed random variables in appropriate ways. These techniques are examples of multivariate Laplace transform methods and can also be derived from the fact that moments of negative orders can be obtained by integrating an MGF. This article reviews, extends and corrects various results scattered in the literature on this joint-MGF approach, and provides four applications of independent interest to demonstrate its power and beauty. The first application, which motivated this article, is for the exact calculation of the moments of a well-known limiting distribution under the unit-root AR(1) model. The second, which builds on Stigler’s Galtonian perspective, reveals a straightforward, non-Bayesian constructive derivation of the Stein estimator, as well as convenient expressions for studying its risk and bias. The third finds an exceedingly simple bound for the bias of a sample correlation from a bivariate normal population, namely the magnitude of the relative bias is not just of order n−1, but actually is bounded above by n−1 for all sample sizes n≥2. The fourth tackles the otherwise intractable problem of studying the finite-sample optimal bridge in the context of bridge sampling for computing normalizing constants. A by-product of the joint-MGF approach is that positive-order fractional moments can be easily obtained from an MGF without invoking the concept of fractional differentiation, a method used by R. A. Fisher in his study of k statistics 45 years before it reappeared in the probability literature.

Article information

Source
Statist. Sci., Volume 20, Number 2 (2005), 141-162.

Dates
First available in Project Euclid: 14 July 2005

Permanent link to this document
https://projecteuclid.org/euclid.ss/1121347637

Digital Object Identifier
doi:10.1214/088342304000000279

Mathematical Reviews number (MathSciNet)
MR2183446

Zentralblatt MATH identifier
1086.62001

Keywords
AR(1) model bias bridge sampling Efron–Morris estimator fractional derivative history of statistics James–Stein estimator Laplace transform normalizing constants R. A. Fisher unit root Wiener process

Citation

Meng, Xiao-Li. From Unit Root to Stein’s Estimator to Fisher’s k Statistics: If You Have a Moment, I Can Tell You More. Statist. Sci. 20 (2005), no. 2, 141--162. doi:10.1214/088342304000000279. https://projecteuclid.org/euclid.ss/1121347637


Export citation

References

  • Abadir, K. M. (1993). The limiting distribution of the autocorrelation coefficient under a unit root. Ann. Statist. 21 1058--1070.
  • Abadir, K. M. and Larsson, R. (1996). The joint moment generating function of quadratic forms in multivariate autoregressive series. Econometric Theory 12 682--704.
  • Abadir, K. M. and Larsson, R. (2001). The joint moment generating function of quadratic forms in multivariate autoregressive series: The case with deterministic components. Econometric Theory 17 222--246.
  • Bennett, C. H. (1976). Efficient estimation of free energy differences from Monte Carlo data. J. Computational Phys. 22 245--268.
  • Billingsley, P. (1995). Probability and Measure, 3rd ed. Wiley, New York.
  • Bock, M. E., Judge, G. G. and Yancey, T. A. (1984). A simple form for the inverse moments of non-central $\chi^2$ and $F$ random variables and certain confluent hypergeometric functions. J. Econometrics 25 217--234.
  • Bowman, K. O. and Shenton, L. R. (1992). Some exact expressions for the mean and higher moments of functions of sample moments. Ann. Inst. Statist. Math. 44 781--798.
  • Brandwein, A. C. and Strawderman, W. E. (1990). Stein estimation: The spherically symmetric case. Statist. Sci. 5 356--369.
  • Ceperley, D. M. (1995). Path integrals in the theory of condensed helium. Rev. Modern Phys. 67 279--355.
  • Chan, N. H. and Wei, C. Z. (1987). Asymptotic inference for nearly nonstationary AR(1) processes. Ann. Statist. 15 1050--1063.
  • Chao, M. T. and Strawderman, W. E. (1972). Negative moments of positive random variables. J. Amer. Statist. Assoc. 67 429--431.
  • Cressie, N. and Borkent, M. (1986). The moment generating function has its moments. J. Statist. Plann. Inference 13 337--344.
  • Cressie, N., Davis, A. S., Folks, J. L. and Policello, G. E. (1981). The moment-generating function and negative integer moments. Amer. Statist. 35 148--150.
  • Davies, N., Pate, M. B. and Petruccelli, J. D. (1985). Exact moments of the sample cross correlations of multivariate autoregressive moving average time series. Sankhyā Ser. B 47 325--337.
  • De Gooijer, J. G. (1980). Exact moments of the sample autocorrelations from series generated by general ARIMA processes of order $(p, d, q)$, $d=0$ or 1. J. Econometrics 14 365--379.
  • DiCiccio, T. J., Kass, R. E., Raftery, A. and Wasserman, L. (1997). Computing Bayes factors by combining simulation and asymptotic approximations. J. Amer. Statist. Assoc. 92 903--915.
  • Dickey, D. A. and Fuller, W. A. (1979). Distribution of the estimators for autoregressive time series with a unit root. J. Amer. Statist. Assoc. 74 427--431.
  • Efron, B. and Morris, C. N. (1973). Stein's estimation rule and its competitors---An empirical Bayes approach. J. Amer. Statist. Assoc. 68 117--130.
  • Elliott, G., Rothenberg, T. and Stock, J. H. (1996). Efficient tests for an autoregression unit root. Econometrica 64 813--836.
  • Evans, G. B. A. and Savin, N. E. (1981). Testing for unit roots. I. Econometrica 49 753--779.
  • Evans, G. B. A. and Savin, N. E. (1984). Testing for unit roots. II. Econometrica 52 1241--1269.
  • Feller, W. (1971). An Introduction to Probability Theory and Its Applications 2, 2nd ed. Wiley, New York.
  • Fisher, R. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika 10 507--521.
  • Fisher, R. A. (1929). Moments and product moments of sampling distributions. Proc. London Math. Soc. (2) 30 199--238.
  • Fisher, R. A. (1930). The moments of the distribution for normal samples of measures of departure from normality. Proc. Roy. Soc. London Ser. A 130 16--28.
  • From, S. G. and Saxena, K. M. L. (1989). Estimating parameters from mixed samples using sample fractional moments. J. Statist. Plann. Inference 21 231--244.
  • Gelman, A. and Meng, X.-L. (1998). Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statist. Sci. 13 163--185.
  • Gonzalo, J. and Pitarakis, J. (1998). On the exact moments of asymptotic distributions in an unstable AR(1) with dependent errors. Internat. Econom. Rev. 39 71--88.
  • Gradshteyn, I. S. and Ryzhik, I. M. (1992). Table of Integrals, Series, and Products, corrected and enlarged ed. Academic Press, San Diego, CA.
  • Hoque, A. (1985). The exact moments of forecast error in the general dynamic model. Sankhyā Ser. B 47 128--143.
  • Hotelling, H. (1953). New light on the correlation coefficient and its transforms (with discussion). J. Roy. Statist. Soc. Ser. B 15 193--232.
  • James, W. and Stein, C. (1961). Estimation with quadratic loss. Proc. Fourth Berkeley Symp. Math. Statist. Probab. 1 361--379. Univ. California Press, Berkeley.
  • Johnson, N. L., Kotz, S. and Balakrishnan, N. (1995). Continuous Univariate Distributions 2, 2nd ed. Wiley, New York.
  • Johnson, N. L., Kotz, S. and Kemp, A. W. (1992). Univariate Discrete Distributions, 2nd ed. Wiley, New York.
  • Johnson, P. D., Jr. (1975). An algebraic definition of fractional differentiation. Fractional Calculus and Its Applications. Lecture Notes in Math. 457 226--231. Springer, Berlin.
  • Jones, M. C. (1986). Expressions for inverse moments of positive quadratic forms in normal variables. Austral. J. Statist. 28 242--250.
  • Jones, M. C. (1987a). Inverse factorial moments. Statist. Probab. Lett. 6 37--42. Correction 6 369.
  • Jones, M. C. (1987b). On moments of ratios of quadratic forms in normal variables. Statist. Probab. Lett. 6 129--136. Correction 6 369.
  • Khuri, A. and Casella, G. (2002). The existence of the first negative moment revisited. Amer. Statist. 56 44--47.
  • Kong, A., McCullagh, P., Meng, X.-L., Nicolae, D. and Tan, Z. (2003). A theory of statistical models for Monte Carlo integration (with discussion). J. R. Stat. Soc. Ser. B Stat. Methodol. 65 585--618.
  • Laue, G. (1980). Remarks on the relation between fractional moments and fractional derivatives of characteristic functions. J. Appl. Probab. 17 456--466.
  • Lin, J. L. (2003). Discussion of ``From unit root to Stein's estimator to Fisher's $k$ statistics: If you have a moment, I can tell you more,'' by X.-L. Meng. Presented at the NSF-NBER Time-Series Conference, Chicago, September 19--20, 2003.
  • Maatta, J. M. and Casella, G. (1990). Developments in decision-theoretic variance estimation (with discussion). Statist. Sci. 5 90--120.
  • Mathai, A. M. (1991). On fractional moments of quadratic expressions in normal variables. Comm. Statist. Theory Methods 20 3159--3174.
  • Mehta, J. S. and Swamy, P. A. V. B. (1978). The existence of moments of some simple Bayes estimators of coefficients in a simultaneous equation model. J. Econometrics 7 1--13.
  • Meng, X.-L. and Schilling, S. (1996). Fitting full-information item factor models and an empirical investigation of bridge sampling. J. Amer. Statist. Assoc. 91 1254--1267.
  • Meng, X.-L. and Schilling, S. (2002). Warp bridge sampling. J. Comput. Graph. Statist. 11 552--586.
  • Meng, X.-L. and Wong, W. H. (1996). Simulating ratios of normalizing constants via a simple identity: A theoretical exploration. Statist. Sinica 6 831--860.
  • Morin, D. (1992). Exact moments of ratios of quadratic forms. Metron 30 59--78.
  • Nankervis, J. C. and Savin, N. E. (1988). The exact moments of the least-squares estimator for the autoregressive model: Corrections and extensions. J. Econometrics 37 381--388.
  • Neveu, J. (1965). Mathematical Foundations of the Calculus of Probability. Holden-Day, San Francisco.
  • Nielsen, B. (1997). Bartlett correction of the unit root test in autoregressive models. Biometrika 84 500--504.
  • Olkin, I. and Pratt, J. W. (1958). Unbiased estimation of certain correlation coefficients. Ann. Math. Statist. 29 201--211.
  • Ott, J. (1979). Maximum likelihood estimation by counting methods under polygenic and mixed models in human pedigrees. American J. Human Genetics 31 161--175.
  • Peters, T. A. (1989). The exact moments of OLS in dynamic regression models with nonnormal errors. J. Econometrics 40 279--305.
  • Piegorsch, W. W. and Casella, G. (1985). The existence of the first negative moment. Amer. Statist. 39 60--62. Comments by N. L. Johnson, 39 240 and J. Hannan, 39 326.
  • Pitarakis, J. (1998). Moment generating functions and further exact results for seasonal autoregressions. Econometric Theory 14 770--782.
  • Provost, S. B. and Rudiuk, E. M. (1994). The exact density function of the ratio of two dependent linear combinations of chi-square variables. Ann. Inst. Statist. Math. 46 557--571.
  • Rao, M. M. (1978). Asymptotic distribution of an estimator of the boundary parameter of an unstable process. Ann. Statist. 6 185--190. Correction 8 1403.
  • Romero, M. (2003). On two topics with no bridge: Bridge sampling with dependent draws and bias of the multiple imputation variance estimator. Ph.D dissertation, Dept. Statistics, Univ. Chicago.
  • Ross, B. (1975). A brief history and exposition of the fundamental theory of fractional calculus. Fractional Calculus and Its Applications. Lecture Notes in Math. 457 1--36. Springer, Berlin.
  • Sawa, T. (1972). Finite sample properties of the $k$-class estimators. Econometrica 40 653--680.
  • Sawa, T. (1978). The exact moments of the least squares estimator for the autoregressive model. J. Econometrics 8 159--172.
  • Servidea, J. (2002). Bridge sampling with dependent random draws: Techniques and strategy. Ph.D. dissertation, Dept. Statistics, Univ. Chicago.
  • Shepp, L. A. and Lloyd, S. P. (1966). Ordered cycle lengths in a random permutation. Trans. Amer. Math. Soc. 121 340--357.
  • Springer, M. D. (1979). The Algebra of Random Variables. Wiley, New York.
  • Stein, C. (1981). Estimation of the mean of a multivariate normal distribution. Ann. Statist. 9 1135--1151.
  • Stigler, S. M. (1990). The 1988 Neyman memorial lecture: A Galtonian perspective on shrinkage estimators. Statist. Sci. 5 147--155.
  • Stuart, A. and Ord, J. K. (1987). Kendall's Advanced Theory of Statistics 1. Distribution Theory, 5th ed. Oxford Univ. Press, London.
  • Tanaka, K. (1996). Time Series Analysis: Nonstationary and Noninvertible Distribution Theory. Wiley, New York.
  • Tsui, A. K. and Ali, M. M. (1994). Exact distributions, density functions and moments of the least squares estimator in a first-order autoregressive model. Comput. Statist. Data Anal. 17 433--454.
  • Voter, A. F. (1985). A Monte Carlo method for determining free-energy differences and transition state theory rate constants. J. Chemical Physics 82 1890--1899.
  • White, J. S. (1958). The limiting distribution of the serial correlation coefficient in the explosive case. Ann. Math. Statist. 29 1188--1197.
  • White, J. S. (1959). The limiting distribution of the serial correlation coefficient in the explosive case. II. Ann. Math. Statist. 30 831--834.
  • Williams, J. D. (1941). Moments of the ratio of the mean square successive difference to the mean square difference in samples from a normal universe. Ann. Math. Statist. 12 239--241.
  • Wolfe, S. J. (1975). On moments of probability distribution functions. Fractional Calculus and Its Applications. Lecture Notes in Math. 457 306--316. Springer, Berlin.