Electronic Journal of Statistics

A generic algorithm for reducing bias in parametric estimation

Ioannis Kosmidis and David Firth

Full-text: Open access


A general iterative algorithm is developed for the computation of reduced-bias parameter estimates in regular statistical models through adjustments to the score function. The algorithm unifies and provides appealing new interpretation for iterative methods that have been published previously for some specific model classes. The new algorithm can usefully be viewed as a series of iterative bias corrections, thus facilitating the adjusted score approach to bias reduction in any model for which the first-order bias of the maximum likelihood estimator has already been derived. The method is tested by application to a logit-linear multiple regression model with beta-distributed responses; the results confirm the effectiveness of the new algorithm, and also reveal some important errors in the existing literature on beta regression.

Article information

Electron. J. Statist., Volume 4 (2010), 1097-1112.

First available in Project Euclid: 15 October 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62F10: Point estimation 62F12: Asymptotic properties of estimators
Secondary: 62F05: Asymptotic properties of tests

Adjusted score asymptotic bias correction beta regression bias reduction fisher scoring prater gasoline data


Kosmidis, Ioannis; Firth, David. A generic algorithm for reducing bias in parametric estimation. Electron. J. Statist. 4 (2010), 1097--1112. doi:10.1214/10-EJS579. https://projecteuclid.org/euclid.ejs/1287147913

Export citation


  • [1] Albert, A. and J. Anderson (1984). On the existence of maximum likelihood estimates in logistic regression models., Biometrika 71 (1), 1–10.
  • [2] Breslow, N. E. and X. Lin (1995). Bias correction in generalised linear mixed models with a single component of dispersion., Biometrika 82, 81–91.
  • [3] Bull, S. B., C. Mak, and C. Greenwood (2002). A modified score function estimator for multinomial logistic regression in small samples., Computational Statistics and Data Analysis 39, 57–74.
  • [4] Cook, R. D., C.-L. Tsai, and B. C. Wei (1986). Bias in nonlinear regression., Biometrika 73, 615–623.
  • [5] Cordeiro, G. and M. Toyama Udo (2008). Bias correction in generalized nonlinear models with dispersion covariates., Communications in Statistics: Theory and Methods 37 (14), 2219–225.
  • [6] Cordeiro, G. M. and P. McCullagh (1991). Bias correction in generalized linear models., Journal of the Royal Statistical Society, Series B: Methodological 53 (3), 629–643.
  • [7] Cordeiro, G. M. and K. L. P. Vasconcellos (1997). Bias correction for a class of multivariate nonlinear regression models., Statistics & Probability Letters 35, 155–164.
  • [8] Cox, D. R. and D. V. Hinkley (1974)., Theoretical Statistics. London: Chapman & Hall Ltd.
  • [9] Cox, D. R. and E. J. Snell (1968). A general definition of residuals (with discussion)., Journal of the Royal Statistical Society, Series B: Methodological 30, 248–275.
  • [10] Cribari-Neto, F. and A. Zeileis (2010). Beta regression in R., Journal of Statistical Software 34 (2), 1–24.
  • [11] Efron, B. (1975). Defining the curvature of a statistical problem (with applications to second order efficiency) (with discussion)., The Annals of Statistics 3, 1189–1217.
  • [12] Ferrari, S. and F. Cribari-Neto (2004). Beta regression for modelling rates and proportions., Journal of Applied Statistics 31 (7), 799–815.
  • [13] Firth, D. (1992). Bias reduction, the Jeffreys prior and GLIM. In L. Fahrmeir, B. Francis, R. Gilchrist, and G. Tutz (Eds.), Advances in GLIM and Statistical Modelling: Proceedings of the GLIM 92 Conference, Munich, New York, pp. 91–100. Springer.
  • [14] Firth, D. (1993). Bias reduction of maximum likelihood estimates., Biometrika 80 (1), 27–38.
  • [15] Gart, J. J., H. M. Pettigrew, and D. G. Thomas (1985). The effect of bias, variance estimation, skewness and kurtosis of the empirical logit on weighted least squares analyses., Biometrika 72, 179–190.
  • [16] Heinze, G. and M. Schemper (2002). A solution to the problem of separation in logistic regression., Statistics in Medicine 21, 2409–2419.
  • [17] Heinze, G. and M. Schemper (2004). A solution to the problem of monotone likelihood in Cox regression., Biometrics 57, 114–119.
  • [18] Kosmidis, I. (2009). On iterative adjustment of responses for the reduction of bias in binary regression models. Technical Report 09-36, CRiSM working paper, series.
  • [19] Kosmidis, I. and D. Firth (2009). Bias reduction in exponential family nonlinear models., Biometrika 96 (4), 793–804.
  • [20] Lin, X. and N. E. Breslow (1996). Bias correction in generalized linear mixed models with multiple components of dispersion., Journal of the American Statistical Association 91, 1007–1016.
  • [21] McCullagh, P. (1987)., Tensor Methods in Statistics. London: Chapman and Hall.
  • [22] Mehrabi, Y. and J. N. S. Matthews (1995). Likelihood-based methods for bias reduction in limiting dilution assays., Biometrics 51, 1543–1549.
  • [23] Ospina, R., F. Cribari-Neto, and K. L. Vasconcellos (2006). Improved point and interval estimation for a beta regression model., Computational Statistics and Data Analysis 51 (2), 960 – 981.
  • [24] Pace, L. and A. Salvan (1997)., Principles of Statistical Inference: From a Neo-Fisherian Perspective. London: World Scientific.
  • [25] Pettitt, A. N., J. M. Kelly, and J. T. Gao (1998). Bias correction for censored data with exponential lifetimes., Statistica Sinica 8, 941–964.
  • [26] Prater, N. H. (1956). Estimate gasoline yields from crudes., Petroleum Refiner 35, 236–238.
  • [27] R Development Core Team (2008)., R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. ISBN 3-900051-07-0.
  • [28] Sartori, N. (2006). Bias prevention of maximum likelihood estimates for scalar skew normal and skew, t distributions. Journal of Statistical Planning and Inference 136, 4259–4275.
  • [29] Schaefer, R. L. (1983). Bias correction in maximum likelihood logistic regression., Statistics in Medicine 2, 71–78.
  • [30] Simas, A. B., W. Barreto-Souza, and A. V. Rocha (2010). Improved estimators for a general class of beta regression models., Computational Statistics and Data Analysis 54 (2), 348–366.
  • [31] Smithson, M. and J. Verkuilen (2006). A better lemon squeezer? Maximum-likelihood regression with beta-distributed dependent variables., Psychological Methods 11 (1), 54–71.
  • [32] Zorn, C. (2005). A solution to separation in binary response models., Political Analysis 13, 157–170.