• Bernoulli
  • Volume 18, Number 3 (2012), 1061-1098.

$\sqrt n$-consistent parameter estimation for systems of ordinary differential equations: bypassing numerical integration via smoothing

Shota Gugushvili and Chris A.J. Klaassen

Full-text: Open access


We consider the problem of parameter estimation for a system of ordinary differential equations from noisy observations on a solution of the system. In case the system is nonlinear, as it typically is in practical applications, an analytic solution to it usually does not exist. Consequently, straightforward estimation methods like the ordinary least squares method depend on repetitive use of numerical integration in order to determine the solution of the system for each of the parameter values considered, and to find subsequently the parameter estimate that minimises the objective function. This induces a huge computational load to such estimation methods. We study the consistency of an alternative estimator that is defined as a minimiser of an appropriate distance between a nonparametrically estimated derivative of the solution and the right-hand side of the system applied to a nonparametrically estimated solution. This smooth and match estimator (SME) bypasses numerical integration altogether and reduces the amount of computational time drastically compared to ordinary least squares. Moreover, we show that under suitable regularity conditions this smooth and match estimation procedure leads to a $\sqrt n$-consistent estimator of the parameter of interest.

Article information

Bernoulli, Volume 18, Number 3 (2012), 1061-1098.

First available in Project Euclid: 28 June 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

M-estimator $\sqrt n$-consistency nonparametric regression ODE system Priestley–Chao estimator


Gugushvili, Shota; Klaassen, Chris A.J. $\sqrt n$-consistent parameter estimation for systems of ordinary differential equations: bypassing numerical integration via smoothing. Bernoulli 18 (2012), no. 3, 1061--1098. doi:10.3150/11-BEJ362.

Export citation


  • [1] Arnol’d, V.I. (1973). Ordinary Differential Equations. Cambridge, MA: MIT Press.
  • [2] Bellman, R. and Roth, R.S. (1971). The use of splines with unknown end points in the identification of systems. J. Math. Anal. Appl. 34 26–33.
  • [3] Benedetti, J.K. (1977). On the nonparametric estimation of regression functions. J. Roy. Statist. Soc. Ser. B 39 248–253.
  • [4] Bickel, P.J., Klaassen, C.A.J., Ritov, Y. and Wellner, J.A. (1998). Efficient and Adaptive Estimation for Semiparametric Models. New York: Springer.
  • [5] Bickel, P.J. and Ritov, Y. (2003). Nonparametric estimators which can be “plugged-in”. Ann. Statist. 31 1033–1053.
  • [6] Bock, H.G. (1983). Recent advances in parameter identification techniques for ODE. In Numerical Treatment of Inverse Problems in Differential and Integral Equations (Heidelberg, 1982). Progr. Sci. Comput. 2 95–121. Boston, MA: Birkhäuser.
  • [7] Brunel, N.J.B. (2008). Parameter estimation of ODE’s via nonparametric estimators. Electron. J. Stat. 2 1242–1267.
  • [8] Chou, I.C. and Voit, E.O. (2009). Recent developments in parameter estimation and structure identification of biochemical and genomic systems. Math. Biosci. 219 57–83.
  • [9] Edelstein-Keshet, L. (2005). Mathematical Models in Biology. Classics in Applied Mathematics 46. Philadelphia, PA: SIAM.
  • [10] Ellner, S.P., Seifu, Y. and Smith, R.H. (2002). Fitting population dynamic models to time-series data by gradient matching. Ecology 83 2256–2270.
  • [11] Fan, J. and Marron, J.S. (1994). Fast implementations of nonparametric curve estimators. J. Comput. Graph. Stat. 3 35–56.
  • [12] Feinberg, M. (1979). Lectures on chemical reaction networks. Lectures delivered at the Mathematics Research Center. Univ. Wisconsin-Madison. Available at
  • [13] Gasser, T. and Müller, H.G. (1984). Estimating regression functions and their derivatives by the kernel method. Scand. J. Statist. 11 171–185.
  • [14] Gasser, T., Müller, H.G. and Mammitzsch, V. (1985). Kernels for nonparametric curve estimation. J. Roy. Statist. Soc. Ser. B 47 238–252.
  • [15] Gelman, A., Bois, F.Y. and Jiang, J. (1996). Physiological pharmacokinetic analysis using population modeling and informative prior distributions. J. Amer. Statist. Assoc. 91 1400–1412.
  • [16] Girolami, M. (2008). Bayesian inference for differential equations. Theoret. Comput. Sci. 408 4–16.
  • [17] Goldstein, L. and Messer, K. (1992). Optimal plug-in estimators for nonparametric functional estimation. Ann. Statist. 20 1306–1328.
  • [18] Hairer, E. and Wanner, G. (1996). Solving Ordinary Differential Equations. II. Stiff and Differential-Algebraic Problems, 2nd ed. Springer Series in Computational Mathematics 14. Berlin: Springer.
  • [19] Hall, P. and Marron, J.S. (1990). On variance estimation in nonparametric regression. Biometrika 77 415–419.
  • [20] Hemker, P.W. (1972). Numerical methods for differential equations in system simulation and in parameter estimation. In Analysis and Simulation of Biochemical Systems (H.C. Hemker and B. Hess, eds.) 59–80. Amsterdam: North Holland.
  • [21] Hlavacek, W.S. and Savageau, M.A. (1996). Rules for coupled expression of regulator and effector genes in inducible circuits. J. Mol. Biol. 255 121–139.
  • [22] Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. J. Amer. Statist. Assoc. 58 13–30.
  • [23] Hooker, G. (2009). Forcing function diagnostics for nonlinear dynamics. Biometrics 65 928–936.
  • [24] Huber, P.J. (1981). Robust Statistics. New York: Wiley.
  • [25] Jennrich, R.I. (1969). Asymptotic properties of non-linear least squares estimators. Ann. Math. Statist. 40 633–643.
  • [26] Jones, M.C., Marron, J.S. and Sheather, S.J. (1996). A brief survey of bandwidth selection for density estimation. J. Amer. Statist. Assoc. 91 401–407.
  • [27] Kikuchi, S., Tominaga, D., Arita, M., Takahashi, K. and Tomita, M. (2003). Dynamic modeling of genetic networks using genetic algorithm and S-system. Bioinformatics 19 643–650.
  • [28] Liang, H. and Wu, H. (2008). Parameter estimation for differential equation models using a framework of measurement error in regression models. J. Amer. Statist. Assoc. 103 1570–1583.
  • [29] Loader, C.R. (1999). Bandwidth selection: Classical or plug-in? Ann. Statist. 27 415–438.
  • [30] Marquardt, D.W. (1963). An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Indust. Appl. Math. 11 431–441.
  • [31] McMurry, T.L. and Politis, D.N. (2004). Nonparametric regression with infinite order flat-top kernels. J. Nonparametr. Stat. 16 549–562.
  • [32] Messer, K. and Goldstein, L. (1993). A new class of kernels for nonparametric curve estimation. Ann. Statist. 21 179–195.
  • [33] Pollard, D. and Radchenko, P. (2006). Nonlinear least-squares estimation. J. Multivariate Anal. 97 548–562.
  • [34] Priestley, M.B. and Chao, M.T. (1972). Non-parametric function fitting. J. Roy. Statist. Soc. Ser. B 34 385–392.
  • [35] Qi, X. and Zhao, H. (2010). Asymptotic efficiency and finite-sample properties of the generalized profiling estimation of parameters in ordinary differential equations. Ann. Statist. 38 435–481.
  • [36] Ramsay, J.O., Hooker, G., Campbell, D. and Cao, J. (2007). Parameter estimation for differential equations: A generalized smoothing approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 741–796. With discussions and a reply by the authors.
  • [37] Schuster, E. and Yakowitz, S. (1979). Contributions to the theory of nonparametric regression, with application to system identification. Ann. Statist. 7 139–149.
  • [38] Sontag, E.D. (2001). Structure and stability of certain chemical networks and applications to the kinetic proofreading model of T-cell receptor signal transduction. IEEE Trans. Automat. Control 46 1028–1047.
  • [39] Stigler, S.M. (1981). Gauss and the invention of least squares. Ann. Statist. 9 465–474.
  • [40] Stortelder, W.J.H. (1996). Parameter estimation in dynamic systems. Math. Comput. Simulat. 42 135–142.
  • [41] Szegő, G. (1975). Orthogonal Polynomials, 4th ed. American Mathematical Society, Colloquium Publications XXIII. Providence, RI: Amer. Math. Soc.
  • [42] Tsybakov, A.B. (2009). Introduction to Nonparametric Estimation. Springer Series in Statistics. New York: Springer.
  • [43] van de Geer, S. (1990). Estimating a regression function. Ann. Statist. 18 907–924.
  • [44] van de Geer, S. and Wegkamp, M. (1996). Consistency for the least squares estimator in nonparametric regression. Ann. Statist. 24 2513–2523.
  • [45] van der Vaart, A.W. (1998). Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics 3. Cambridge: Cambridge Univ. Press.
  • [46] van der Vaart, A.W. and Wellner, J.A. (2000). Weak Convergence and Empirical Processes: With Applications to Statistics, 2nd ed. Springer Series in Statistics. New York: Springer.
  • [47] van Es, A.J. (1991). Aspects of Nonparametric Density Estimation. CWI Tract 77. Amsterdam: Stichting Mathematisch Centrum Centrum voor Wiskunde en Informatica.
  • [48] Varah, J.M. (1982). A spline least squares method for numerical parameter estimation in differential equations. SIAM J. Sci. Statist. Comput. 3 28–46.
  • [49] Voit, E.O. (2000). Computational Analysis of Biochemical Systems: A Practical Guide for Biochemists and Molecular Biologists. Cambridge: Cambridge Univ. Press.
  • [50] Voit, E.O. and Almeida, J. (2004). Decoupling dynamical systems for pathway identification from metabolic profiles. Bioinformatics 10 1670–1681.
  • [51] Voit, E.O. and Savageau, M.A. (1982). Power-law approach to modeling biological systems; III. Methods of analysis. J. Ferment. Technol. 60 233–241.
  • [52] Wächter, A. and Biegler, L.T. (2006). On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program. 106 25–57.
  • [53] Wand, M.P. and Jones, M.C. (1995). Kernel Smoothing. Monographs on Statistics and Applied Probability 60. London: Chapman & Hall.
  • [54] Wand, M.P. and Schucany, W.R. (1990). Gaussian-based kernels. Canad. J. Statist. 18 197–204.
  • [55] Wasserman, L. (2006). All of Nonparametric Statistics. Springer Texts in Statistics. New York: Springer.
  • [56] Wolfram Research, Inc. (2007). Mathematica, Version 6.0. Champaign, IL.
  • [57] Wu, C.F. (1981). Asymptotic theory of nonlinear least squares estimation. Ann. Statist. 9 501–513.
  • [58] Xue, H., Miao, H. and Wu, H. (2010). Sieve estimation of constant and time-varying coefficients in nonlinear ordinary differential equation models by considering both numerical error and measurement error. Ann. Statist. 38 2351–2387.