Communications in Applied Mathematics and Computational Science

Discrete nonhomogeneous and nonstationary logistic and Markov regression models for spatiotemporal data with unresolved external influences

Jana de Wiljes, Lars Putzig, and Illia Horenko

Full-text: Open access


Dynamical systems with different characteristic behavior at multiple scales can be modeled with hybrid methods combining a discrete model (e.g., corresponding to the microscale) triggered by a continuous mechanism and vice versa. A data-driven black-box-type framework is proposed, where the discrete model is parametrized with adaptive regression techniques and the output of the continuous counterpart (e.g., output of partial differential equations) is coupled to the discrete system of interest in the form of a fixed exogenous time series of external factors. Data availability represents a significant issue for this type of coupled discrete-continuous model, and it is shown that missing information/observations can be incorporated in the model via a nonstationary and nonhomogeneous formulation. An unbiased estimator for the discrete model dynamics in presence of unobserved external impacts is derived and used to construct a data-based nonstationary and nonhomogeneous parameter estimator based on an appropriately regularized spatiotemporal clustering algorithm. One-step and long-term predictions are considered, and a new Bayesian approach to discrete data assimilation of hidden information is proposed. To illustrate our method, we apply it to synthetic data sets and compare it with standard techniques of the machine-learning community (such as maximum-likelihood estimation, artificial neural networks and support vector machines).

Article information

Commun. Appl. Math. Comput. Sci., Volume 9, Number 1 (2014), 1-46.

Received: 29 November 2012
Revised: 22 October 2013
Accepted: 15 January 2014
First available in Project Euclid: 20 December 2017

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62-07: Data analysis 62H30: Classification and discrimination; cluster analysis [See also 68T10, 91C20] 62M05: Markov processes: estimation 62M10: Time series, auto-correlation, regression, etc. [See also 91B84] 65C60: Computational problems in statistics
Secondary: 62M02: Markov processes: hypothesis testing 62M20: Prediction [See also 60G25]; filtering [See also 60G35, 93E10, 93E11] 62M30: Spatial processes 62M45: Neural nets and related approaches 62H11: Directional data; spatial statistics

nonstationary nonhomogeneous discrete spatiotemporal time-series analysis Markov regression logistic data assimilation


de Wiljes, Jana; Putzig, Lars; Horenko, Illia. Discrete nonhomogeneous and nonstationary logistic and Markov regression models for spatiotemporal data with unresolved external influences. Commun. Appl. Math. Comput. Sci. 9 (2014), no. 1, 1--46. doi:10.2140/camcos.2014.9.1.

Export citation


  • H. Akaike, A new look at the statistical model identification, IEEE Trans. Automatic Control 19 (1974), no. 6, 716–723.
  • C. M. Bishop, Neural networks for pattern recognition, Clarendon Press, New York, 1995.
  • C. Blume, K. Matthes, and I. Horenko, Supervised learning approaches to classify stratospheric warming events, J. Atmos. Sci. 69 (2012), no. 6, 1824–1840.
  • A. Bordes, S. Ertekin, J. Weston, and L. Bottou, Fast kernel classifiers with online and active learning, J. Mach. Learn. Res. 6 (2005), 1579–1619.
  • K. P. Burnham and D. R. Anderson, Model selection and multimodel inference: a practical information-theoretic approach, 2nd ed., Springer, New York, 2002.
  • S. Chib and E. Greenberg, Understanding the Metropolis–Hastings algorithm, Am. Stat. 49 (1995), no. 4, 327–335.
  • J. S. Chipman, The foundations of utility, Econometrica 28 (1960), no. 2, 193–224.
  • N. Cristianini and J. Shawe-Taylor, An introduction to support vector machines and other kernel-based learning methods, Cambridge University Press, 2000.
  • J. de Wiljes, A. Majda, and I. Horenko, An adaptive Markov chain Monte Carlo approach to time series clustering of processes with regime transition behavior, Multiscale Model. Simul. 11 (2013), no. 2, 415–441.
  • A. J. Dobson and A. G. Barnett, An introduction to generalized linear models, 3rd ed., CRC Press, Boca Raton, 2008.
  • J. Fox, Applied regression analysis, linear models, and related methods, SAGE Publications, Thousand Oaks, 1997.
  • J. Gill, Generalized linear models: a unified approach, Quantitative Applications in the Social Sciences, no. 134, SAGE Publications, Thousand Oaks, 2001.
  • J. Hadamard, Sur les problèmes aux dérivées partielles et leur signification physique, Princeton University Bulletin 13 (1902), no. 4, 49–52.
  • M. T. Hagan and M. B. Menhaj, Training feedforward networks with the Marquardt algorithm, IEEE Trans. Neural Networks 5 (1994), no. 6, 989–993.
  • L. D. Haugh and G. E. P. Box, Identification of dynamic regression (distributed lag) models connecting two time series, J. Amer. Statist. Assoc. 72 (1977), no. 357, 121–130.
  • I. Horenko, Nonstationarity in multifactor models of discrete jump processes, memory, and application to cloud modeling, J. Atmos. Sci. 68 (2011), no. 7, 1493–1506.
  • ––––, On analysis of nonstationary categorical data time series: dynamical dimension reduction, model selection, and applications to computational sociology, Multiscale Model. Simul. 9 (2011), no. 4, 1700–1726.
  • K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2 (1989), no. 5, 359–366.
  • M. A. Katsoulakis, A. J. Majda, and A. Sopasakis, Multiscale couplings in prototype hybrid deterministic/stochastic systems, I: Deterministic closures, Commun. Math. Sci. 2 (2004), no. 2, 255–294.
  • ––––, Multiscale couplings in prototype hybrid deterministic/stochastic systems, II: Stochastic closures, Commun. Math. Sci. 3 (2005), no. 3, 453–478.
  • ––––, Hybrid deterministic stochastic systems with microscopic look-ahead dynamics, Commun. Math. Sci. 8 (2010), no. 2, 409–437.
  • S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi, Optimization by simulated annealing, Science 220 (1983), no. 4598, 671–680.
  • V. Kurkova, Kolmogorov's theorem and multilayer neural networks, Neural Networks 5 (1992), no. 3, 501–506.
  • J. Lawrence, Introduction to neural networks: design, theory, and applications, 6th ed., California Scientific Software Press, Nevada City, CA, 1994.
  • W. D. Li and C. A. McMahon, A simulated annealing-based optimization approach for integrated process planning and scheduling, Int. J. Comp. Integ. M. 20 (2007), no. 1, 80–95.
  • T. F. Liao, Interpreting probability models: logit, probit, and other generalized linear models, SAGE Publications, Thousand Oaks, 1994.
  • R. D. Luce, Individual choice behavior: a theoretical analysis, Wiley, New York, 1959.
  • C. F. Manski and D. McFadden (eds.), Structural analysis of discrete data with econometric applications, MIT Press, Cambridge, MA, 1981.
  • D. McFadden, Conditional logit analysis of qualitative choice behaviour, Frontiers in econometrics (P. Zarembka, ed.), Academic Press, New York, 1974, pp. 105–142.
  • P. Metzner, L. Putzig, and I. Horenko, Analysis of persistent nonstationary time series and applications, Commun. Appl. Math. Comput. Sci. 7 (2012), no. 2, 175–229.
  • A. Y. Ng and M. I. Jordan, On discriminative vs. generative classifiers: a comparison of logistic regression and naive Bayes, Adv. Neural Inf. Process. Syst. (2002), 841–848.
  • A. Pankratz, Forecasting with dynamic regression models, Wiley, Hoboken, NJ, 1991.
  • C. P. Robert and G. Casella, Monte Carlo statistical methods, 2nd ed., Springer, New York, 2004.
  • S. Shalev-Shwartz and N. Srebro, SVM optimization: inverse dependence on training set size, Proceedings of the 25th International Conference on Machine Learning (A. McCallum and S. Roweis, eds.), 2008, pp. 928–935.
  • J. Shawe-Taylor and N. Cristianini, Kernel methods for pattern analysis, Cambridge University Press, 2004.
  • K. E. Train, Discrete choice methods with simulation, 2nd ed., Cambridge University Press, 2009.
  • S. A. Vavasis, Quadratic programming is in NP, Inform. Process. Lett. 36 (1990), no. 2, 73–77.
  • J. von Neumann, Various techniques used in connection with random digits, National Bureau of Standards Applied Math Series 11 (1951), 36–38.
  • H. Yu and B. M. Wilamowski, Levenberg–Marquardt training, Intelligent systems (B. M. Wilamowski and J. D. Irwin, eds.), Industrial Electronics Handbook, no. 5, CRC Press, Boca Raton, 2nd ed., 2011.