Stochastic Systems

On the convergence of simulation-based iterative methods for solving singular linear systems

Mengdi Wang and Dimitri P. Bertsekas

Full-text: Open access


We consider the simulation-based solution of linear systems of equations, $Ax=b$, of various types frequently arising in large-scale applications, where $A$ is singular. We show that the convergence properties of iterative solution methods are frequently lost when they are implemented with simulation (e.g., using sample average approximation), as is often done in important classes of large-scale problems. We focus on special cases of algorithms for singular systems, including some arising in least squares problems and approximate dynamic programming, where convergence of the residual sequence $\{Ax_{k}-b\}$ may be obtained, while the sequence of iterates $\{x_{k}\}$ may diverge. For some of these special cases, under additional assumptions, we show that the iterate sequence is guaranteed to converge. For situations where the iterates diverge but the residuals converge to zero, we propose schemes for extracting from the divergent sequence another sequence that converges to a solution of $Ax=b$.

Article information

Stoch. Syst., Volume 3, Number 1 (2013), 38-95.

First available in Project Euclid: 24 February 2014

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 15A06: Linear equations
Secondary: 60H99: None of the above, but in this section 65C05: Monte Carlo methods

Stochastic algorithm singular system Monte-Carlo estimation simulation proximal method regularization approximate dynamic programming


Wang, Mengdi; Bertsekas, Dimitri P. On the convergence of simulation-based iterative methods for solving singular linear systems. Stoch. Syst. 3 (2013), no. 1, 38--95. doi:10.1214/12-SSY074.

Export citation


  • [BB96] Bradtke, S. J. and Barto, A. G., Linear least-squares algorithms for temporal difference learning. Machine Learning, 22:33–57, 1996.
  • [Bel70] Bellman, R., Introduction to Matrix Analysis, 2nd edition. McGraw-Hill, N. Y, 1970.
  • [Ber10] Bertsekas, D. P., Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9:310–335, 2010.
  • [Ber11] Bertsekas, D. P., Temporal difference methods for general projected equations. IEEE Trans. on Automatic Control, 56:2128–2139, 2011.
  • [Ber12] Bertsekas, D. P., Dynamic Programming and Optimal Control, Vol. II: Approximate Dynamic Programming. Athena Scientific, Belmont, M. A, 2012.
  • [BG82] Bertsekas, D. P. and Gafni, E. M., Projection methods for variational inequalities with application to the traffic assignment problem. Mathematical Programming Study, 17:139–159, 1982.
  • [BIG74] Ben-Israel, A. and Greville, T. N. E., Generalized Inverse: Theory and Applications. Wiley-Interscience, Springer-Verlag, N. Y, 1974.
  • [BMP90] Benveniste, A., Metivier, M., and Priouret, P., Adaptive Algorithms and Stochastic Approximations. Springer-Verlag, NY, 1990.
  • [BN89] Bertsekas, D. P. and Tsitsiklis, J. N. Parallel and Distributed Computation: Numerical Methods. Athena Scientific, Belmont, MA, 1989.
  • [BN96] Bertsekas, D. P. and Tsitsiklis, J. N. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996.
  • [Bor08] Borkar, V. S., Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, MA, 2008.
  • [Boy02] Boyan, J. A., Technical update: Least-squares temporal difference learning. Machine Learning, 49:233–246, 2002.
  • [BY09] Bertsekas, D. P. and Yu, H., Projected equation methods for approximate solution of large linear systems. Journal of Computational and Applied Mathematics, 227:27–50, 2009.
  • [CPS92] Cottle, R. W., Pang, J. S., and Stone, R. E., The Linear Complementarity Problem. Academic Press, Boston, MA, 1992.
  • [Dax90] Dax, A., The convergence of linear stationary iterative processes for solving singular unstructured systems of linear equations. SIAM Review, 32:611–635, 1990.
  • [DMM06] Drineas, P., Mahoney, M. W., and Muthukrishnan, S., Sampling algorithms for L2 regression and applications. Proc. 17th Annual SODA, pages 1127–1136, 2006.
  • [DMMS11] Drineas, P., Mahoney, M. W., Muthukrishnan, S., and Sarlos, T., Faster least squares approximation. Numerische Mathematik, 117:219–249, 2011.
  • [FP03] Facchinei, F. and Pang, J. S., Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, NY, 2003.
  • [Gol91] Goldberg, J. L., Matrix Theory with Applications. McGraw-Hill, N. Y, 1991.
  • [HY81] Hageman, L. A. and Young, D. M., Applied Iterative Methods. Academic Press, NY, 1981.
  • [Kel65] Keller, H. B., On the solution of singular and semidefinite linear systems by iteration. J. SIAM: Series B, Numerical Analysis, 2:281–290, 1965.
  • [KNS13] Koshal, J., Nedić, A., and Shanbhag, U. V., Regularized iterative stochastic approximation methods for stochastic variational inequality problems. IEEE Transactions on Automatic Control, 58:594–608, 2013.
  • [Kon02] Konda, V., Actor-critic algorithms. PhD Thesis, MIT, 2002.
  • [Kor76] Korpelevich, G. M., An extragradient method for finding saddle points and for other problems. Matecon, 12:747–756, 1976.
  • [Kos09] Kosinski, K. M., On the functional limits for sums of a function of partial sums. Statistics and Probability Letters, 79:1552–1527, 2009.
  • [Kra72] Krasnoselskii, M. A., Approximate Solution of Operator Equations. D. Wolters-Noordhoff Pub., Groningen, 1972.
  • [KS13] Kannan, A. and Shanbhag, U. V., Distributed online computation of nash equilibria via iterative regularization techniques. SIAM Journal of Optimization (to appear), 2013.
  • [KSHdM02] Kleywegt, A. J., Shapiro, A., and Homem-de Mello, T., The sample average approximation method for stochastic discrete optimization. SIAM J. Optim., 12:479–502, 2002.
  • [KY03] Kushner, H. J. and Yin, G., Stochastic Approximation and Recursive Algorithms and Applications. Springer, NY, 2003.
  • [LP89] Luo, Z. Q. and Tseng, P. On the convergence of a matrix splitting algorithm for the symmetric monotone linear complementarity problem. SIAM Journal of Control and Optimization, 29:1037–1060, 1989.
  • [M.70] Sibony, M. Methodes iteratives pour les equations et inequations aux derivees partielles non lineaires de type monotone. Calcolo, 7:65–183, 1970.
  • [Mar70] Martinet, B., Regularisation d’inequation variationelles par approximations successives. Rev. Francaise Inf. Rech. Oper., pages 154–159, 1970.
  • [Mey07] Meyn, S. P., Control Techniques for Complex Systems. Cambridge University Press, MA, 2007.
  • [NB03] Nedić, A. and Bertsekas, D. P., Least-squares policy evaluation algorithms with linear function approximation. Journal of Discrete Event Systems, 13:79–110, 2003.
  • [NJLS09] Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A., Robust stochastic approximation approach t stochastic programming. SIAM J. of Optimization, 19:1574–1609, 2009.
  • [Pas10] Pasupathy, R., On choosing parameters in retrospective-approximation algorithms for stochastic root finding and simulation optimization. Operations Research, 58:889=901, 2010.
  • [Pow11] Powell, B. W., Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd Ed. J. Wiley, N. Y, 2011.
  • [Put94] Puterman, M. L., Markovian Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley and Sons, New York, NY, 1994.
  • [PWB10] Polydorides, N., Wang, M., and Bertsekas, D. P., A quasi monte carlo method for large-scale linear inverse problems. Proc. Monte Carlo and Quasi-Monte Carlo, Springer, N. Y, 2010.
  • [Roc76] Rockafellar, R. T., Monotone operators and the proximal point algorithm. SIAM Journal on Control and Optimization, 14:877–898, 1976.
  • [Saa03] Saad, Y., Iterative Methods for Sparse Linear Systems. J. SIAM, Philadelphia, P. A, 2003.
  • [SB98] Sutton, R. S. and Barto, A. G., Reinforcement Learning. MIT Press, Cambridge, M. A, 1998.
  • [SDR09] Shapiro, A., Dentcheva, D., and Ruszczynski, A., Lectures on Stochastic Programming: Modeling and Theory. SIAM, Phila., PA, 2009.
  • [Sha03] Shapiro, A., Monte Carlo Sampling Methods, in Stochastic Programming. Handbook in OR & MS, Vol. 10, North-Holland, Amsterdam, 2003.
  • [SS90] Stewart, G. W. and Sun, J. G., Matrix Perturbation Theory. Academic Press, Boston, M. A, 1990.
  • [Ste73] Stewart, G. W., Introduction to Matrix Computations. Academic Press, NY, 1973.
  • [Ste90] Stewart, G. W., Perturbation Theory for the Singular Value Decomposition. CS-TR-2539, College Park, M. D. 1990.
  • [Tan74] Tanabe, K., Characterization of linear stationary iterative processes for solving a singular system of linear equations. Numerische Mathematik, 22:349–359, 1974.
  • [TB97] Trefethen, L. N. and Bau, D., Numerical Linear Algebra. J. SIAM, Philadelphia, Philadelphia, P. A, 1997.
  • [WB11] Wang, M. and Bertsekas, D. P., Stabilization of stochastic iterative methods for singular and nearly singular linear systems. Lab. for Information and Decision Systems Report, LIDS-P-2878:MIT, Cambridge, MA, 2011.
  • [Wed12] Wedin, P. A., Perturbation bounds in connection with singular value decomposition. BIT, 12:99–111, 2012.
  • [WPB09] Wang, M., Polydorides, N., and Bertsekas, D. P., Approximate simulation-based solution of large-scale least squares problems. Lab. for Information and Decision Systems Report LIDS-P-2819, MIT, Cambridge, M. A, 2009.
  • [YB12] Yu, H. and Bertsekas, D. P., Weighted bellman equations and their applications in dynamic programming. Lab. for Information and Decision Systems Report LIDS-P-2876, MIT, 2012.
  • [You72] Young, D. M., On the consistency of linear stationary iterative methods. SIAM Journal on Numerical Analysis, 9:89–96, 1972.
  • [ZH07] Zhang, L. X. and Huang, W., A note on the invariance principle of the product of sums of random variables. Elect. Comm. in Probability, 12:51–56, 2007.