We propose a strategy for computing estimators in some non-standard M-estimation problems, where the data are distributed across different servers and the observations across servers, though independent, can come from heterogeneous sub-populations, thereby violating the identically distributed assumption. Our strategy fixes the super-efficiency phenomenon observed in prior work on distributed computing in (i) the isotonic regression framework, where averaging several isotonic estimates (each computed at a local server) on a central server produces super-efficient estimates that do not replicate the properties of the global isotonic estimator, i.e. the isotonic estimate that would be constructed by transferring all the data to a single server, and (ii) certain types of M-estimation problems involving optimization of discontinuous criterion functions where M-estimates converge at the cube-root rate. The new estimators proposed in this paper work by smoothing the data on each local server, communicating the smoothed summaries to the central server, and then solving a non-linear optimization problem at the central server. They are shown to replicate the asymptotic properties of the corresponding global estimators, and also overcome the super-efficiency phenomenon exhibited by existing estimators.
References
[1] Anevski, D., Hössjer, O., et al. (2006). A general asymptotic scheme for inference under order restrictions., The Annals of Statistics, 34(4):1874–1930. 1246.62019 10.1214/009053606000000443 euclid.aos/1162567637[1] Anevski, D., Hössjer, O., et al. (2006). A general asymptotic scheme for inference under order restrictions., The Annals of Statistics, 34(4):1874–1930. 1246.62019 10.1214/009053606000000443 euclid.aos/1162567637
[2] Azadbakhsh, M., Jankowski, H., and Gao, X. (2014). Computing confidence intervals for log-concave densities., Computational Statistics & Data Analysis, 75:248–264. 06983959 10.1016/j.csda.2014.01.020[2] Azadbakhsh, M., Jankowski, H., and Gao, X. (2014). Computing confidence intervals for log-concave densities., Computational Statistics & Data Analysis, 75:248–264. 06983959 10.1016/j.csda.2014.01.020
[3] Banerjee, M., Durot, C., Sen, B., et al. (2019). Divide and conquer in nonstandard problems and the super-efficiency phenomenon., The Annals of Statistics, 47(2):720–757. 07033149 10.1214/17-AOS1633 euclid.aos/1547197236[3] Banerjee, M., Durot, C., Sen, B., et al. (2019). Divide and conquer in nonstandard problems and the super-efficiency phenomenon., The Annals of Statistics, 47(2):720–757. 07033149 10.1214/17-AOS1633 euclid.aos/1547197236
[4] Banerjee, M. et al. (2007). Likelihood based inference for monotone response models., The Annals of Statistics, 35(3):931–956. 1133.62328 10.1214/009053606000001578 euclid.aos/1185303993[4] Banerjee, M. et al. (2007). Likelihood based inference for monotone response models., The Annals of Statistics, 35(3):931–956. 1133.62328 10.1214/009053606000001578 euclid.aos/1185303993
[5] Banerjee, M. and Wellner, J. A. (2001). Likelihood ratio tests for monotone functions., Annals of Statistics, pages 1699–1731. 1043.62037 10.1214/aos/1015345959 euclid.aos/1015345959[5] Banerjee, M. and Wellner, J. A. (2001). Likelihood ratio tests for monotone functions., Annals of Statistics, pages 1699–1731. 1043.62037 10.1214/aos/1015345959 euclid.aos/1015345959
[6] Battey, H., Fan, J., Liu, H., Lu, J., and Zhu, Z. (2018). Distributed testing and estimation under sparse high dimensional models., Annals of statistics, 46(3):1352. 1392.62060 10.1214/17-AOS1587 euclid.aos/1525313085[6] Battey, H., Fan, J., Liu, H., Lu, J., and Zhu, Z. (2018). Distributed testing and estimation under sparse high dimensional models., Annals of statistics, 46(3):1352. 1392.62060 10.1214/17-AOS1587 euclid.aos/1525313085
[7] Billingsley, P. (2013)., Convergence of probability measures. John Wiley & Sons. 0172.21201[7] Billingsley, P. (2013)., Convergence of probability measures. John Wiley & Sons. 0172.21201
[8] Groeneboom, P., Jongbloed, G., Witte, B. I., et al. (2010). Maximum smoothed likelihood estimation and smoothed maximum likelihood estimation in the current status model., The Annals of Statistics, 38(1):352–387. 1181.62157 10.1214/09-AOS721 euclid.aos/1262271618[8] Groeneboom, P., Jongbloed, G., Witte, B. I., et al. (2010). Maximum smoothed likelihood estimation and smoothed maximum likelihood estimation in the current status model., The Annals of Statistics, 38(1):352–387. 1181.62157 10.1214/09-AOS721 euclid.aos/1262271618
[10] Hsieh, C.-J., Si, S., and Dhillon, I. (2014). A divide-and-conquer solver for kernel support vector machines. In, International Conference on Machine Learning, pages 566–574.[10] Hsieh, C.-J., Si, S., and Dhillon, I. (2014). A divide-and-conquer solver for kernel support vector machines. In, International Conference on Machine Learning, pages 566–574.
[11] Kim, J. and Pollard, D. (1990). Cube root asymptotics., Annals of Statistics, 18:191–219. 0703.62063 10.1214/aos/1176347498 euclid.aos/1176347498[11] Kim, J. and Pollard, D. (1990). Cube root asymptotics., Annals of Statistics, 18:191–219. 0703.62063 10.1214/aos/1176347498 euclid.aos/1176347498
[12] Li, R., Lin, D. K., and Li, B. (2013). Statistical inference in massive data sets., Applied Stochastic Models in Business and Industry, 29(5):399–409.[12] Li, R., Lin, D. K., and Li, B. (2013). Statistical inference in massive data sets., Applied Stochastic Models in Business and Industry, 29(5):399–409.
[13] Lu, J., Cheng, G., and Liu, H. (2016). Nonparametric heterogeneity testing for massive data., arXiv preprint arXiv: 1601.06212.[13] Lu, J., Cheng, G., and Liu, H. (2016). Nonparametric heterogeneity testing for massive data., arXiv preprint arXiv: 1601.06212.
[14] Mammen, E. (1991). Estimating a smooth monotone regression function., The Annals of Statistics, pages 724–740. 0737.62038 10.1214/aos/1176348117 euclid.aos/1176348117[14] Mammen, E. (1991). Estimating a smooth monotone regression function., The Annals of Statistics, pages 724–740. 0737.62038 10.1214/aos/1176348117 euclid.aos/1176348117
[15] Massart, P. (1990). The tight constant in the dvoretzky-kiefer-wolfowitz inequality., The Annals of Probability, pages 1269–1283. 0713.62021 10.1214/aop/1176990746 euclid.aop/1176990746[15] Massart, P. (1990). The tight constant in the dvoretzky-kiefer-wolfowitz inequality., The Annals of Probability, pages 1269–1283. 0713.62021 10.1214/aop/1176990746 euclid.aop/1176990746
[16] Mukerjee, H. (1988). Monotone nonparametric regression., The Annals of Statistics, pages 741–750. 0647.62042 10.1214/aos/1176350832 euclid.aos/1176350832[16] Mukerjee, H. (1988). Monotone nonparametric regression., The Annals of Statistics, pages 741–750. 0647.62042 10.1214/aos/1176350832 euclid.aos/1176350832
[17] Robertson, T., Wright, F. T., and Dykstra, R. L. (1988)., Order restricted statistical inference. Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. John Wiley & Sons Ltd., Chichester.[17] Robertson, T., Wright, F. T., and Dykstra, R. L. (1988)., Order restricted statistical inference. Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. John Wiley & Sons Ltd., Chichester.
[18] Rosenthal, H. P. (1970). On the subspaces of $L_p,(p>2)$ spanned by sequences of independent random variables., Israel Journal of Mathematics, 8(3):273–303. 0213.19303 10.1007/BF02771562[18] Rosenthal, H. P. (1970). On the subspaces of $L_p,(p>2)$ spanned by sequences of independent random variables., Israel Journal of Mathematics, 8(3):273–303. 0213.19303 10.1007/BF02771562
[19] Shang, Z. and Cheng, G. (2017). Computational limits of a distributed algorithm for smoothing spline., The Journal of Machine Learning Research, 18(1):3809–3845.[19] Shang, Z. and Cheng, G. (2017). Computational limits of a distributed algorithm for smoothing spline., The Journal of Machine Learning Research, 18(1):3809–3845.
[20] Shi, C., Lu, W., and Song, R. (2018). A massive data framework for m-estimators with cubic-rate., Journal of the American Statistical Association, 113(524):1698–1709. 1409.62105 10.1080/01621459.2017.1360779[20] Shi, C., Lu, W., and Song, R. (2018). A massive data framework for m-estimators with cubic-rate., Journal of the American Statistical Association, 113(524):1698–1709. 1409.62105 10.1080/01621459.2017.1360779
[21] Tang, R., Banerjee, M., Kosorok, M. R., et al. (2012). Likelihood based inference for current status data on a grid: A boundary phenomenon and an adaptive inference procedure., The Annals of Statistics, 40(1):45–72. 1246.62090 10.1214/11-AOS942 euclid.aos/1331830774[21] Tang, R., Banerjee, M., Kosorok, M. R., et al. (2012). Likelihood based inference for current status data on a grid: A boundary phenomenon and an adaptive inference procedure., The Annals of Statistics, 40(1):45–72. 1246.62090 10.1214/11-AOS942 euclid.aos/1331830774
[22] Van Der Vaart, A. and Van Der Laan, M. (2003). Smooth estimation of a monotone density., Statistics: A Journal of Theoretical and Applied Statistics, 37(3):189–203. 1037.62028 10.1080/0233188031000124392[22] Van Der Vaart, A. and Van Der Laan, M. (2003). Smooth estimation of a monotone density., Statistics: A Journal of Theoretical and Applied Statistics, 37(3):189–203. 1037.62028 10.1080/0233188031000124392
[23] van der Vaart, A. W. and Wellner, J. A. (1996)., Weak convergence and empirical processes. Springer Series in Statistics. Springer-Verlag, New York. With applications to statistics. 0862.60002[23] van der Vaart, A. W. and Wellner, J. A. (1996)., Weak convergence and empirical processes. Springer Series in Statistics. Springer-Verlag, New York. With applications to statistics. 0862.60002
[24] Volgushev, S., Chao, S.-K., and Cheng, G. (2019). Distributed inference for quantile regression processes., The Annals of Statistics, 47(3):1634–1662. 07053521 10.1214/18-AOS1730 euclid.aos/1550026852[24] Volgushev, S., Chao, S.-K., and Cheng, G. (2019). Distributed inference for quantile regression processes., The Annals of Statistics, 47(3):1634–1662. 07053521 10.1214/18-AOS1730 euclid.aos/1550026852
[25] Zhang, R., Kim, J., and Woodroofe, M. (2001). Asymptotic analysis of isotonic estimation for grouped data., Journal of statistical planning and inference, 98(1–2):107–117. 0978.62030 10.1016/S0378-3758(00)00301-3[25] Zhang, R., Kim, J., and Woodroofe, M. (2001). Asymptotic analysis of isotonic estimation for grouped data., Journal of statistical planning and inference, 98(1–2):107–117. 0978.62030 10.1016/S0378-3758(00)00301-3
[27] Zhao, T., Cheng, G., and Liu, H. (2016). A partially linear framework for massive heterogeneous data., Annals of statistics, 44(4):1400. 1358.62050 10.1214/15-AOS1410 euclid.aos/1467894703[27] Zhao, T., Cheng, G., and Liu, H. (2016). A partially linear framework for massive heterogeneous data., Annals of statistics, 44(4):1400. 1358.62050 10.1214/15-AOS1410 euclid.aos/1467894703
