This paper studies the inference problem in quantile regression (QR) for a large sample size $n$ but under a limited memory constraint, where the memory can only store a small batch of data of size $m$. A natural method is the naive divide-and-conquer approach, which splits data into batches of size $m$, computes the local QR estimator for each batch and then aggregates the estimators via averaging. However, this method only works when $n=o(m^{2})$ and is computationally expensive. This paper proposes a computationally efficient method, which only requires an initial QR estimator on a small batch of data and then successively refines the estimator via multiple rounds of aggregations. Theoretically, as long as $n$ grows polynomially in $m$, we establish the asymptotic normality for the obtained estimator and show that our estimator with only a few rounds of aggregations achieves the same efficiency as the QR estimator computed on all the data. Moreover, our result allows the case that the dimensionality $p$ goes to infinity. The proposed method can also be applied to address the QR problem under distributed computing environment (e.g., in a large-scale sensor network) or for real-time streaming data.
Ann. Statist.
47(6):
3244-3273
(December 2019).
DOI: 10.1214/18-AOS1777
Banerjee, M., Durot, C. and Sen, B. (2018). Divide and conquer in non-standard problems and the super-efficiency phenomenon. Ann. Statist. To appear. 1416.62259 10.1214/17-AOS1633 euclid.aos/1547197236Banerjee, M., Durot, C. and Sen, B. (2018). Divide and conquer in non-standard problems and the super-efficiency phenomenon. Ann. Statist. To appear. 1416.62259 10.1214/17-AOS1633 euclid.aos/1547197236
Bang, H. and Tsiatis, A. A. (2002). Median regression with censored cost data. Biometrics 58 643–649. 1210.62041 10.1111/j.0006-341X.2002.00643.xBang, H. and Tsiatis, A. A. (2002). Median regression with censored cost data. Biometrics 58 643–649. 1210.62041 10.1111/j.0006-341X.2002.00643.x
Battey, H., Fan, J., Liu, H., Lu, J. and Zhu, Z. (2018). Distributed estimation and inference with statistical guarantees. Ann. Statist. 46 1352–1382. 1392.62060 10.1214/17-AOS1587 euclid.aos/1525313085Battey, H., Fan, J., Liu, H., Lu, J. and Zhu, Z. (2018). Distributed estimation and inference with statistical guarantees. Ann. Statist. 46 1352–1382. 1392.62060 10.1214/17-AOS1587 euclid.aos/1525313085
Beck, A. (2014). Introduction to Nonlinear Optimization. MOS-SIAM Series on Optimization 19. SIAM, Philadelphia, PA. 1320.90001Beck, A. (2014). Introduction to Nonlinear Optimization. MOS-SIAM Series on Optimization 19. SIAM, Philadelphia, PA. 1320.90001
Belloni, A. and Chernozhukov, V. (2011). $\ell_{1}$-penalized quantile regression in high-dimensional sparse models. Ann. Statist. 39 82–130. 1209.62064 10.1214/10-AOS827 euclid.aos/1291388370Belloni, A. and Chernozhukov, V. (2011). $\ell_{1}$-penalized quantile regression in high-dimensional sparse models. Ann. Statist. 39 82–130. 1209.62064 10.1214/10-AOS827 euclid.aos/1291388370
Belloni, A., Chernozhukov, V., Chetverikov, D. and Fernandez-Val, I. (2011). Conditional Quantile Processes based on Series or Many Regressors. Technical report. Preprint. Available at arXiv:1105.6154v3. 1105.6154v3 MR4013213 07121298 10.1016/j.jeconom.2019.04.003Belloni, A., Chernozhukov, V., Chetverikov, D. and Fernandez-Val, I. (2011). Conditional Quantile Processes based on Series or Many Regressors. Technical report. Preprint. Available at arXiv:1105.6154v3. 1105.6154v3 MR4013213 07121298 10.1016/j.jeconom.2019.04.003
Chen, X., Liu, W. and Zhang, Y. (2019). Supplement to “Quantile regression under memory constraint.” DOI:10.1214/18-AOS1777SUPP. MR4025741 1436.62134 10.1214/18-AOS1777 euclid.aos/1572487392Chen, X., Liu, W. and Zhang, Y. (2019). Supplement to “Quantile regression under memory constraint.” DOI:10.1214/18-AOS1777SUPP. MR4025741 1436.62134 10.1214/18-AOS1777 euclid.aos/1572487392
Falk, M. (1999). A simple approach to the generation of uniformly distributed random variables with prescribed correlations. Comm. Statist. Simulation Comput. 28 785–791. 0968.65502 10.1080/03610919908813578Falk, M. (1999). A simple approach to the generation of uniformly distributed random variables with prescribed correlations. Comm. Statist. Simulation Comput. 28 785–791. 0968.65502 10.1080/03610919908813578
Galvao, A. F. and Kato, K. (2016). Smoothed quantile regression for panel data. J. Econometrics 193 92–112. 1420.62483 10.1016/j.jeconom.2016.01.008Galvao, A. F. and Kato, K. (2016). Smoothed quantile regression for panel data. J. Econometrics 193 92–112. 1420.62483 10.1016/j.jeconom.2016.01.008
Gama, J., Sebastião, R. and Rodrigues, P. P. (2013). On evaluating stream learning algorithms. Mach. Learn. 90 317–346. 1260.68329 10.1007/s10994-012-5320-9Gama, J., Sebastião, R. and Rodrigues, P. P. (2013). On evaluating stream learning algorithms. Mach. Learn. 90 317–346. 1260.68329 10.1007/s10994-012-5320-9
Greenwald, M. B. and Khanna, S. (2004). Power-conserving computation of order-statistics over sensor networks. In Proceedings of the ACM Symposium on Principles of Database Systems.Greenwald, M. B. and Khanna, S. (2004). Power-conserving computation of order-statistics over sensor networks. In Proceedings of the ACM Symposium on Principles of Database Systems.
Gu, Y., Fan, J., Kong, L., Ma, S. and Zou, H. (2018). ADMM for high-dimensional sparse penalized quantile regression. Technometrics 60 319–331. MR3847169 10.1080/00401706.2017.1345703Gu, Y., Fan, J., Kong, L., Ma, S. and Zou, H. (2018). ADMM for high-dimensional sparse penalized quantile regression. Technometrics 60 319–331. MR3847169 10.1080/00401706.2017.1345703
Guha, S. and McGregor, A. (2008/09). Stream order and order statistics: Quantile estimation in random-order streams. SIAM J. Comput. 38 2044–2059. 1181.68154 10.1137/07069328XGuha, S. and McGregor, A. (2008/09). Stream order and order statistics: Quantile estimation in random-order streams. SIAM J. Comput. 38 2044–2059. 1181.68154 10.1137/07069328X
He, X. and Shao, Q.-M. (2000). On parameters of increasing dimensions. J. Multivariate Anal. 73 120–135. 0948.62013 10.1006/jmva.1999.1873He, X. and Shao, Q.-M. (2000). On parameters of increasing dimensions. J. Multivariate Anal. 73 120–135. 0948.62013 10.1006/jmva.1999.1873
Hestenes, M. R. and Stiefel, E. (1952). Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand., B Math. Sci. 49 409–436. MR0060307 0048.09901 10.6028/jres.049.044Hestenes, M. R. and Stiefel, E. (1952). Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand., B Math. Sci. 49 409–436. MR0060307 0048.09901 10.6028/jres.049.044
Horowitz, J. L. (1998). Bootstrap methods for median regression models. Econometrica 66 1327–1351. 1056.62517 10.2307/2999619Horowitz, J. L. (1998). Bootstrap methods for median regression models. Econometrica 66 1327–1351. 1056.62517 10.2307/2999619
Huang, Z., Wang, L., Yi, K. and Liu, Y. (2011). Sampling based algorithms for quantile computation in sensor networks. In Proceedings of the ACM SIGMOD International Conference on Management of Data.Huang, Z., Wang, L., Yi, K. and Liu, Y. (2011). Sampling based algorithms for quantile computation in sensor networks. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
Johnson, R. and Zhang, T. (2013). Accelerating stochastic gradient descent using predictive variance reduction. In Proceedings of the Advances in Neural Information Processing Systems.Johnson, R. and Zhang, T. (2013). Accelerating stochastic gradient descent using predictive variance reduction. In Proceedings of the Advances in Neural Information Processing Systems.
Jordan, M. I., Lee, J. D. and Yang, Y. (2018). Communication-efficient distributed statistical inference. J. Amer. Statist. Assoc. To appear. 1420.62097 10.1080/01621459.2018.1429274Jordan, M. I., Lee, J. D. and Yang, Y. (2018). Communication-efficient distributed statistical inference. J. Amer. Statist. Assoc. To appear. 1420.62097 10.1080/01621459.2018.1429274
Kleiner, A., Talwalkar, A., Sarkar, P. and Jordan, M. I. (2014). A scalable bootstrap for massive data. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 795–816.Kleiner, A., Talwalkar, A., Sarkar, P. and Jordan, M. I. (2014). A scalable bootstrap for massive data. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 795–816.
Koenker, R., Chernozhukov, V., He, X. and Peng, L., eds. (2018). Handbook of Quantile Regression. Chapman & Hall/CRC Handbooks of Modern Statistical Methods. CRC Press, Boca Raton, FL.Koenker, R., Chernozhukov, V., He, X. and Peng, L., eds. (2018). Handbook of Quantile Regression. Chapman & Hall/CRC Handbooks of Modern Statistical Methods. CRC Press, Boca Raton, FL.
Kong, E., Linton, O. and Xia, Y. (2013). Global Bahadur representation for nonparametric censored regression quantiles and its applications. Econometric Theory 29 941–968. 1290.62035 10.1017/S0266466612000813Kong, E., Linton, O. and Xia, Y. (2013). Global Bahadur representation for nonparametric censored regression quantiles and its applications. Econometric Theory 29 941–968. 1290.62035 10.1017/S0266466612000813
Lee, J. D., Liu, Q., Sun, Y. and Taylor, J. E. (2017). Communication-efficient sparse regression. J. Mach. Learn. Res. 18 1–30. MR3625709 1434.62157Lee, J. D., Liu, Q., Sun, Y. and Taylor, J. E. (2017). Communication-efficient sparse regression. J. Mach. Learn. Res. 18 1–30. MR3625709 1434.62157
Leng, C. and Tong, X. (2014). Censored quantile regression via Box–Cox transformation under conditional independence. Statist. Sinica 24 221–249. 1285.62043Leng, C. and Tong, X. (2014). Censored quantile regression via Box–Cox transformation under conditional independence. Statist. Sinica 24 221–249. 1285.62043
Luo, X., Huang, C.-Y. and Wang, L. (2013). Quantile regression for recurrent gap time data. Biometrics 69 375–385. 1274.62832 10.1111/biom.12010Luo, X., Huang, C.-Y. and Wang, L. (2013). Quantile regression for recurrent gap time data. Biometrics 69 375–385. 1274.62832 10.1111/biom.12010
Manku, G. S., Rajagopalan, S. and Lindsay, B. G. (1998). Approximate medians and other quantiles in one pass and with limited memory. In Proceedings of the ACM SIGMOD International Conference on Management of Data.Manku, G. S., Rajagopalan, S. and Lindsay, B. G. (1998). Approximate medians and other quantiles in one pass and with limited memory. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
Munro, J. I. and Paterson, M. S. (1980). Selection and sorting with limited storage. Theoret. Comput. Sci. 12 315–323. 0441.68067 10.1016/0304-3975(80)90061-4Munro, J. I. and Paterson, M. S. (1980). Selection and sorting with limited storage. Theoret. Comput. Sci. 12 315–323. 0441.68067 10.1016/0304-3975(80)90061-4
Pang, L., Lu, W. and Wang, H. J. (2012). Variance estimation in censored quantile regression via induced smoothing. Comput. Statist. Data Anal. 56 785–796. 1243.62057 10.1016/j.csda.2010.10.018Pang, L., Lu, W. and Wang, H. J. (2012). Variance estimation in censored quantile regression via induced smoothing. Comput. Statist. Data Anal. 56 785–796. 1243.62057 10.1016/j.csda.2010.10.018
Portnoy, S. and Koenker, R. (1997). The Gaussian hare and the Laplacian tortoise: Computability of squared-error versus absolute-error estimators. Statist. Sci. 12 279–300. 0955.62608 10.1214/ss/1030037960 euclid.ss/1030037960Portnoy, S. and Koenker, R. (1997). The Gaussian hare and the Laplacian tortoise: Computability of squared-error versus absolute-error estimators. Statist. Sci. 12 279–300. 0955.62608 10.1214/ss/1030037960 euclid.ss/1030037960
Rajagopal, J., Wainwright, M. and Varaiya, P. (2006). Universal quantile estimation with feedback in the communication-constrained setting. In Proceedings of the IEEE International Symposium on Information Theory.Rajagopal, J., Wainwright, M. and Varaiya, P. (2006). Universal quantile estimation with feedback in the communication-constrained setting. In Proceedings of the IEEE International Symposium on Information Theory.
Shamir, O., Srebro, N. and Zhang, T. (2014). Communication efficient distributed optimization using an approximate Newton-type method. In Proceedings of the International Conference on Machine Learning.Shamir, O., Srebro, N. and Zhang, T. (2014). Communication efficient distributed optimization using an approximate Newton-type method. In Proceedings of the International Conference on Machine Learning.
Sherwood, B., Wang, L. and Zhou, X.-H. (2013). Weighted quantile regression for analyzing health care cost data with missing covariates. Stat. Med. 32 4967–4979. MR3127188 10.1002/sim.5883Sherwood, B., Wang, L. and Zhou, X.-H. (2013). Weighted quantile regression for analyzing health care cost data with missing covariates. Stat. Med. 32 4967–4979. MR3127188 10.1002/sim.5883
Shi, C., Lu, W. and Song, R. (2017). A massive data framework for M-estimators with cubic-rate. J. Amer. Statist. Assoc. To appear. 1409.62105 10.1080/01621459.2017.1360779Shi, C., Lu, W. and Song, R. (2017). A massive data framework for M-estimators with cubic-rate. J. Amer. Statist. Assoc. To appear. 1409.62105 10.1080/01621459.2017.1360779
Shrivastava, N., Buragohain, C., Agrawal, D. and Suri, S. (2004). Medians and beyond: New aggregation techniques for sensor networks. In Proceedings of the International Conference on Embedded Networked Sensor Systems.Shrivastava, N., Buragohain, C., Agrawal, D. and Suri, S. (2004). Medians and beyond: New aggregation techniques for sensor networks. In Proceedings of the International Conference on Embedded Networked Sensor Systems.
Siegmund, D. (1969). On moments of the maximum of normed partial sums. Ann. Math. Stat. 40 527–531. 0177.21704 10.1214/aoms/1177697720 euclid.aoms/1177697720Siegmund, D. (1969). On moments of the maximum of normed partial sums. Ann. Math. Stat. 40 527–531. 0177.21704 10.1214/aoms/1177697720 euclid.aoms/1177697720
Volgushev, S., Chao, S.-K. and Cheng, G. (2018). Distributed inference for quantile regression processes. Ann. Statist. To appear. 1418.62174 10.1214/18-AOS1730 euclid.aos/1550026852Volgushev, S., Chao, S.-K. and Cheng, G. (2018). Distributed inference for quantile regression processes. Ann. Statist. To appear. 1418.62174 10.1214/18-AOS1730 euclid.aos/1550026852
Volgushev, S., Wagener, J. and Dette, H. (2014). Censored quantile regression processes under dependence and penalization. Electron. J. Stat. 8 2405–2447. 1349.62488 10.1214/14-EJS54Volgushev, S., Wagener, J. and Dette, H. (2014). Censored quantile regression processes under dependence and penalization. Electron. J. Stat. 8 2405–2447. 1349.62488 10.1214/14-EJS54
Wang, X. and Dunson, D. B. (2014). Parallelizing MCMC via Weierstrass sampler. Technical report. Preprint. Available at arXiv:1312.4605. 1312.4605Wang, X. and Dunson, D. B. (2014). Parallelizing MCMC via Weierstrass sampler. Technical report. Preprint. Available at arXiv:1312.4605. 1312.4605
Wang, H. J., Stefanski, L. A. and Zhu, Z. (2012). Corrected-loss estimation for quantile regression with covariate measurement errors. Biometrika 99 405–421. 1239.62047 10.1093/biomet/ass005Wang, H. J., Stefanski, L. A. and Zhu, Z. (2012). Corrected-loss estimation for quantile regression with covariate measurement errors. Biometrika 99 405–421. 1239.62047 10.1093/biomet/ass005
Wang, H. J. and Wang, L. (2009). Locally weighted censored quantile regression. J. Amer. Statist. Assoc. 104 1117–1128. 1388.62289 10.1198/jasa.2009.tm08230Wang, H. J. and Wang, L. (2009). Locally weighted censored quantile regression. J. Amer. Statist. Assoc. 104 1117–1128. 1388.62289 10.1198/jasa.2009.tm08230
Wang, L., Wu, Y. and Li, R. (2012). Quantile regression for analyzing heterogeneity in ultra-high dimension. J. Amer. Statist. Assoc. 107 214–222. 1328.62468 10.1080/01621459.2012.656014Wang, L., Wu, Y. and Li, R. (2012). Quantile regression for analyzing heterogeneity in ultra-high dimension. J. Amer. Statist. Assoc. 107 214–222. 1328.62468 10.1080/01621459.2012.656014
Wang, L., Luo, G., Yi, K. and Cormode, G. (2013). Quantiles over data streams: An experimental study. In Proceedings of the ACM SIGMOD International Conference on Management of Data.Wang, L., Luo, G., Yi, K. and Cormode, G. (2013). Quantiles over data streams: An experimental study. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
Wang, X., Guo, F., Heller, K. and Dunson, D. (2015). Parallelizing MCMC with random partition trees. In Proceedings of the Advances in Neural Information Processing Systems.Wang, X., Guo, F., Heller, K. and Dunson, D. (2015). Parallelizing MCMC with random partition trees. In Proceedings of the Advances in Neural Information Processing Systems.
Wang, J., Kolar, M., Srebro, N. and Zhang, T. (2017). Efficient distributed learning with sparsity. In: Proceedings of the International Conference on Machine Learning.Wang, J., Kolar, M., Srebro, N. and Zhang, T. (2017). Efficient distributed learning with sparsity. In: Proceedings of the International Conference on Machine Learning.
Whang, Y.-J. (2006). Smoothed empirical likelihood methods for quantile regression models. Econometric Theory 22 173–205. 1138.62017 10.1017/S0266466606060087Whang, Y.-J. (2006). Smoothed empirical likelihood methods for quantile regression models. Econometric Theory 22 173–205. 1138.62017 10.1017/S0266466606060087
Wu, Y., Ma, Y. and Yin, G. (2015). Smoothed and corrected score approach to censored quantile regression with measurement errors. J. Amer. Statist. Assoc. 110 1670–1683. 1373.62164 10.1080/01621459.2014.989323Wu, Y., Ma, Y. and Yin, G. (2015). Smoothed and corrected score approach to censored quantile regression with measurement errors. J. Amer. Statist. Assoc. 110 1670–1683. 1373.62164 10.1080/01621459.2014.989323
Xu, G., Sit, T., Wang, L. and Huang, C.-Y. (2017). Estimation and inference of quantile regression for survival data under biased sampling. J. Amer. Statist. Assoc. 112 1571–1586. MR3750882 10.1080/01621459.2016.1222286Xu, G., Sit, T., Wang, L. and Huang, C.-Y. (2017). Estimation and inference of quantile regression for survival data under biased sampling. J. Amer. Statist. Assoc. 112 1571–1586. MR3750882 10.1080/01621459.2016.1222286
Yu, L., Lin, N. and Wang, L. (2017). A parallel algorithm for large-scale nonconvex penalized quantile regression. J. Comput. Graph. Statist. 26 935–939.Yu, L., Lin, N. and Wang, L. (2017). A parallel algorithm for large-scale nonconvex penalized quantile regression. J. Comput. Graph. Statist. 26 935–939.
Zhang, Y., Duchi, J. and Wainwright, M. (2015). Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates. J. Mach. Learn. Res. 16 3299–3340. 1351.62142Zhang, Y., Duchi, J. and Wainwright, M. (2015). Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates. J. Mach. Learn. Res. 16 3299–3340. 1351.62142
Zhang, Q. and Wang, W. (2007). A fast algorithm for approximate quantiles in high speed data streams. In Proceedings of the International Conference on Scientific and Statistical Database Management.Zhang, Q. and Wang, W. (2007). A fast algorithm for approximate quantiles in high speed data streams. In Proceedings of the International Conference on Scientific and Statistical Database Management.
Zhao, T., Cheng, G. and Liu, H. (2016). A partially linear framework for massive heterogeneous data. Ann. Statist. 44 1400–1437. 1358.62050 10.1214/15-AOS1410 euclid.aos/1467894703Zhao, T., Cheng, G. and Liu, H. (2016). A partially linear framework for massive heterogeneous data. Ann. Statist. 44 1400–1437. 1358.62050 10.1214/15-AOS1410 euclid.aos/1467894703
Zheng, S. (2011). Gradient descent algorithms for quantile regression with smooth approximation. International Journal of Machine Learning and Cybernetics 2 191.Zheng, S. (2011). Gradient descent algorithms for quantile regression with smooth approximation. International Journal of Machine Learning and Cybernetics 2 191.
Zheng, Q., Peng, L. and He, X. (2018). High dimensional censored quantile regression. Ann. Statist. 46 308–343. 1416.62236 10.1214/17-AOS1551 euclid.aos/1519268432Zheng, Q., Peng, L. and He, X. (2018). High dimensional censored quantile regression. Ann. Statist. 46 308–343. 1416.62236 10.1214/17-AOS1551 euclid.aos/1519268432