Open Access
2017 Sketching meets random projection in the dual: A provable recovery algorithm for big and high-dimensional data
Jialei Wang, Jason D. Lee, Mehrdad Mahdavi, Mladen Kolar, Nathan Srebro
Electron. J. Statist. 11(2): 4896-4944 (2017). DOI: 10.1214/17-EJS1334SI


Sketching techniques scale up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, without sacrificing their statistical properties. In this paper, we study sketching from an optimization point of view. We first show that the iterative Hessian sketch is an optimization process with preconditioning and develop an accelerated version using this insight together with conjugate gradient descent. Next, we establish a primal-dual connection between the Hessian sketch and dual random projection, which allows us to develop an accelerated iterative dual random projection method by applying the preconditioned conjugate gradient descent on the dual problem. Finally, we tackle the problems of large sample size and high-dimensionality in massive data sets by developing the primal-dual sketch. The primal-dual sketch iteratively sketches the primal and dual formulations and requires only a logarithmic number of calls to solvers of small sub-problems to recover the optimum of the original problem up to arbitrary precision. Our iterative sketching techniques can also be applied for solving distributed optimization problems where data are partitioned by samples or features. Experiments on synthetic and real data sets complement our theoretical results.


Download Citation

Jialei Wang. Jason D. Lee. Mehrdad Mahdavi. Mladen Kolar. Nathan Srebro. "Sketching meets random projection in the dual: A provable recovery algorithm for big and high-dimensional data." Electron. J. Statist. 11 (2) 4896 - 4944, 2017.


Received: 1 June 2017; Published: 2017
First available in Project Euclid: 15 December 2017

zbMATH: 06825036
MathSciNet: MR3738201
Digital Object Identifier: 10.1214/17-EJS1334SI

Primary: 62H12 , 68T05 , 90C06

Keywords: acceleration , conjugate gradient , dual random projection , Iterative Hessian sketch , preconditioning , primal-dual conversion , primal-dual sketch

Vol.11 • No. 2 • 2017
Back to Top