We consider the problem of sampling from a strongly log-concave density in , and prove an information theoretic lower bound on the number of stochastic gradient queries of the log density needed. Several popular sampling algorithms (including many Markov chain Monte Carlo methods) operate by using stochastic gradients of the log density to generate a sample; our results establish an information theoretic limit for all these algorithms.
We show that for every algorithm, there exists a well-conditioned strongly log-concave target density for which the distribution of points generated by the algorithm would be at least ε away from the target in total variation distance if the number of gradient queries is less than , where is the variance of the stochastic gradient. Our lower bound follows by combining the ideas of Le Cam deficiency routinely used in the comparison of statistical experiments along with standard information theoretic tools used in lower bounding Bayes risk functions. To the best of our knowledge our results provide the first nontrivial dimension-dependent lower bound for this problem.
We gratefully acknowledge the support of the NSF through grants IIS-1619362 and IIS-1909365. Part of this work was completed while NC was interning at Google.
We are grateful to Aditya Guntuboyina for pointing us towards the literature on Le Cam deficiency. We would also like to thank Jelena Diakonikolas, Sébastien Gerchinovitz, Michael Jordan, Aldo Pacchiano, Aaditya Ramdas and Morris Yau for many helpful conversations. We thank Kush Bhatia for helpful comments that improved the presentation of the results.
"Oracle lower bounds for stochastic gradient sampling algorithms." Bernoulli 28 (2) 1074 - 1092, May 2022. https://doi.org/10.3150/21-BEJ1377