Translator Disclaimer
August 2021 Infinite-dimensional gradient-based descent for alpha-divergence minimisation
Kamélia Daudel, Randal Douc, François Portier
Author Affiliations +
Ann. Statist. 49(4): 2250-2270 (August 2021). DOI: 10.1214/20-AOS2035

Abstract

This paper introduces the (α,Γ)-descent, an iterative algorithm which operates on measures and performs α-divergence minimisation in a Bayesian framework. This gradient-based procedure extends the commonly-used variational approximation by adding a prior on the variational parameters in the form of a measure. We prove that for a rich family of functions Γ, this algorithm leads at each step to a systematic decrease in the α-divergence and derive convergence results. Our framework recovers the Entropic Mirror Descent algorithm and provides an alternative algorithm that we call the Power Descent. Moreover, in its stochastic formulation, the (α,Γ)-descent allows to optimise the mixture weights of any given mixture model without any information on the underlying distribution of the variational parameters. This renders our method compatible with many choices of parameters updates and applicable to a wide range of Machine Learning tasks. We demonstrate empirically on both toy and real-world examples the benefit of using the Power Descent and going beyond the Entropic Mirror Descent framework, which fails as the dimension grows.

Citation

Download Citation

Kamélia Daudel. Randal Douc. François Portier. "Infinite-dimensional gradient-based descent for alpha-divergence minimisation." Ann. Statist. 49 (4) 2250 - 2270, August 2021. https://doi.org/10.1214/20-AOS2035

Information

Received: 1 May 2020; Revised: 1 October 2020; Published: August 2021
First available in Project Euclid: 29 September 2021

Digital Object Identifier: 10.1214/20-AOS2035

Subjects:
Primary: 62F15
Secondary: 62F30 , 62F35 , 62G07 , 62L99

Keywords: alpha-divergence , Kullback–Leibler divergence , Mirror Descent , variational inference

Rights: Copyright © 2021 Institute of Mathematical Statistics

JOURNAL ARTICLE
21 PAGES

This article is only available to subscribers.
It is not available for individual sale.
+ SAVE TO MY LIBRARY

SHARE
Vol.49 • No. 4 • August 2021
Back to Top