Stochastic Natural Gradient EP @ NIPS

I will present our work on distributed Bayesian learning with stochastic natural gradient EP at the Advances in Approximate Bayesian Inference Workshop and the Bayesian Deep Learning Workshop at NIPS 2016. It’s joint work with Stefan Webb, Thibaut Lienart, Sebastian Vollmer (all Oxford), Balaji Lakshminarayanan, Charles Blundell (both Google DeepMind) and Yee Whye Teh (Oxford & Google DeepMind). Here is a link to our arxiv preprint.

Here’s the abstract:

This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not require any simplifying assumptions on the distribution of interest, beyond the existence of some Monte Carlo sampler for estimating the moments of the EP tilted distributions. Further, as opposed to EP which has no guarantee of convergence, SNEP can be shown to be convergent, even when using Monte Carlo moment estimates. Secondly, we propose a novel architecture for distributed Bayesian learning which we call the posterior server. The posterior server allows scalable and robust Bayesian learning in cases where a dataset is stored in a distributed manner across a cluster, with each compute node containing a disjoint subset of data. An independent Monte Carlo sampler is run on each compute node, with direct access only to the local data subset, but which targets an approximation to the global posterior distribution given all data across the whole cluster. This is achieved by using a distributed asynchronous implementation of SNEP to pass messages across the cluster. We demonstrate SNEP and the posterior server on distributed Bayesian learning of logistic regression and neural networks.

SNEP is a neat extension of EP that lends itself well to distributed computing. It is both interesting as a variational inference method and shows great potential in the context of Bayesian neural networks. In our experiments SNEP is roughly as fast as existing distributed SGD methods but has the advantage of being Bayesian. We’ve recently released the code at https://github.com/BigBayes/PosteriorServer. Check it out!

Relativistic Monte Carlo @ BDL2016

Our work (joint work with Xiaoyu Lu, Valerio Perrone, Sebastian Vollmer and Yee Whye Teh) on relativistic Monte Carlo was accepted as a contributed talk at the Bayesian Deep Learning Workshop at NIPS 2016. Come along at 11:20am on December 10 when Valerio and Xiaoyu will be presenting our work.

We are excited about Relativistic Monte Carlo as a way to derive an MCMC algorithm more robust to hyperparameter choice. Here is the abstract:

Hamiltonian Monte Carlo (HMC) is a popular Markov chain Monte Carlo (MCMC) algorithm that generates proposals for a Metropolis-Hastings algorithm by simulating the dynamics of a Hamiltonian system. However, HMC is sensitive to large time discretizations and performs poorly if there is a mismatch between the spatial geometry of the target distribution and the scales of the momentum distribution. In particular the mass matrix of HMC is hard to tune well. In order to alleviate these problems we propose relativistic Hamiltonian Monte Carlo, a version of HMC based on relativistic dynamics that introduce a maximum velocity on particles. We also derive stochastic gradient versions of the algorithm and show that the resulting algorithms bear interesting relationships to gradient clipping, RMSprop, Adagrad and Adam, popular optimisation methods in deep learning. Based on this, we develop relativistic stochastic gradient descent by taking the zero-temperature limit of relativistic stochastic gradient Hamiltonian Monte Carlo. In experiments we show that the relativistic algorithms perform better than classical Newtonian variants and Adam.