Distributed Markov Chain Monte Carlo for Bayesian Hierarchical Models

This article proposes a distributed Markov chain Monte Carlo (MCMC) algorithm for estimating Bayesian hierarchical models when the number of cross-sectional units is very large and the objects of interest are the unit-level parameters. The two-stage algorithm is asymptotically exact, retains the flexibility of a standard MCMC algorithm, and is easy to implement. The algorithm constructs an estimator of the posterior predictive distribution of the unit-level parameters in the first stage, and uses the estimator as the prior distribution in the second stage for the unit-level draws. Both stages are embarrassingly parallel. The algorithm is demonstrated with simulated data from a hierarchical logit model and is shown to be faster and more efficient (in effective sample size generated per unit of computing) than a single machine algorithm by at least an order of magnitude. For a relatively small number of observations per cross-sectional unit, the algorithm is both faster and has better mixing properties than the standard hybrid Gibbs sampler. We illustrate our approach with data on 1,100,000 donors to a charitable organization, and simulations with up to 100 million units