Skip to main content Skip to navigation


GPUs in Computational Statistics

University of Warwick, 25 January, 2012

Organisers: Zoubin Gharamani, Jim Griffin, Anthony Lee, Andrew Liddle, David Wild

All talks will take place in CS1.04/5, Computer Science Building




Introduction - Anthony Lee


Christian Robert - Vanilla Rao-Blackwellisations of Metropolis-Hastings algorithms


Christophe Andrieu - Exact approximations seeking GPUs


Buffet Lunch


Pierre Jacob - Parallel Wang-Landau for automated density exploration


Chris Holmes - Exploratory analysis of regions of genetic association signal using Bayesian adaptive regression methods on GPUs




Chris Barnes - Accelerating inference for dynamical models using GPUs


Michael Stumpf - Statistical analysis of network data and evolution on GPUs


Workshop Ends / Wine and Cheese Reception



Christian Robert - Vanilla Rao-Blackwellisations of Metropolis-Hastings algorithms

Casella and Robert (1996, Biometrika) presented a general Rao-Blackwell principle for accept-reject and Metropolis-Hastings schemes that leads to significant decreases in the variance of the resulting estimators, but at a high cost in computing and storage. Adopting a completely different perspective, we introduce instead a universal scheme that guarantees variance reductions in all Metropolis-Hastings based estimators while keeping the computing cost under control. The principle relates to the availability of an unbiased estimator of the acceptance probability. In a second if related part, we consider the implications of the fact that parallel raw-power can be exploited by a generic Metropolis--Hastings algorithm if the proposed values are independent. In particular, we present improvements to the independent Metropolis--Hastings algorithm that significantly decrease the variance of any estimator derived from the MCMC output, for a null computing cost since those improvements are based on a fixed number of target density evaluations. Furthermore, those techniques do not jeopardize the Markovian convergence properties of the algorithm, since they are based on the Rao--Blackwell principles of Gelfand and Smith (1990), already exploited in Casella and Robert (1996). We illustrate those improvements both on a toy normal example and on a classical probit regression model, but stress the fact that they are applicable in any case where the independent Metropolis-Hastings is applicable. Extensions to the random walk Metropolis--Hastings algorithm will also be discussed.

These are joint works with Randal Douc (Paristech-Telecom), available as v2 and Pierre Jacob (Paris-Dauphine & CREST) and Murray Smith (NIWA, NZ), available as

Christophe Andrieu - Exact approximations seeking GPUs

Exact approximations of MCMC algorithms aim to approximate "idealised", that is efficient but difficult or impossible to implement, algorithms. One of their characteristic is that where, for example, a standard Metropolis-Hastings update would require one proposed transition at each iteration of the Markov chain, exact approximations often require the simulation of multiple potential transitions, and therefore may lend themselves naturally to parallel computations.

The aim of the presentation is to provide an overview of these approaches in various contexts and discuss some of their known properties, which will hopefully generate an interest among specialists of GPU programming, and more generally parallel computing techniques.

Pierre Jacob - Parallel Wang-Landau for automated density exploration

The Wang-Landau algorithm is an adaptive MCMC algorithm which generates a Markov chain designed to move efficiently in the state space, by constantly penalizing already-visited regions. It hence falls into the class of exploratory algorithms, especially when the chosen regions correspond to different levels of density values. We explore the extension of this algorithm to parallel chains, where the chains are still dependent one on the other through a common component. That component essentially represents the history of already-visited regions, computed on all the chains. We show the benefit of using parallel chains even if a single processing unit is available, in terms of stabilization of the schedule used in the adaptation process. Finally we discuss some foreseen benefits of using GPUs when available.

Chris Holmes - Exploratory analysis of regions of genetic association signal using Bayesian adaptive regression methods on GPUs

We show how the use of GPU technologies allow for efficient exploration of disease association signals arising in modern genetic epidemiology. In particular we introduce a scale-mixture ``sparsity'' prior on logistic regression model coefficients and show how by sweeping through the prior parameter governing the extent of shrinkage we map through a path of models of increasing complexity. This allows for visual interpretation and quantification of multiple predictor effects. The computational algorithm using SMC samplers would be prohibitively expensive to run on single-threaded computing, but is ideally suited to GPU implementation.

Chris Barnes - Accelerating inference for dynamical models using GPUs

Mathematical modelling has become ubiquitous in systems and synthetic biology and there are many tools available to model biological systems. These models are usually of high dimension with many unknown parameters. This can be problematic since measuring kinetic parameter values in vivo is difficult and interrogated parameter values from experiments performed in vitro are often not applicable. Therefore, in a growing number of cases, the parameter values must be inferred from data such as time series measurements. Additionally there is often a set of competing models proposed to describe a process and the most appropriate model must be chosen given a set of measurements. Sequential Monte Carlo (SMC) is one approach that can be applied to the problem of inference in dynamical models. Combined with approximate Bayesian computation (ABC) these methods can be used in both deterministic and stochastic systems. Here I will show how SMC methods can take advantage of the highly parallel architecture of Graphics Processing Units (GPUs) and demonstrate their utility in examples from biochemical signaling and the design of synthetic biological systems.

Michael Stumpf - Statistical analysis of network data and evolution on GPUs

Network analysis typically involves as set of repetitive tasks that are particularly amenable to poor-man's parallelization. This is therefore an ideal application are for GPU architectures, which help to alleviate the tedium inherent to statistically sound analysis of network data. Here we will illustrate the use of GPUs in a range of applications, which include percolation processes on networks, the evolution of protein-protein interaction networks, and the fusion of different types of biomedical and disease data in the context of molecular interaction networks. We will pay particular attention to the numerical performance of different routines that are frequently invoked in network analysis problems.

Workshop Home