Abstract: Variable selection can be naturally seen as a model selection problem where the entertained models differ in which subset of variables explains the outcome of interest. In this setting, posterior probabilities of the models are a simple combination of Bayes factors, a well-known inferential tool that is key in the formal Bayesian approach to testing and model choice. This approach to variable selection automatically provides sparse answers that, quite importantly, are accompanied with probabilistic assessments regarding the confidence we have in them. This methodology is, however, not exempt from difficulties including prior elicitation and numerical challenges related with its practical implementation. Particularly in the context of linear and generalized linear models, the so-called g-priors have attracted the interest of many researchers due to their appealing properties formally described in [2]. In this chapter we review the main aspects concerned with the implementation of the Bayesian approach to variable selection based on Bayes factors in linear and generalized linear models, using g-priors. The material presented here has a clear focus on applicability and emphasis is placed on providing: i) practical guides for implementation, including documentation for the use of R packages (particularly BayesVarSel and glmBfp) and ii) the analysis of real examples which illustrate the enormous potential of this approach to variable selection.
Model Averaging and its Use in Economics, Journal of Economic Literature, 58, (2020), 644-719.
Abstract: The method of model averaging has become an important tool to deal with model uncertainty, for example in situations where a large amount of different theories exist, as are common in economics. Model averaging is a natural and formal response to model uncertainty in a Bayesian framework, and most of the paper deals with Bayesian model averaging. The important role of the prior assumptions in these Bayesian procedures is highlighted. In addition, frequentist model averaging methods are also discussed. Numerical methods to implement these methods are explained, and I point the reader to some freely available computational resources. The main focus is on uncertainty regarding the choice of covariates in normal linear regression models, but the paper also covers other, more challenging, settings, with particular emphasis on sampling models commonly used in economics. Applications of model averaging in economics are reviewed and discussed in a wide range of areas, among which growth economics, production modelling, finance and forecasting macroeconomic quantities.
Methods and Tools for Bayesian Variable Selection and Model Averaging in Normal Linear Regression, with Anabel Forte and Gonzalo Garcia-Donato; International Statistical Review, 86, (2018), 237-258.
Abstract: In this paper we briefly review the main methodological aspects concerned with the application of the Bayesian approach to model choice and model averaging in the context of variable selection in regression models. This includes prior elicitation, summaries of the posterior distribution and computational strategies. We then examine and compare various publicly available R-packages, summarizing and explaining the differences between packages and giving recommendations for applied users. We find that all packages reviewed (can) lead to very similar results, but there are potentially important differences in flexibility and efficiency of the packages.
Bayesian Model Averaging, Wiley StatsRef: Statistics Reference Online, N. Balakrishnan, Paolo Brandimarte, Brian Everitt, Geert Molenberghs, Walter Piegorsch and Fabrizio Ruggeri, eds. (2016)
Abstract: Bayesian Model averaging is a natural response to model uncertainty. It has become an important practical tool for dealing with model uncertainty, in particular in empirical settings with large numbers of potential models and relatively limited numbers of observations. Most of this paper focuses on the problem of variable selection in normal linear regression models, but also briefly considers a more general setting. The article surveys a number of prior structures proposed in the literature, and presents in some detail the most commonly used prior setup. Model selection consistency and practical implementation are also briefly discussed.
Mixtures of g-priors for Bayesian Model Averaging with Economic Applications ; Journal of Econometrics, 171, (2012), 251-266, with Eduardo Ley.
Abstract: We examine the issue of variable selection in linear regression modeling, where we have a potentially large amount of possible covariates and economic theory offers insufficient guidance on how to select the appropriate subset. Bayesian Model Averaging presents a formal Bayesian solution to dealing with model uncertainty. Our main interest here is the effect of the prior on the results, such as posterior inclusion probabilities of regressors and predictive performance. We combine a Binomial-Beta prior on model size with a g-prior on the coefficients of each model. In addition, we assign a hyperprior to g, as the choice of g has been found to have a large impact on the results. For the prior on g, we examine the Zellner-Siow prior and a class of Beta shrinkage priors, which covers most choices in the recent literature. We propose a benchmark Beta prior, inspired by earlier findings with fixed g, and show it leads to consistent model selection. The effect of this prior structure on penalties for complexity and lack of fit is described in some detail. Inference is conducted through a Markov chain Monte Carlo sampler over model space and g. We examine the performance of the various priors in the context of simulated and real data. For the latter, we consider two important applications in economics, namely cross-country growth regression and returns to schooling. Recommendations to applied users are provided.
Bayesian Model Averaging and forecasting, invited paper for the Special Issue (number 200) of the Bulletin of E.U. and U.S. Inflation and Macroeconomic Analysis, (2011), 30-41.
Abstract: This paper focuses on the problem of variable selection in linear regression models. I briefly review the method of Bayesian model averaging, which has become an important tool in empirical settings with large numbers of potential regressors and relatively limited numbers of observations. Some of the literature is discussed with particular emphasis on forecasting in economics. The role of the prior assumptions in these procedures is highlighted, and some recommendations for applied users are given.
On the effect of prior assumptions in Bayesian Model Averaging with applications to growth regression; Journal of Applied Econometrics, 24, (2009), 651-674, with Eduardo Ley.
Abstract: We consider the problem of variable selection in linear regression models. Bayesian model averaging has become an important tool in empirical settings with large numbers of potential regressors and relatively limited numbers of observations. We examine the effect of a variety of prior assumptions on the inference concerning model size, posterior inclusion probabilities of regressors and on predictive performance. We illustrate these issues in the context of crosscountry growth regressions using three datasets with 41 to 67 potential drivers of growth and 72 to 93 observations. Finally, we recommend priors for use in this and related contexts.
Comments on `Jointness of growth determinants'; Journal of Applied Econometrics, 24, (2009), 248-251, with Eduardo Ley.
Abstract: We consider the measures of jointness proposed by Doppelhofer and Weeks (2009) and Strachan (2009) inthe context of variable selection. Using the general criteria suggested in Ley and Steel (2007), we identifysome shortcomings of these measures, which are illustrated with empirically relevant example cases. Weargue that careful examination of the properties of any jointness measure is critical before using it to informdecisions, and favour the use of the measures proposed in Ley and Steel (2007).
Jointness in Bayesian Variable Selection With Applications to Growth Regression; Journal of Macroeconomics, 29, (2007), 476-493, with Eduardo Ley.
Abstract: We present a measure of jointness to explore dependence among regressors, in the context of Bayesian model selection. The jointness measure proposed here equals the posterior odds ratio between those models that include a set of variables and the models that only include proper subsets. We illustrate its application in cross-country growth regressions using two datasets from Fernandez et al.(2001) and Sala-i-Martin et al.(2004).
Bayesian Modeling of Catch in a Northwest Atlantic Fishery; Journal of the Royal Statistical Society, Series C (Applied Statistics), 51, (2002), 257-280, with Carmen Fernández and Eduardo Ley.
Abstract: We model daily catches of fishing boats in the Grand Bank fishing grounds. We use data on catches per species for a number of vessels collected by the European Union in the context of the Northwest Atlantic Fisheries Organization. Many variables can be thought to influence the amount caught: a number of ship characteristics (such as the size of the ship, the fishing technique used, the mesh size of the nets, etc.), are obvious candidates, but one can also consider the season or the actual location of the catch. Our database leads to 28 possible regressors (arising from six continuous variables and four categorical variables, whose 22 levels are treated separately), resulting in a set of 177 million possible linear regression models for the log of catch. Zero observations are modelled separately through a probit model. Inference is based on Bayesian model averaging, using a Markov chain Monte Carlo approach. Particular attention is paid to prediction of catch for single and aggregated ships.
Benchmark Priors for Bayesian Model Averaging; Journal of Econometrics, 100, (2001), 381-427, with Carmen Fernández and Eduardo Ley
Abstract: In contrast to a posterior analysis given a particular sampling model, posterior model probabilities in the context of model uncertainty are typically rather sensitive to the specification of the prior. In particular, "diffuse'' priors on model-specific parameters can lead to quite unexpected consequences. Here we focus on the practically relevant situation where we need to entertain a (large) number of sampling models and we have (or wish to use) little or no subjective prior information. We aim at providing an "automatic'' or "benchmark'' prior structure that can be used in such cases. We focus on the Normal linear regression model and propose a partly noninformative prior structure based on a Natural Conjugate g-prior specification. We limit the amount of subjective information requested from the user to the choice of a single scalar hyperparameter g0j. The consequences of different choices for g0j are examined. We investigate theoretical properties, such as consistency of the implied Bayesian procedure. Links with classical information criteria are provided. In addition, we examine the finite sample properties of several choices for g0j in a simulation study. The use of the MC3 algorithm of Madigan and York (1995), combined with efficient coding in Fortran, makes it feasible to conduct large simulations. In addition to posterior criteria, we shall also compare the predictive performance of different priors. A classic example concerning the economics of crime will also be provided and contrasted with results in the literature. The main findings of the paper will lead us to propose a "benchmark'' prior specification in a linear regression context with model uncertainty.
Model Uncertainty in Cross-Country Growth Regressions; Journal of Applied Econometrics, 16, (2001), 563-576, with Carmen Fernández and Eduardo Ley .
Abstract: We investigate the issue of model uncertainty in cross-country growth regressions using Bayesian Model Averaging (BMA). We find that the posterior probability is very spread among many models suggesting the superiority of BMA over choosing any single model. Out-of-sample predictive results support this claim. In contrast with Levine and Renelt (1992), our results broadly support the more "optimistic'' conclusion of Sala-i-Martin (1997b), namely that some variables are important regressors for explaining cross-country growth patterns. However, care should be taken in the methodology employed. The approach proposed here is firmly grounded in statistical theory and immediately leads to posterior and predictive inference.
Links
A nice introduction to BMA methods: Jennifer A. Hoeting, David Madigan, Adrian E. Raftery and Chris T. Volinsky (1999), Bayesian Model Averaging: A Tutorial; Statistical Science, 14:4, 382–417. [PDF]
Very useful and freely available R code (possibly with Matlab interface) by Martin Feldkircher and Stefan Zeugner for Bayesian Model Averaging. They also have a more general resource page for BMA.
Chris Papageorgiou and myself have edited a special issue on "Model Uncertainty in Economics" of the European Economic Review (Vol. 81, 2016) in Eduardo's honour.
Together with Gonzalo Garcia-Donato, Jan Magnus and Xinyu Zhang, I edited a special issue of Econometrics on "Bayesian and Frequentist Model Averaging". Papers are available here.
