Decision Research at Warwick (DR@W Forum)

Krefeld-Schwalb-Exposing omitted moderators- Explaining why effect sizes differ in the social sciences-2024-Proceedings of the National Academy of Sciences.pdf

Some behavioral research aims to affect policy and management practice with quickly applied insights. To do this, research findings should generalize from the settings of the original study to applications. However, research results reflect both the manipulation of interest as well as a myriad of unobserved sources of heterogeneity. To generalize, we need to understand this heterogeneity. We re-analyze existing data and simulations to propose a toolbox to help researchers leverage heterogeneity in the service of generalization. The five tools that consumer researchers might use to increase the generalizability of their findings are: 1) Measure proximal moderators that describe respondents' interaction with the setting. 2) Exploit purposive variation to increase variation on relevant moderators and settings. 3) Measure manipulation intensity and measurement error. 4) Use survey para-data to estimate moderators. 5) Harness proximal moderators and purposive variation to generalize effect sizes. We suggest that our toolbox can help advance our field towards higher practical impact by moving beyond understanding what works, to understanding what works when, where and why.