Speaker: Dr. Tim Rakow, King's College

Title:  

Replication, replication, replication: Psychology’s open secret from its closed era of research.

Host: Mikhail Spektor/Manos Konstantinidis

Abstract:

Everyone knows that psychology has long suffered from an impoverished replication culture, which until recently discouraged replication studies and made it difficult to publish replication research. Indicative of this, from database searches of psychology’s top-100 journals, Makel et al (2012) estimated that 2-3% of psychology articles describe or discuss replication and only 1-2% of psychology articles report new replication research. In seven new studies, including two pre-registered replications, I re-evaluated psychology’s replication practices in the era preceding the current era of Open Science. I found that over 50% of the 1,833 articles inspected explicitly described or discussed replication. Brief coding all articles doing so identified 20% as a conservative lower-bound estimate of the rate of psychology research articles that report new studies replicating prior methods or findings. If diversity in the terms used to report replication is accepted, it may be that as many as 40-50% of articles report some replication of previous research. A failure akin to poor model recovery explains the discrepancy between previous prevalence estimates and mine: put simply, database searches are ineffective for identifying which articles report replication. I conclude that researchers can be a little more upbeat about their past research practices but should be a little more downbeat about how much research might be missed by meta-analyses and systematic reviews.