Publication and related bias in health services and delivery research (HSDR)
What is this project about?
It is important that the people who are responsible for making important decisions about the ways our health services are delivered have the best and most reliable evidence available to them to make sure that they make the right choices. Sometimes, however, the evidence available doesn’t tell the ‘full story’ or the evidence available may be biased. One problem is ‘publication bias’, which happens when only certain studies, usually those which are more exciting or are ‘positive’ are published. There are other forms of bias, too. Sometimes, researchers might not get exciting or positive results, so they use other ways to analyse the data or analyse only parts of the full data set until they get a more interesting result. This is called ‘p-hacking’. There is also ‘outcome reporting bias’, which happens when a study collects lots of data, but only the ‘positive’ or interesting results are published. In this project, we use the term ‘publication bias’ broadly to cover these various forms of bias as well. It is important to study publication bias because if the people making decisions don’t have all the evidence available to them, they might make wrong decisions, which may have a negative effect on our health services and the way the services are delivered.
In clinical research, which focuses on medical treatments for patients, there has been a lot of research on publication bias and examples such as the non-publication of data showing harmful effects from drug trials has hit the headline. It is very important to make sure that there is no publication bias as treatment decision based on biased information might be wasteful or even harmful. Because of the potentially serious consequences, all clinical trials now have to be registered so that people making decisions about treatments have all the information available to them to make the right choices.
In Health Services and Delivery Research (HSDR), which looks at the best ways our services are delivered to patients, there hasn’t been much research on publication bias. This might be because health service delivery research is very complicated. In this study, we will look at publication bias in HSDR to see if it exists, what kinds of publication bias there are, and how we can detect or deal with it. We want to obtain and then share the information about publication bias so that researchers doing HSDR and users of findings from the research can take measures to make sure that the occurrence and impact of publication bias are minimised. Investigation of publication bias in HSDR is very important because its findings are often used to make policy decisions that will influence many people.
Overview of the project
The study has five work packages (WPs), which are all linked as shown in the picture below.
The work packages include:
Work Package 1 (WP1): a systematic review of empirical and methodological studies concerning the occurrence, potential impact and/or methodology related to publication bias in HSDR and cognate fields to provide a summary of what is known from current literature.
Work Package 2 (WP2): an overview of systematic reviews of intervention and association studies in HSDR to describe current practice and potential challenges in assessing publication bias during evidence synthesis.
Work Package 3 (WP3): in-depth case studies to evaluate the applicability of different methods for detecting and mitigating publication bias in HSDR and to provide guidance for future research and practice.
Work Package 4 (WP4): a retrospective study to follow up the publication status of cohorts of HSDR studies to directly observe publication bias in HSDR
Work Package 5 (WP5): semi-structured interviews with health services researchers and commissioners, journal editors and other stakeholders to explore their perception and experience related to publication bias.
Work Package 1 (WP1): a systematic review of empirical and methodological studies concerning the occurrence, potential impact and/or methodology related to publication bias in HSDR and cognate fields to provide a summary of what is known from current literature.
Work Package 2 (WP2): an overview of systematic reviews of intervention and association studies in HSDR to describe current practice and potential challenges in assessing publication bias during evidence synthesis.
Work Package 3 (WP3): in-depth case studies to evaluate the applicability of different methods for detecting and mitigating publication bias in HSDR and to provide guidance for future research and practice.
Work Package 4 (WP4): a retrospective study to follow up the publication status of cohorts of HSDR studies to directly observe publication bias in HSDR
Work Package 5 (WP5): semi-structured interviews with health services researchers and commissioners, journal editors and other stakeholders to explore their perception and experience related to publication bias.
Project outputs
Key publications
- Ayorinde, A.A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R.J., Chen, Y-F. (2020) Assessment of publication bias and outcome reporting bias in systematic reviews of health services and delivery research: A meta-epidemiological study. PLoS ONE 15(1): https://doi.org/10.1371/journal.pone.0227580
- Williams, I., Ayorinde, A.A., Mannion, R., Song, F., Skrybant, M., Lilford, R.J., Chen, Y-F. (2020) Stakeholder views on publication bias in health services research. Journal of health services research & policy; 25(3) pp162-171. https://doi.org/10.1177/1355819620902185
- Ayorinde, A.A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R.J., Chen, Y-F. (2020) Publication and related biases in health services research: a systematic review of empirical evidence. BMC Med Res Methodol 20, 137. https://doi.org/10.1186/s12874-020-01010-1
Published abstracts
- Ayorinde AA, Williams I, Mannion R, Song F, Skrybant M, Lilford RJ, Chen Y-F. (2019) Does all health services and delivery research (HSDR) get published? A study to follow up the publication status of cohorts of HSDR. J Epidemiology Community Health 2019;73:A73. http://dx.doi.org/10.1136/jech-2019-SSMabstracts.154
- Ayorinde AA, Williams I, Mannion R, Song F, Skrybant M, Lilford RJ, Chen Y-F. (2018) Assessment of publication
bias in systematic reviews of health services and delivery research. Journal of Epidemiology and Community Health 72(Suppl 1):A79.2-A79. http://dx.doi.org/10.1136/jech-2018-SSMabstracts.167
Other outputs
- Cochrane Colloquium 2019 screencast https://www.youtube.com/watch?v=QZK9I9VVrQY&feature=youtu.be
Meet the Team
Core Research and Administrative Team:
Dr Yen-Fu Chen (University of Warwick) - Principal Investigator, evidence synthesis and project management
Dr Iestyn Williams (University of Birmingham) - indepth interviews (WP5)
Dr Abimbola Ayorinde (University of Warwick) - evidence synthesis, data collections, day to day running of the project
Ms Sian Scogings (University of Warwick) - administrative support
Project Management Team:
Prof Richard Lilford (University of Warwick) - HSDR methodology and senior management
Prof Russell Mannion (University of Birmingham) - HSDR methodology and mixed methods
Prof Fujian Song (University of East Anglia) - expertise in publication bias and statistical advice
Dr Magdalena Skrybant (University of Birmingham) - Patient and Public Involvement and Engagement (PPIE) Lead
Study Steering Committee:
Prof Stephen Sutton (University of Cambridge) - Chair of the Study Steering Committee
Revd Barry Clark (PPI representative) - patient and public opinion and input
Dr Kaveh Shojania (University of Toronto) - HSDR methodolody and journal editor perspective
Prof Jeremy Grimshaw (University of Ottawa) - HSDR methodology and evidence synthesis perspective
Prof Timothy Hofer (University of Michigan) - HSDR methodology and health professional perspective
Dr Christopher Chiswell (Birmingham Children's Hospital) - health professional and service provider perspective
Mr Tim Sacks (East Leicestershire and Rutland Clinical Commissioning Group) - service provider and management perspective)
Prof James Thomas (University College London) - evidence synthesis and policy perspective
Prof Richard Lilford (University of Warwick) - HSDR methodology and senior management
Link to NIHR project website
Project News:
Abimbola Ayorinde and Iestyn Williams presented findings of the study at Health Services Research UK Conference 2019.
We had an academic dissemination event on the 7th of June 2019. It was attended by senior researchers in HSDR and evidence synthesis, journal editors, funders and a patient and public representative. We are grateful to all the delegates for their invaluable contribution.
We presented a poster at the Society for Social Medicine Annual Scientific Meeting held in Glasgow (5th - 7th of September 2018). Please click here to view the poster which is based on Work Package 2.
A poster based on Work Package 4 of the study was presented at the AcademyHealth Annual Research Meeting held in Seattle, USA on the 24th to the 26th of June 2018. The poster can be found here.
The HSRUK Symposium, which is the 'voice of Health Services Research' was held in Nottingham on 6-7 July 2017. Researchers from our team were represented at the event. In addition to presenting a poster, which included information about our project, Yen-Fu Chen, Principal Investigator, presented in the 'Methods' section. His talk was entitled "Publication and related biases in health services and delivery research: the dog that did not bark?"
Please find the poster here
Contact us: if you would like to know more about this project and/or want to share your experience or idea with us, please contact Dr Yen-Fu Chen or Dr Abimbola Ayorinde