Publication and related bias
Publication bias refers to the phenomenon by which research findings that are statistically significant and/or are perceived to be interesting or favourable are more likely to be published than those that are not statistically significant or perceived to be interesting. Publication bias occurs at study level (i.e. when a study is not published at all owing to the nature of its findings). Similar bias may manifest at outcome level (i.e. a study may have evaluated several outcomes, but only the findings related to a subset of the outcomes that are statistically significant or judged to be worth noting are reported in its publication). This is termed ‘outcome reporting bias’.1 Bias can also occur when researchers carry out analyses of data collected from research. For example, multiple analyses may be performed using different techniques or different subsets of the data until statistically significant results are obtained, which are then published. This is referred to as ‘data dredging’ or ‘p-hacking’.2,3 These biases are a major threat to evidence-based decision-making, as they distort the full picture of the ‘true’ evidence gathered from research and may lead to misinformed decisions. Here we use the term ‘publication and related bias’ to refer to these biases collectively (Figure 1). Further bias may occur following the publication of research findings, such as citation bias and media attention bias, which alongside publication and related bias are collectively known as dissemination bias.4 For the present study, we focus on publication and related bias that occurs up to the stage of publication (see Figure 1), as biases that occur following publication can largely be overcome through systematic literature search, which is becoming a standard practice when evidence is synthesised to support decision-making.
FIGURE 1 Various forms of bias in the process of research publication. See Glossary for the definition of ‘winner’s curse’. (Figure not reproduced here; see the original article.)
A large body of literature has demonstrated the existence of publication and related bias in clinical research.1,4,5 In particular, several high-profile cases highlighting the non-publication and/or under-reporting of unfavourable results from clinical trials of new pharmaceuticals have raised ethical concerns over publication bias and have highlighted the potential harm it may cause to patients.6–8 These concerns have led to mandatory clinical trial registration9,10 and, more recently, mandatory reporting of findings of registered trials.11 By contrast, health services and delivery research (HSDR) has not been subject to similar levels of regulation and scrutiny, and the issue of publication bias appears to be largely ignored. Our preliminary scoping of the health services research literature found only a small number studies addressing this topic, which suggests the possible existence of publication bias.12–15 The paucity of documented evidence was affirmed in our discussions with some leading experts in health services research.
The lack of published literature on publication and related bias in HSDR compared with clinical research is surprising, given that publication bias has also been documented in many other scientific disciplines, such as social science,16 management research17 and ecology.18 As there is no obvious reason to believe that HSDR is immune to publication and related bias, and such bias could have substantial implications for health service decisions relying on HSDR, we set out to explore the existence and extent of publication and related bias in this field in order to inform best practice in the reporting and synthesis of evidence from HSDR.
Aims and objectives
The primary aims of this study were to gather prima facie evidence of the existence and potential impact of publication and related bias in HSDR, to examine current practice in systematic reviews of HSDR literature, and to explore common methods in relation to the detection and mitigation of the bias.
The above aims were to be achieved through five distinct but inter-related work packages (WPs), each with a specific objective:
- WP 1 – a systematic review of empirical and methodological studies concerning the occurrence, potential impact and/or methodology related to publication bias in HSDR, to provide a summary of what is known from current literature.
- WP 2 – an overview of systematic reviews of intervention and association studies in HSDR, to describe current practice and potential challenges in assessing publication bias during evidence synthesis.
- WP 3 – in-depth case studies to evaluate the applicability of different methods for detecting and mitigating publication bias in HSDR and to provide guidance for future research and practice.
- WP 4 – a retrospective study to follow up the publication status of cohorts of HSDR studies to directly observe publication bias in HSDR.
- WP 5 – semistructured interviews with health services researchers and commissioners, journal editors and service managers, and a focus group discussion with patient representatives to explore their perception and experience related to publication bias.
These WPs complemented each other to provide a full picture of both empirical evidence and current practices related to publication bias in HSDR (Figure 2). Detailed methods for the WPs are described in the next chapter.
FIGURE 2 Overview of the five complementary WPs included in this study. HSRProj, Health Services Research Projects in Progress; NIHR, National Institute for Health Research. (Figure not reproduced here; see the original article.)
Structure of the report
The rest of this report is organised as follows: Chapter 2 details the scope and methods of each of the five WPs described above. Chapters 3–7 present the findings from each of the WPs. Chapter 8 includes discussions of specific issues related to methods and findings of individual WPs, and provides some discourse of findings across the WPs and wider issues related to use of evidence to inform decision-making in health services. The chapter concludes with issues worth considering by different stakeholders of HSDR in relation to publication and related bias, and recommendations for future research.