Discussion and conclusion

In this chapter we discuss findings from individual WPs presented in Chapters 3–7. These are followed by an overview of learnings across WPs and limitations of the project, and a brief discourse of wider issues surrounding publication and related bias, and use of evidence to inform health-care service delivery. We then provide some points for consideration for stakeholders who fund, generate, review, publish and use HSDR in relation to publication and related bias. The chapter ends with some recommendations for future research and an overall conclusion.

Work package 1: systematic review of empirical evidence on publication bias in HSDR

Our systematic review set out to examine available evidence on publication and related bias in HSDR. Despite an extensive literature search, we found only four methodological studies13–15,50 that specifically investigated this issue, with three14,15,50 of them focusing on health informatics. Although the evidence is very limited and there are methodological weaknesses in the identified studies, the available evidence does indicate the existence of publication bias: study findings were not always published; differences may exist between published and unpublished evidence; and examples, such as null or unfavourable findings being associated with non-publication,53 and magnitude of effect sizes being associated with publication in high-impact journals, have been reported.13

Reasons for non-publication of HSDR findings described in the only survey we found concerning the publication of evaluation studies in health informatics appear to be similar to those for clinical research.87 Lack of time and interest on the part of the researcher appears to be a major factor, which could be exacerbated when the study findings are perceived as uninteresting. Some of the reported reasons, such as ‘not of interest for others’ and ‘only meant for internal use’, highlighted the context-sensitive nature of evidence that is common for HSDR. These reasons for non-publication also highlight issues arising from the vague boundary between research and non-research (such as quality improvement projects and service evaluations) for many interventions and data collection activities undertaken in health-care organisations. Another reason for non-publication of particular relevance to HSDR is ‘political and legal reasons’. Publication bias arising from conflict of interest is well documented in clinical research,4 and similar situations may arise when funding for the evaluation is provided by commercial, charitable or health-care organisations which developed and/or implemented the service intervention. We did not identify methodological research specifically related to the impact of conflict of interest on publication of findings in HSDR, although anecdotal evidence of financial arrangements influencing editorial process exists88 and there are debates concerning the public’s access to information related to health services and policy.89

Among systematic reviews of substantive HSDR topics that have reported findings of assessment of publication bias, most have used funnel plots and regression tests and approximately half of them found some evidence of bias. The required conditions for utilising these methods and caveats for interpretation of their findings,26,38 as illustrated and discussed in detail in case studies presented in Chapter 5, hold true here and warrant particular attention in view of heterogeneity commonly found among HSDR, resulting from the complexity and variability of service delivery interventions and the influence of contextual factors.90,91

Searching grey literature and contacting researchers and funders who may be involved in research on the topic of interest remain important measures to reduce the risk of publication and related bias when undertaking evidence synthesis for HSDR, although these are often time and resource intensive, and to what extent the invested effort outweighs the negative impact of the bias requires careful evaluation. The findings reported by Batt et al.,51 that published and grey literature from low- and middle-income countries may differ in volume, quality and geographical coverage, have important implications for stakeholders who are involved in the synthesis and use of HSDR related to global health.

This systematic review has several strengths, but also has some weaknesses. We used a wide range of search terms (see Appendix 1) and searched several databases and information sources. Nevertheless, we could not rule out the possibility that some evidence was missed in the process. In addition, we may not have covered all pertinent areas of HSDR owing to the definition that we adopted. We focused only on publication and related bias in quantitative studies and did not consider qualitative research, which plays an important role in HSDR. Given the limited evidence we found, which was mainly collected in the field of health informatics, the generalisability of the findings for other fields of HSDR is uncertain. Finally, despite our efforts in extensive literature search, the possibility that our findings are subject to publication and related bias cannot be ruled out.

Work package 2: overview of systematic reviews of intervention and association studies in HSDR

Work package 2 examined a random sample of 200 systematic reviews in HSDR in relation to reviewers’ practice in the assessment of publication bias and outcome reporting bias. Although 43% of the systematic reviews mentioned publication bias, only 10% formally assessed it. Outcome reporting bias was mentioned and assessed in 17% of the reviews. The proportions of reviews in which these biases were mentioned and assessed were significantly higher among intervention reviews than association reviews.

There are similarities and differences between our findings and those of previous meta-epidemiological studies on assessment of publication and outcome reporting biases in systematic reviews, mostly concerning clinical research (Table 14). Based on existing literature, approximately one-third to a half of systematic reviews mentioned publication bias and thus demonstrated awareness of this issue, with the notable exception of systematic reviews of genetic association studies, of which 70% mentioned publication bias. Mention of outcome reporting bias was lower than 30% across the board, with very low rates observed in reviews of HSDR association studies and reviews of epidemiological risk factors.

Although the findings suggest that there is still substantial scope for improvement in raising awareness of these potential biases and in explicitly acknowledging them in the conduct of HSDR systematic reviews, a few inter-related issues warrant further consideration when interpreting these findings and the making recommendations for further actions. These relate to research traditions and the nature of evidence, as well as the requirement for research registration, which we explicate below.

The research traditions and nature of evidence in different subject disciplines may influence the perceived importance and relevance for considering publication and outcome reporting biases in the review process. These might have contributed to the apparently low prevalence of assessing and documenting these biases in HSDR reviews. For example, both our study and the other HSDR-related study31 concerning health policy research mentioned earlier found that a meta-analysis was conducted in < 40% of all reviews, and in our study the prevalence was found to be as low as 10% for reviews of association. In contrast, at least 60% of reviews of clinical research included meta-analysis (see Table 14). There is a general recognition that HSDR requires consideration of multiple factors in a complex health system,19 and that evidence generated from HSDR tends to be context specific. It is therefore possible that HSDR systematic reviews, and particularly those examining associations between the myriads of structure, process, outcome measures and contextual factors, may tend to adopt a more configurative, descriptive approach (as opposed to the more aggregative, meta-analytical approach in reviews of various types of clinical research).97 As generating an overall estimate of a ‘true effect’ is not the main focus, the issue of publication and outcome reporting biases may be perceived as unimportant or irrelevant in reviews adopting configurative approaches. This may partly explain the strong association between inclusion of meta-analysis and assessment of publication bias in our multivariable analysis.

The diverse, context-specific nature of evidence in HSDR may have further impeded formal assessment of publication bias. Funnel plots and related techniques, the methods most commonly used in HSDR, as in other fields, require that at least 10 studies that are sufficiently similar can be included in an analysis.26 The requirement has probably prevented many HSDR reviews from carrying out formal statistical assessment of publication bias, as the level of heterogeneity among studies included in HSDR systematic reviews is often high. Irrespective of the feasibility of adopting statistical methods, these techniques provide only indirect evidence suggestive of the presence or absence of publication bias, and their limitations have been well documented.26

Irrespective of research approaches and nature of evidence, a major barrier for evaluating both publication and outcome reporting biases in HSDR is the lack of prospective registration of study protocols. Comprehensive study registration and accessible study protocols are essential for identifying unpublished studies and unreported outcomes, which in turn is a prerequisite for directly assessing publication and outcome reporting biases. As mandatory registration of research protocols has been enforced among clinical studies on human subjects only, lack of study registry and protocols is likely to have contributed to the low prevalence of assessing these biases, particularly among reviews of observational studies (e.g. 4% among HSDR association reviews in our study and 7% among epidemiological risk factor reviews examined by Song et al.4). However, irrespective of evidence synthesis approaches and challenges, the potential threats from these biases do not dissipate as long as the intention is to quantify an intervention effect or an association. Pre-registration of study protocols, which is the ultimate safeguard against publication and outcome reporting biases, may therefore be worth serious consideration for both HSDR intervention studies and at least some types of association studies. Accessibility to pre-registered study protocols will also alleviate current difficulties in assessing outcome reporting bias.

Calls for comprehensive registration of research studies and their protocols are not new, but making further progress beyond clinical trials is likely to require careful debates and assessments with regard to feasibility and practical value for registering different types of studies, weighing potential benefits against costs and potential harms. Meanwhile, it is important to continue raising awareness of these biases and improving the levels of documenting the awareness when evidence from quantitative HSDR is synthesised. Our findings show that systematic reviews mentioning the use of a systematic review guideline are five times more likely to include an assessment of publication bias than those without a mention. A previous study also found that the proportion of systematic reviews which assessed publication bias was significantly higher after the publication of PRISMA (53%) than before PRISMA (39%).94 Methodological standards, such as Cochrane Collaboration’s Methodological Expectations of Cochrane Intervention Reviews, and systematic reviews reporting guidelines, such as PRISMA and Meta analysis of Observational Studies in Epidemiology,33 are therefore likely to play an important role. Nevertheless, the suboptimal level of documenting awareness found in our study and others highlights that additional mechanisms may be required to enforce them.

For the assessment of outcome reporting bias, we noted that all the reviews which assessed this did so as part of quality assessment of individual studies. Currently, outcome reporting bias is a standard item in the Cochrane’s risk-of-bias tool,64 which is most widely used in intervention reviews. However, this item is not included in tools commonly used for assessing observational studies, such as the Newcastle–Ottawa Scale.98 Given that the risk of outcome reporting bias is substantially higher for observational studies, this is an important deficit which developers of quality assessment tools for observational studies need to address in the future.

Finally, the search and inclusion of grey and unpublished literature remains a potentially important strategy in minimising the potential effect of publication bias. However, the practice varies widely between different types of reviews. Interestingly, our study showed that reviewers who searched for grey literature do not always assess or discuss the potential effect of publication bias. This suggests that some review authors might have followed the good practice of searching the grey and unpublished literature to ensure comprehensiveness, without considering minimising publication bias as a rationale behind this. Alternatively, these authors may consider it unnecessary to assess and/or discuss the potential impact of publication bias in addition to the efforts in locating unpublished studies. Limited evidence suggests that data included in published HSDR studies differ in quality and nature from those included in grey literature.51 More empirical evidence is needed to guide future practice regarding search of grey and unpublished literature, taking into account the trade-off between biases averted and additional resources required.

One of the strengths of the current study is that the systematic reviews were randomly selected from the HSE database, which covers multiple sources of literature, and our selection was neither limited by a single source of literature nor restricted to highly ranked journals. We therefore believe that the sample is representative of quantitative HSDR systematic reviews. In addition, study selection and data extraction were carried out by one person and checked by another in order to ensure accuracy and completeness. We also evaluated factors which may influence the assessment of publication and outcome reporting bias. The main limitation of this study is that the results of the multivariable logistic regression analyses produced ORs with fairly wide CIs. We are aware that variables which we examined may interact in various ways, but believe that collinearity is not a major issue, judging from the broad consistency between the results of univariable and multivariable analyses.

In conclusion, this study has shown that publication and outcome reporting biases are not consistently considered or assessed in HSDR systematic reviews. Until a comprehensive registration of HSDR studies and their protocols becomes available, formal assessment of publication bias and outcome reporting biases may not always be possible. In such cases, review authors could still consider and acknowledge the potential implications. Developers of quality assessment tools for observational studies should consider including items for outcome reporting bias. Adherence to existing systematic review guidelines may also improve the consistency in assessment of these biases. The findings of this study would enhance awareness of publication and outcome reporting biases in HSDR systematic reviews and inform future systematic review methodologies and reporting.

Work package 3: in-depth case studies on the applicability of methods for detecting and mitigating publication and related biases in HSDR

In case study 1, described in Chapter 5, we utilised data from a systematic review of studies on the weekend mortality effect to explore in detail the applicability of common methods that have been used in clinical research to detect and/or adjust for publication bias. We highlighted several issues pertinent to the use of these methods in evidence synthesis in HSDR, and the findings may be useful in similar scenarios in which HSDR mainly relies on evidence from analyses of observational data obtained from routine administrative databases.

The case study shows that, although funnel plot and related methods can be used in HSDR as a tool to explore the presence of small-study effects (which are a common manifestation of publication/reporting bias), several features of HSDR based on observational databases could contradict requirements for their appropriate use and, in particular, could invalidate the crucial assumption that any observed small-study effects are caused by publication or reporting bias. Special attention needs to be paid to the underlying heterogeneity of studies included in a review. Factors that could define distinctive subgroups and settings (e.g. types of admissions, availability of clinical data) may be potential confounding factors that are independently associated with effect sizes and sample sizes and may therefore confound the assumed publication bias. Any findings from these statistical techniques will thus remain highly speculative, unless these potential confounding factors can be ruled out, which is likely to be difficult.

Our detailed exploration in this case study suggests that some level of publication or reporting bias is possible in the literature included in the chosen systematic review, but its effect on the pooled estimate of the weekend effect is limited. A few points are worth mentioning. First, the authors of the review excluded grey literature, such as conference abstracts and institutional reports, because it was difficult to assess their quality. This precluded direct assessment of potential publication bias by comparing published and grey literature. Second, the impact of publication bias may be relatively small when data from population databases are available. Nevertheless, caution is required in such circumstances, as precise estimates generated by large study samples are not necessarily accurate. Evaluating other sources of bias, such as the potential impact of unaccounted confounding, may be of higher priority and importance than publication bias. Finally, a deeper epistemological consideration is the utility of a ‘pooled effect’ in HSDR in general. In contrast to the priority of pursuing an accurate, universal estimate of the effect of a treatment or exposure in clinical research (however, this might change in the era of ‘precision medicine’), a pooled effect estimate, such as the weekend effect across various patient populations, health systems and geographical locations, may be of interest to no more than a small number of persons. Examining the mediators and contextual moderators of an effect is likely to play a more prominent role in HSDR. Funnel plot and related methods that are designed to assess the impact of publication bias surrounding a pooled estimate may therefore be of limited value, except in examples such as multivariate meta-regression, in which investigation of the impact of multiple factors can be carried out. Issues identified in case study 1, such as heterogeneity and potential confounding, were mirrored in other case studies included in the WP.

Work package 4: follow-up of HSDR study cohorts for investigating publication bias in HSDR

This WP collected direct, and thus the strongest, evidence on publication bias. Rates of publication among the selected HSDR cohorts are relatively high, except the ISQua cohort. This may reflect the nature of these cohorts, most comprising studies being well-funded research carried out by researchers. We noted that many of the studies from the ISQua cohort were reported by practitioners and were conducted in a single institution, as exemplified by the larger proportion of before-and-after studies without a control group. Many of these were likely to be studies of in-house improvement project and service evaluation that were not intended to be published.

The significantly lower rate of publication in this cohort than in other cohorts therefore suggests that the motivation for conducting these studies is likely to be one of the key factors influencing subsequent publication of the findings. Nonetheless, our multivariable analysis provided some indication that statistical significance and perceived positivity of the findings did appear to influence the likelihood of publication having adjusted for other factors associated with motivation, funding and methodological rigour. This finding should be considered tentative, given our limited sample size and potential subjectivity in classifying research findings. In addition, we were unable to obtain further information from investigators for 34 studies, with 23 of these from the ISQua cohort. This might have introduced some bias, although it could be argued that from the perspective of accessibility of evidence, any publications that might have existed but were missed by our own searches may be difficult to reach by decision-makers and other users of HSDR. Given the differential motivation, publication bias may be more profound in cohorts, such as the ISQua, than in cohorts of independently funded research. However, our sample size was too small to investigate this.

The 100% accessibility of study findings from the NIHR cohort demonstrates the important influence that research funders could have. Archives of historical projects accessible online and the now established NIHR Journals Library facilitate the dissemination of research findings irrespective of whether or not they are published in ‘traditional’ academic journals. An unexpected finding from this work was that the rate of publication in academic journals is lower for the NIHR cohort than for the HSRProj cohort. We could offer two plausible explanations among others. First, for consistency, we have classified all full reports from NIHR-funded projects as ‘grey literature’. These included project reports from the previous SDO and Health Service Research programme available online, as well as more recent monographs published in the Health Services and Delivery Research series of the NIHR Journals Library. If the latter had been classified as an academic journal, the publication rate would have appeared higher. Second, given that the journals included in the NIHR Journals Library are indexed in MEDLINE and are openly accessible, researchers might not perceive it necessary to publish the findings in other academic journals. In addition, during our communications with HSDR researchers, we were made aware that some academic journals in the field appear to be more reluctant to publish findings that would also be described and made available in NIHR Journals Library. Furthermore, as the production of the project final report as a monograph in the NIHR Journals Library requires substantial effort, this might have incurred some opportunity costs in relation to preparation of publications in other academic journals.

Work package 5: key informant interviews and focus group discussion to explore publication bias in HSDR

A notable finding from WP 5 was that many respondents were uncertain as to how significant a problem publication bias presented in HSDR. Although this is perhaps unsurprising in the case of those not directly involved in publishing, for example patients and health-care managers, it was also a feature of interviews with researchers, journal editors and research funders. Although the majority of these believed publication bias in some form to be present, few were able to refer to instances when this had been directly observed. This lack of clarity may reflect the apparently equivocal findings of previous WPs and our interviews therefore provide some potentially valuable qualitative insights.

One area that influenced views as to the presence of publication bias was the subdiscipline of HSDR to which respondents belonged. The researchers in our sample spanned diverse scholarly subfields, and this was reflected in the journals they targeted to publish their work. Although respondents sought to publish HSDR results in medical journals, many did so as a response to institutional pressures to publish in journals with high impact scores. In these circumstances, there was some consensus that significant (if not necessarily positive) results would be more likely to be (1) submitted and (2) accepted for publication. In contrast, many believed that other criteria were more important when seeking to publish in their disciplinary ‘home’ journals. Here, strength of results was believed to be secondary to other potential sources of ‘novelty’ in shaping publication outcomes.

As well as journal variation, interviewees drew a distinction between externally funded and peer-reviewed research on the one hand, and end user-funded quality improvement projects on the other, with the latter category attributing higher susceptibility to publication bias instigated by the research teams. Respondents also identified risks in ex post facto decisions to submit for publication, data which have been gathered primarily for other purposes. In the former category, there was general agreement that HSDR studies often contain more than summative assessments or measurements of associations. The typical presence of multiple study objectives, complex interventions, higher levels of ‘mess and noise’ in the data and their interpretation, was seen as reducing the importance ascribed to effect sizes and significance of associations. For this reason, a substantial minority of respondents were resistant to the proposal that all HSDR research be pre-registered.

There was some support in interview findings for the claim that forms of bias are linked to study type, with, for example, association studies considered more susceptible to p-hacking and selective outcome reporting, and evaluation studies at greater risk of funder pressure. However, respondents contrasted the level of external scrutiny (e.g. from industry and health systems) in clinical research with the tendency towards ‘lower stakes’ in HSDR, which was again seen as mitigating the incentive towards publication and related bias. Overall, interviewees considered incentives towards publication and related bias to be present but to a lesser degree.

The research cultures described by interviewees confirm the previously reported pressure to publish in high-impact journals.99 However, in comparison with clinical research, HSDR is fragmented, and ‘subcultural’ factors may attenuate the drivers of publication and related bias. Other variables that warrant further investigation include researcher seniority (e.g. senior researchers in our sample were more likely to believe that negative or null results did not present an impediment to them publishing their work).

The interviews and focus group discussions with those not directly involved in publishing research (health-care decision-makers, patients and citizens) raise issues that go beyond the specific question of publication and related bias. These include the relatively weak relationship between HSDR and decision-making in health care, and the wider ethical implications of research waste, especially for participating patients and the allocation of scarce public resources.

Overall learning from the project and study limitations

Having described lessons learned from individual WPs above, we offer below some learning that emerges from across the WPs and highlight key limitations of this project.

Gleaning from the evidence collected across all WPs, it is reasonably clear that publication and related bias can and does occur in HSDR. However, it is currently difficult to gauge the true scale and impact of the bias given the sparse high-quality evidence, which in turn may be associated with difficulties in identifying study cohorts due to lack of comprehensive study registration. Solid evidence on publication bias and selective outcome reporting in clinical research has primarily been obtained from RCTs, as study protocols are made available in the trial registration process.1 Stakeholders in our interviews generally supported registration of HSDR, albeit with some reservations, and quality improvement projects and implementation studies were identified as being potentially at high risk of these bias. As repositories of quality improvement projects emerge,100 HSDR and quality improvement communities will need to consider and evaluate the feasibility and values of adopting these practices for studies of varied types and purposes.

The lack of prospective study registration poses further challenges in assessing outcome reporting bias, which could be a greater concern for HSDR than clinical research, given the more exploratory approaches to examining a larger number of variables and associations in HSDR. Interestingly, our hypothesis that association HSDR studies based on observational evidence are more susceptible to publication and reporting bias was supported neither by the quantitative evidence that we collected so far, nor by recent research from the field of organisational studies (described in more detail in Publication and related bias in other cognate fields).101–104 Nevertheless, the potential risk could still not be dismissed, as most of available evidence relies on methods that have some weakness. It is possible that the focus on formative evaluations rather than summative measures may have mitigated the bias to some extent.

Like many issues in HSDR, publication and related bias is the product of complex social processes and is likely be context dependent. Qualitative and quantitative data obtained from different WPs identified several potential factors (see Table 13) that may affect on the risk of its occurrence. Further empirical evidence needs to be collected to explore the relative influence of these factors and to ascertain whether interventions or policies targeting some of these factors can effectively reduce or prevent publication and related bias.

The strengths and limitations of individual pieces of work from this project have been discussed in detail in each of the WPs. Overall, although our findings have shed new light on our understanding of publication and related bias, the project and its findings have some important limitations:

1. We focus only on quantitative HSDR investigating interventions and associations.
2. Few studies have systematically examined publication and related bias in HSDR, limiting the conclusion that we can draw from review of the literature.
3. The numbers of HSDR systematic reviews that we evaluated in WP 2 and HSDR cohorts that we followed in WP 4 are still relatively small.
4. Although issues with the application of statistical methods to detect publication and related bias in HSDR are highlighted in WPs 2 and 3, novel methods need to be developed to overcome the issues identified.
5. Owing to lack of comprehensive registration of study protocols, we have not been able to collect direct evidence on publication and related bias concerning quality improvement projects, and to examine outcome reporting bias in study cohorts that we followed.
6. The sample size for our qualitative work in WP 5 is relatively small, and within the scope of the current project we have not explored some of the wider issues related to use of evidence from HSDR in making service decisions and transparency and public accountability of this process.

These limitations need to be addressed in future research.

Publication and related bias in other cognate fields

This project investigated publication and related bias in HSDR. A major challenge for the project was to draw a boundary between HSDR and other fields of research, given its multidisciplinary nature. In our systematic review presented in Chapter 3, we have focused on research in areas that fall under the definition of HSDR adopted in this project. However, we are aware that literature on publication and related bias has emerged in other scientific disciplines. Although it was not possible for us to systematically review this diffuse literature within the scope of this project, we provide below a brief description of key papers that may have some relevance to HSDR.

In our literature search for the systematic review described in Chapter 3, and through citation check, we identified several studies on publication and related bias in cognate fields. We describe these studies briefly in this section.

Organisational behaviour, organisational psychology and human resource management

Four studies examined publication and outcome reporting bias in the broad area of organisational behaviour, organisational psychology and human resource management. All four studies focused on measures of correlations and all found that publication and outcome reporting bias did not seem to be a major issue. The studies are described below.

Kepes et al.101 evaluated publication bias in organisational sciences. Using data on employment interview validities (total sample size of 25,244 interviews and 160 correlation coefficients), the authors conducted multiple analyses, including trim and fill, contour-enhanced funnel plots, Egger’s regression test, Begg and Mazumdar’s rank correlation, meta-regression, cumulative meta-analysis and selection models. When all the samples were analysed together, many of the publication bias detection methods suggested potential publication bias, although the effect seems minimal and it would not have affected the conclusions. The authors anticipated that between-sample heterogeneity may have influenced the results and, therefore, created subgroups and assessed publication bias within the subgroups. This resulted in analyses of 40 distributions, in three of which the authors concluded that publication bias is likely to be present. Although most of the results from different methods were consistent, there were many occasions when at least one method indicated conflicting results to the other methods. Most of the disagreements may be related to problem of statistical power. Notwithstanding, the interpretation was inconclusive in only 6 out of 40 analyses performed. Using cumulative meta-analyses by year of publication, indication of potential time-lag bias was also observed: the two samples published in 1947 have a cumulative mean estimate of 0.67 and all eight samples published before 1970 have a cumulative estimate of 0.33, whereas the final meta-analytic estimate is 0.24. However, the data covered a 50-year time period and time-lag bias would not have changed the practical conclusions of the analyses. The authors concluded that publication bias has minimal effect in the data set assessed. They recommended that meta-analytic reviews include multiple publication bias methods that are based on different assumptions. To mitigate heterogeneity, potential moderating variables could be identified using meta-regression, in order to form more homogeneous subgroups before evaluating publication bias.

Dalton et al.102 evaluated the extent of the file-drawer problem (an alternative term for publication and related bias frequently used in social sciences) in non-experimental research in organisational behaviour and human resource management, industrial and organisational psychology, and related fields.102 They estimated the frequency and percentage of statistically non-significant correlations reported in correlation matrices of primary studies from four different sources: (1) randomly selected issues of three organisational management journals published between 1985 and 2009 (403 correlation matrices, including 37,970 correlations); (2) 51 published meta-analyses (6935 correlations); (3) unpublished manuscripts obtained from randomly selected faculty members of 30 schools of business and 30 industrial and organisational psychology programmes (167 correlation matrices, including 13,943 correlations); and (4) 50 randomly selected dissertations identified from the ProQuest Dissertations & Theses A&I Database (217 correlation matrices, including 20,860 correlations). They found that the percentages of statistically non-significant correlations were all in the range between 44% and 51%, with overlapping CIs. In addition, they compared the average magnitude of randomly selected 1002 correlations from published primary studies [i.e. source (1) above] with that of randomly selected 1224 correlations from unpublished dissertations [i.e. source (4) above] and found similar average magnitude in both groups (0.227 vs. 0.228, respectively). They therefore concluded that the file-drawer problem does not severely threaten the validity of conclusions derived from meta-analyses and does not lead to overestimation of effect sizes in meta-analyses, as commonly believed. However, they also acknowledged that they did not compare published and unpublished findings among studies of specific focal areas in the broad field.

Stemig and Sackett103 assessed the file-drawer issue in industrial and organisational psychology. The authors examined meta-analyses that compared the effect sizes from published with unpublished studies, for which publication status was examined as a potential moderator variable. The meta-analyses were selected from two journals that the authors deemed to be journals in which meta-analyses are most prevalent in industrial and organisation research: Journal of Applied Psychology (JAP) and Personnel Psychology. Of 181 meta-analyses from JAP, 145 (80.1%) reported inclusion of unpublished studies and only 16 (8.8%) analysed publication status as a moderator variable. Twenty out of 43 meta-analytic studies from Personnel Psychology included unpublished studies, but only four (9.3%) examined publication status as a moderator variable. Thus, 20 studies (16 from JAP and four from Personnel Psychology) presenting 84 comparisons of published and unpublished studies were included. The overall mean observed correlations in the published and unpublished studies were similar (0.221 vs. 0.224, respectively). Differences between mean correlations of published and unpublished studies were small in majority of the comparisons (60% were < 0.1; 24% were between 0.1 and 0.2; 12% were between 0.2 and 0.3; and 4% were > 0.3). Most of the supposedly large differences were not statistically significant. Significant differences between published and unpublished studies were observed in seven of the correlations, but only two of the significant differences exceeded 0.2. Interestingly, both of these showed larger mean effects in unpublished studies, which is contrary to what is expected for a file-drawer problem. The authors concluded that, although average effect sizes from published and unpublished studies are similar, there are instances when they produce different results. However, the authors acknowledge that the study had a relatively small sample size and that unpublished studies identified in the included meta-analyses may not be representative of all unpublished studies.

Paterson et al.104 assessed the file-drawer problem in meta-analyses of organisational behaviour and human resources studies. The authors searched 30 management journals that were considered to have the most impact which had published at least one meta-analysis by 1 June 2012. The search identified 350 articles, from 11 journals, that included words relating to meta-analyses in their titles. The 350 articles were examined to identify those that (1) reported a meta-analytic relationship between two variables; (2) were based on three or more independent samples from at least two sets of authors; and (3) involved ‘micro’ domains of organisational behaviour or human resources, or industrial and organisational psychology (i.e. ‘macro’ domains, such as business policy and strategic management, were excluded). Ultimately, 258 meta-analyses which generated 776 meta-analytic conclusions were included. The study assessed file-drawer issues in two ways. First, the correlation between the effect size and proportion of unpublished studies was calculated for the 456 analyses in which the proportion of unpublished studies was reported. The result showed no significant relationship between effect size and proportion of unpublished studies (r = –0.065; p > 0.05). Second, the relationship between sample size and effect size was assessed and the result also showed no statistically significant relationship (r = −0.026; p > 0.05). Hence, the authors concluded that publication bias does not pose a major threat in the micro-oriented management literature. This conclusion is similar to the previous studies described above. However, the study focused on meta-analyses published in top management journals and so the sample may not be representative of all studies.

Social sciences

Utilising a Time-sharing Experiments for the Social Sciences programme sponsored by the US National Science Foundation, Franco et al.16 undertook an inception cohort study to investigate publication bias in social sciences. Researchers submitted research proposals of survey-based experiments to be run on representative samples of American adults (e.g. to test the effect of questionnaire wording) to the Time-sharing Experiments for the Social Sciences programme. These were peer reviewed and grants were allocated on a competitive basis and, therefore, studies in the cohort assembled from the programme passed a certain quality threshold. The authors followed up 221 of 249 selected studies to verify the statistical significance of their findings and their publication status. Study findings were classified as strong (all or most of the hypotheses were supported by the statistical tests), null (all or most hypotheses were not supported) or mixed (the remainder of studies). They found a strong association (chi-squared test p < 0.001) between strength of findings and publication status, with > 60% of studies with strong findings published compared with around 20% of studies with null results. In addition, they found that the authors did not write up the findings in only 4% of studies with strong results compared with 65% of studies with null results, indicating that authors’ motivation is a key factor contributing to publication bias.

Points for consideration by stakeholders in relation to publication and related bias in HSDR

In this project we collected prima facie evidence of publication and related bias in quantitative HSDR using a variety of approaches that generated both direct and indirect evidence, as described in the previous chapters. Overall, our findings suggest that publication and related bias can and does occur in quantitative HSDR. However, there is substantial uncertainty with regard to the nature and extent of its occurrence and its impact on decision-making concerning health service delivery.

We presented our findings in a meeting convened at the end of the project, in which we invited a group of key stakeholders of HSDR (see Acknowledgements) to help us make sense of the evidence gathered in this project and formulate potential recommendations for future research practice. The attendees included senior researchers in HSDR and evidence synthesis, journal editors, funders and a PPI representative. Diverse views and perspectives were presented in the meeting, echoing the findings from our interviews. Taken in the round, the emerging consensus seemed to be that a nuanced interpretation, with due attention paid to the epistemological and methodological diversity in HSDR and the changing landscape of research publication, is needed rather than rushing to replicate measures (such as compulsory study registration in an attempt to reduce occurrence of this bias) that have been adopted in clinical research or dismissing the importance of this issue due to current lack of concrete evidence. Here, we highlight pertinent issues that different stakeholders might wish to consider when commissioning, undertaking, reviewing, publishing or using HSDR.

Epistemological and methodological diversity in HSDR

As stated in Chapter 2, we adopted a pragmatic definition of HSDR in line with that used by the NIHR HSDR programme for this project, and focused our attention on two specific types of quantitative studies: those evaluating the effectiveness of an intervention and those assessing the association between various structures, processes, outcomes and contexts along the service delivery causal chain. These allowed us to draw a boundary and maintain the practicality of our investigation. The findings of our project are naturally bounded by this chosen scope. They cannot be considered representative of the full spectrum of HSDR, which is broad and diverse in terms of subject areas, epistemological stances and methodological approaches. For example, the project did not consider publication bias in qualitative studies, case studies and simulation studies in operational research, all of which play an important role in HSDR. The mechanisms, occurrence (or lack of) and magnitude of publication and related bias may also differ between subspecialties of HSDR. Nevertheless, we contend that the two types of quantitative studies that we have focused on are important sources of information that can potentially influence health service delivery and policy and, therefore it is important for us to understand if and how publication and related bias might affect the dissemination and use of this evidence.

The methodological and epistemological diversity of HSDR has implications for the generation and interpretation of quantitative evidence, and corresponding bearing on publication and related bias. For example, mixed-methods studies are widely used in HSDR, in which evidence gathered from quantitative and qualitative methods is intended to provide a deeper understanding of the phenomenon than either methodological approach could offer alone. The appropriateness of separating quantitative data from such studies in order to combine them with data from other quantitative studies in a meta-analysis, and assessing the evidence for the presence of publication and related bias using the assumptions underpinning a quantitative research paradigm, may be questionable.

Changing landscape of research publication

A large volume of scientific literature on publication bias was accumulated in the middle to late twentieth century, when academic journals were the main, and sometimes the only, outlet for disseminating research findings. Publication bias in the form of selective publication of positive and/or statistically significant findings by academic journals was a major concern, as it would be very difficult for people other than the investigators themselves to locate and access ‘unpublished’ research findings. The situation has dramatically changed since the advent of the internet, through which research findings can be easily shared with other people all around the world in a variety of forms, such as technical reports, pre-publication manuscripts, discussion papers and so forth, even if they are not submitted or accepted for publication by academic journals. This has implications for the study and measurement of publication bias, which may increasingly focus on the extent and ease of online access to research findings, rather than availability in traditional peer-reviewed outlets.

Within academic research, mandatory publication of findings in open access platforms, such as the principle advocated by the Plan S initiative,105 is gaining strong momentum that could potentially minimise publication bias for funded research. The creation of journals which aim to provide a platform for publishing research findings according to methodological rigour, irrespective of the perceived interest of the findings, is a further response to well-documented concerns regarding publication bias in clinical research. The increasing availability of such journals (e.g. PLOS ONE and BMJ Open) may reduce potential publication bias across scientific disciplines, including HSDR. However, this creates other risks, including bias associated with the availability of funding to pay article processing charges, and conflicts of interest as, for example, motivation for publishing in such journals may be stronger when there is a potential for financial or reputational gains associated with the publication.106 This development also diversifies the roles played by academic journals, which traditionally serve both as a competitive space for showcasing research excellence and as an archive of tested knowledge. Measures to reduce publication bias might therefore lead to the unintended introduction or exacerbation of other forms of bias and so should be adopted with caution.

The changing landscape of research publication and information dissemination also means that the definitions of terms, such as ‘publication’ and ‘grey literature’ are evolving, with the boundary between the two becoming increasingly vague. Careful selection, definition and use of appropriate terms will be required (see Terminology related to bias in reporting research findings).

Terminology related to bias in reporting research findings

Both publication bias that occurs at the study level and selective outcome reporting that occurs at individual outcome levels result in a distorted picture of the totality of evidence available to decision-makers. Given the rapid evolution of research publication described above and associated changes to the nature of biases, there is a move to adopt ‘non-reporting bias’ as the collective term to cover these biases by the Cochrane Collaboration (Professor James Thomas, University College London, 7 June 2019, personal communication). Consistent use of this term should help facilitate exploration and debate in future.

Prevention of publication and related bias in HSDR

A clear message from the findings of WP 4 (follow-up of HSDR cohorts) is that research funders can play a crucial role in minimising publication and related bias and the associated research waste through a clear policy of mandating the publication of findings of commissioned research, and by providing suitable platforms for archiving and disseminating funded research, such as those adopted by various research programmes of the UK NIHR.

However, implementation of such policy and arrangements requires substantial resources which may not be available to smaller-scale funders. In addition, many HSDR studies are motivated by the desire to understand service issues or to gauge the impact of efforts to improve services and are carried out with limited or no funding. Without the supporting infrastructure and incentives or sanctions from the funders, these studies may be more susceptible to publication and related bias. Although findings from WP 5 (key informant interviews) suggest that HSDR stakeholders generally support the practice of registering research protocols (or making them available in openly accessible repositories) when feasible, there are also concerns that inconsiderate and prescriptive adoption of these practices may stifle the creativity and diversity that are crucial for resolving challenges in health service delivery.

An area clearly requiring further attention is quality improvement projects and service evaluations. These activities often occupy a grey area between research and non-research, are often conducted locally and/or retrospectively, and are not (at least initially) carried out with an intention to produce generalisable knowledge. An isolated study included in our systematic review (WP 1) and findings from our interviews with HSDR stakeholders (WP 5) suggested that any such studies that subsequently appear in the academic literature may be those with the most favourable results. Therefore, caution may be required when studies of this nature are included in systematic reviews and evidence syntheses: policy-makers and service-planners should be aware that isolated quality improvement studies reported in the literature may represent a subset of locally evaluated interventions, rather than proven best practice that can be easily replicated or scaled up in other settings.

Comprehensive registration of quality improvement projects may reduce potential publication and reporting bias associated with these studies. It may further enhance the potential utility of evidence generated from them, providing an audit trail and a denominator for the totality of evidence, as well as improving the ease of locating and accessing these studies. However, these benefits need to be balanced against the resources required for the creation and maintenance of such registries; the opportunity costs of the efforts required by service planners and practitioners to complete the registration and the practicality against other service and policy imperatives; the quality and value of evidence that can be obtained from the registries; and the potential burden of examining this type of evidence in future systematic reviews of HSDR.

Assessment and mitigation of publication and related bias in systematic reviews of HSDR

Findings from our meta-epidemiological study of published HSDR systematic reviews (WP 2) showed a low level of documented awareness of the bias. It is difficult to ascertain whether the lack of documentation was due to reviewers genuinely lacking awareness of the bias or to their perception of the (lack of) importance and relevance of the bias, either in relation to the suitability of available methods for detecting and mitigating the bias or in relation to the aims and epistemological stance (e.g. configurative vs. aggregative) of the review. Our findings from WP 2 pointed to the latter, but further investigation is required.

In terms of assessment of publication bias and outcome reporting bias, funnel plots and related techniques remain the dominant methods used in the reviews in the absence of comprehensive study registries. As our case studies (WP 3) illustrated, application and interpretation of funnel plots and related techniques are often problematic owing to the inherent heterogeneity of evidence from HSDR and potentially incompatible assumptions arising from epistemological differences. These techniques are best used as part of broader efforts to investigate the source of statistical heterogeneity between studies, with extra caution needed in ascertaining the assumptions and requirements for appropriate application of these methods. In particular, injudicious interpretation of small-study effects illustrated by funnel plot asymmetry as direct evidence of publication bias should be avoided.26 Further training of evidence reviewers on appropriate use of these techniques and development of novel methods may be required.

Limited evidence included in our systematic review (WP 1) suggested that important differences in the quantity, quality and nature of evidence may exist between published academic literature and other sources of (grey) literature.51 Further documentation and accumulation of evidence on the extent to which search for and inclusion of non-academic literature affects the findings and conclusions of HSDR systematic reviews are needed.

It is also worth highlighting that publication and related bias is just one of many different biases that could affect the trustworthiness of research evidence. For example, case study 1 described in Chapter 5 (WP 3) illustrated that unaccounted confounding may be a greater concern in meta-analyses of HSDR studies based on analyses of administrative databases. Guidance on interpretation of overall quality of evidence, such as GRADE, may support reviewers to make appropriate judgements in interpreting the importance and relevance of publication and related bias relative to other factors.35

Potential impact of publication and related bias in HSDR

Given the paucity of evidence and the challenges in establishing the occurrence and magnitude of publication and related bias in HSDR, it is currently not possible to estimate the impact of this bias on decision-making related to health services, or assess its further effects downstream (e.g. resulting service organisation and patient outcomes). One pertinent issue is that, although evidence-based decision-making has become an integrated part of health care for individual patients, the same may not be true when decisions on the organisation and delivery of health services are made. Indeed, emerging evidence has shown a very different decision-making process in the planning and organisation of health services. Qualitative research has shown that health-care commissioners’ decision-making process is highly pragmatic,107,108 with many sources of information used in a dynamic environment, including best practice guidance, the opinions of clinicians and service users, local data, conversations and story sharing within informal networks and innovations from elsewhere. Research evidence may therefore play only a minor part in informing service planning and policy. As a consequence, publication and related bias in research evidence may appear likely to have only a small, indirect, impact on SDO. However, although explicit use of research evidence may seem rare in decision-making concerning health-care services and policies, research evidence could also have influenced the decision-making process indirectly through tacit knowledge, precedent decisions based on research evidence and conceptual use of research.109 Consequently, the importance of the issue of publication and related bias in HSDR should not be overlooked. In our focus group discussion, patients and members of the public attributed high importance to decision-making being based on unbiased evidence (WP 5).

Recommendations for future research

To the best of our knowledge, this project is the first to systematically investigate publication and related bias across different subdisciplines of HSDR. Overall, our findings highlight a paucity of evidence, with many gaps and weaknesses in the evidence base informing this subject. We present below our recommendations for future research that may help in bridging these gaps.

Prevention

• To explore the feasibility, acceptability and sustainability of establishing new registries and/or making the best use of existing registries for various types of HSDR, including quality improvement projects and service evaluations. As measures related to prevention could reduce the need for subsequent efforts in the detection and mitigation of publication and related bias, exploration of these measures may deserve a high priority.

Occurrence, magnitude and relevance

• To accumulate further empirical evidence on publication and related bias in different subject areas and for studies of different designs within HSDR, in particular through direct methods that allow verification of the actual occurrence of non-publication and/or selective outcome reporting, as well as reasons behind these in cohorts of HSDR studies for which a denominator (i.e. the total number of studies undertaken) is known.
• To explore the existence, nature, relevance and impact of publication bias in qualitative and mixed-methods studies in HSDR and compare these with quantitative studies.
• To characterise HSDR in relation to other disciplines of research and features of HSDR which might increase or mitigate the occurrence of publication and related bias.
• To investigate what aspects of multicomponent HSDR studies are subsequently published in academic journals.
• To explore the influence of the composition of the research team on the occurrence of publication and related bias.

Methods of detection and mitigation

• To evaluate the utility and most efficient methods of including non-academic literature in systematic reviews of HSDR, taking into account the burden on reviewers and impact on the findings and conclusion of the reviews.
• To explore reasons behind the apparent low levels of awareness of publication and related bias in HSDR systematic reviews.
• To develop new tools for assessing the risk bias associated with potential non-reporting of data in HSDR.

Wider methodological development of HSDR

• To explore the value of including findings from quality improvement projects and service evaluation in systematic reviews of HSDR.

Conclusion

This project collected prima facie evidence on publication and related bias in quantitative HSDR. Overall, our findings suggested that this bias can and does occur in this field, although literature on this topic is scant. The epistemological and methodological diversity of HSDR, the changing landscape of research publication and the boundary delineated by project scope all need to be taken into account when interpreting our findings.

The occurrence of publication and related bias and its impact are likely to vary between studies with different methodological designs for different purposes, with evaluations motivated by service improvement locally most susceptible to such bias. Variation in the practice of research publication may exist between different subspecialties in HSDR but this requires further exploration. The bias is likely to be most profound in relation to publication of HSDR findings in academic journals with a general medical orientation.

The precise magnitude of publication and related bias is difficult to quantify owing to the lack of comprehensive study registration and the vague boundary between research and activities undertaken for service improvement. Epistemology and methodological approaches that do not focus solely on summative assessments of outcomes or associations and that do not heavily rely on inferences based on statistical significance may to some extent mitigate the occurrence and influence of publication and related bias in HSDR.

Documentation of awareness of publication and related bias and formal assessment of the bias were low in HSDR systematic reviews, particularly in reviews of association studies. Methodological guidelines might have some positive impact on practice. Application of statistical tools, such as funnel plots, and related tests for detecting publication and related bias in HSDR is often problematic owing to the heterogeneity of studies and potential confounding in any associations observed between sample size (or precision of estimate) and effect size. These tools are best used, with great caution, as part of the broader effort in investigating potential modifiers of intervention effects or associations. Use of funnel plot asymmetry as a sole indicator of the presence or absence of publication and related bias should be avoided.

Further research to collect empirical evidence based on methods that allow for direct observation and inference of the occurrence of publication and related bias in various subdisciplines and study designs of HSDR is required. The utility and most efficient ways of including quality improvement projects and other evidence beyond academic literature in systematic reviews of HSDR need to be explored.