This chapter presents findings from WP 1 of the project, in which we systematically searched and reviewed empirical studies that set out to investigate the occurrence of publication and related bias in HSDR.
Literature search and study selection
Initial searches of electronic databases in March 2017 retrieved 7732 records. After removing duplicates, the titles and abstracts of 6155 records were screened. Of these, 422 records were considered potentially relevant and their full-text articles were retrieved. Six additional full-text articles were identified and obtained from other sources.
Of the 428 full-text articles examined, 188 were retained. Of these, four were methodological studies that set out specifically to investigate publication and related bias in HSDR13–15,50 and three were systematic reviews of substantive HSDR topics, in which evidence from published literature was compared with grey literature and unpublished studies (and thus provided direct evidence on publication bias).51–53 These seven studies were examined and are described in detail in this chapter. The remaining 181 studies were systematic reviews of substantive HSDR topics, in which publication and outcome reporting bias was assessed as part of the review process. As these reviews provided only indirect evidence on publication bias, they are briefly described in this chapter and are summarised in Appendix 6.
Two hundred and forty retrieved full-text articles did not meet the inclusion criteria and were excluded. The primary reasons for exclusion were not mentioning publication and related bias at all (we examined systematic reviews that might have examined publication and related bias, even if this was not explicitly stated in the titles and abstracts); mentioning these biases but without assessing them; and topics being outside the definition of HSDR adopted for this project. A flow diagram for the literature retrieval and study selection process is shown in Figure 4. We carried out an updated search in July/August 2018. This retrieved 1328 new records, but no relevant methodological studies were identified.
FIGURE 4 Flow diagram for literature search and study selection process for the systematic review. This figure has been reproduced from Ayorinde et al.54 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text. (Figure not reproduced here; see the original article.)
Methodological studies investigating publication and related bias in HSDR
Four studies specifically set out to investigate publication and related bias in a substantive topic area of HSDR. The objectives, methods, key findings and limitations of these studies are summarised in Table 2. Three studies investigated publication bias in health informatics research,14,15,50 and one study explored potential reporting bias or p-hacking arising from researchers competing for limited space of publication in high-impact journals in health economics and policy literature.13
| Study (HSDR topic) | Objective(s) | Method(s) | Key finding(s) |
|---|---|---|---|
| Machan et al. 200650 (health informatics) | To determine (1) the percentage of evaluation studies describing positive, mixed or negative results; (2) the possibility of statistical assessment of publication bias in health informatics; and (3) the quality of reviews and meta-analysis in health informatics with regard to publication bias | Descriptive analysis of random sample of 86 evaluation studies and planned to construct funnel plot Examined characteristics and quality of reviews and meta-analyses (n = 54) in medical informatics | For the primary studies, 69.8% positive results, 14% negative and 16.3% unclassified For the reviews 36.6% had a positive conclusion, 61.5% were inconclusive and only one review came to a negative conclusion |
| Ammenwerth and de Keizer 200715 (health informatics) | To determine (1) the percentage of IT evaluation studies that are not published in international journals or proceedings; and (2) typical reasons for not publishing the results of an IT evaluation study | E-mail-based survey conducted in spring 2006. Participants were drawn from various working groups of the American Medical Informatics Association, the European Federation for Medical Informatics and the International Medical Informatics Association; and first authors of MEDLINE-indexed IT evaluation papers published between 2001 and 2006 (total n = 722) | Response rate 19% (136/722). 118 of the respondents reported completion of 217 evaluation studies. Of these studies, 47% (103/217) were published in peer-reviewed international journals, proceedings or books; 49% (107/217) were unpublished or published only locally. Common reasons for non-publication included ‘not of interest for others’, ‘no time for writing’, ‘limited scientific quality’, ‘political and legal reasons’ and ‘only meant for internal use’ |
| Vawdrey and Hripcsak 201314 (health informatics) | To measure the rate of non-publication and assess possible publication bias in clinical trials of electronic health records | Follow-up of health informatics trials registered in ClinicalTrials.gov (2000–8) | Trials with positive results were more likely to be published (35/38, 92%) than trials with null results (10/14, 71%; p = 0.052); the study authors reported p < 0.001 |
| Costa-Font et al. 201313 (health policy) | To examine the winner’s curse phenomenon (studies needing to have more extreme results to be published in high-impact journals) and publication selection bias, using quantitative findings on income and price elasticities, as reported in health economics research | Funnel plot and multivariate analysis to examine the association between estimated effect sizes (and their statistical significance) and the impact factors of the journals in which they were published | Meta-regression analysis demonstrated that both publication bias (reflected by positive correlation between effect size and standard error) and the winner’s curse (reflected by an independent association between effect size and journal impact factor) influence the estimated income/price elasticity |
Of the four studies,13–15,50 only one was an inception cohort study that tracked individual research projects from the start and thus provided direct evidence of publication bias.14 Studies included in this cohort were clinical trials of electronic health records registered with ClinicalTrials.gov during 2000–8. Findings from 76% (47/62) of completed studies were published. Of these, 74% (35/47) reported predominantly positive findings, 21% (10/47) reported neutral results (no significant effects) and 4% (2/47) reported negative or harmful results. Of the 15 unpublished trials, three had positive findings and four had neutral results based on information supplied by the investigators. Findings for the remaining eight studies were unknown. The authors found that trials with positive findings were more likely to be published than those with neutral findings (see Table 2), but cautioned that the sample included in the cohort may be atypical of general studies in the field.
Another study15 in health informatics was an e-mail-based survey of people who were likely to be involved in the evaluation of health information systems. Participants were asked about (1) what information systems they had evaluated in the past 3 years; (2) where they published the results of the evaluation; and (3) the reasons for non-publication of the results, if this was the case. A response rate of 19% (136/722) was achieved, with 118 respondents reporting the completion of a total of 217 evaluation studies. Most of these respondents were from an academic background (92/118), with a small number from information technology management, industry and government institutions. Approximately half of the identified evaluation studies were published in peer-reviewed journals, proceedings or books, whereas the remaining half either were published only locally (e.g. internal reports) or were unpublished (see Table 2). Reasons cited for not publishing included: not of global interest (35%), publication in preparation (31%), no time for publication (22%), limited scientific quality (17%), political and legal reasons (14%) and for internal use only (13%).
A low response rate was the major limitation of this study. Nevertheless, the survey provided some insights concerning reasons behind non-publication. Like most surveys, the study findings could be influenced by sampling, response and recall bias. It is also worth noting that publication was still being considered or under way for about one-third of the unpublished studies at the time of the survey, and, therefore, the actual publication rate might be higher.
The third methodological study50 in health informatics utilised evaluation studies identified from a specialist health informatics database that covered literature published between 1982 to 2002, and adopted three different approaches to assessing publication bias: (1) statistical analyses of the small study effect; (2) examination of the percentage of evaluation studies with positive findings compared with percentage of studies with mixed or negative findings; and (3) examination of the percentage of systematic reviews reporting positive, neutral or negative results. The authors did not identify sufficient number of studies with the same outcome measures to carry out statistical analyses of the small study effect. Although the percentages of primary studies with negative findings and systematic reviews reporting negative results were low (see Table 2), these were not good indicators for the existence of publication bias, as there is no estimate of what the ‘unbiased’ proportion of negative findings should be for evaluation studies and reviews of health informatics interventions.
The fourth methodological study13 included in this review examined quantitative estimates of income elasticity of health care and price elasticity of prescription drugs reported in the published health economics literature.13 The authors identified a positive correlation between effect sizes and the standard errors of income and price elasticity estimates using funnel plots and meta-regression, which suggested potential existence of publication bias. Having adjusted for this, they also found an independent association between effect size and journal impact factor. This suggested that studies reporting larger effect sizes (i.e. more striking findings) were more likely to be published in ‘high-impact’ journals. However, the finding still needs to be interpreted with caution, as other confounding factors could not be ruled out for these observed associations. In addition, the authors acknowledged that studies in the field concerned were often reported in grey literature, which was not examined in this study.
Systematic reviews of substantive HSDR topics providing evidence on publication and related bias
In addition to the four methodological studies13–15,50 that set out to investigate publication and related bias, we identified 184 systematic reviews of substantive HSDR topics, in which findings from assessment of publication bias and outcome reporting bias were reported. Therefore, these systematic reviews, although not undertaken to specifically investigate publication and related bias, also provided empirical evidence on this topic. In particular, three of these reviews51–53 provided direct evidence on publication bias by comparing evidence from studies published in academic journals with evidence from grey literature or unpublished studies. These reviews are described in detail below. The remaining 181 reviews provided only indirect evidence and are summarised briefly in the following section (and further details can be found in Appendix 6).
Health services research systematic reviews comparing published and grey and unpublished evidence
Table 3 provides a summary of the three HSDR systematic reviews51–53 that compared evidence from published and grey and unpublished literature. The first review53 set out to evaluate the effectiveness of mass mailings for increasing the utilisation of influenza vaccine. The review included evidence only from controlled trials. In addition to computerised bibliographic databases, the authors had access and were able to search a Health Care Quality Improvement Project database, which included records of projects carried out to improve Medicare-funded services. Of the six studies identified, one published study reported statistically significant intervention effects. However, five of the studies remained unpublished and they all reported clinically trivial intervention effects (no effect or an increase in uptake of less than two percentage points). The authors highlighted that even at the time when they were disseminating findings from this review, further mass mailing interventions were being considered by service planners on the basis of results from the first published study. The generalisability of the findings may be limited by the inclusion of a small number of trials identified from a single study registry and targeting a specific US population.
| Study (HSDR topic) | Topic | Methods of identifying grey literature/unpublished studies | Key findings of comparison between published literature and grey literature/unpublished studies |
|---|---|---|---|
| Maglione et al. 200253 (immunisation programme) | Effectiveness of mass mailings to increase utilisation of influenza vaccine among Medicare beneficiaries | Search of the Medicare Peer Review Organisation Health Care Quality Improvement Project database | Six controlled trials were identified. Only one (earliest) trial reporting modest but statistically significant improvement in vaccination rate (2–8%, depending on the format of the letter and location of the study) was published. Five subsequent trials that found smaller, clinically trivial improvement in vaccination rate of no more than 2% remained unpublished |
| Batt et al. 200451 (immunisation programme) | Costs, effects and cost-effectiveness of strategies to increase coverage of routine immunisations in low- and middle-income countries | Hand-searches in institutional documentation centres, including WHO and USAID; interviews with 28 international experts; search of grey literature databases; searches of the internet, conference proceedings and web pages of pertinent organisations | Quality of data on effect and cost-effectiveness was similar between published and grey literature, but the quality of costing data were poorer in grey literature. Inclusion of grey literature doubled the quantity of available evidence. Interventions examined in the grey literature were more up to date, associated with more complex interventions aimed at health systems, and better represented West Africa and the Middle East. Conclusions drawn from the two sets of literature therefore differed |
| Fang 200752 (organisational studies) | Relationships between organisational culture, organisational climate, and nurses’ job satisfaction and turnover | Extensive search of 35 databases, ‘footnote chasing’ and searching by author | Of the 10 associations for which findings were compared between published articles and unpublished doctoral dissertations, significant differences were found for two of them: global climate and job satisfaction; and reward orientation climate and job satisfaction. Both differences were related to magnitude rather than direction of the estimated association |
The second review51 examined evidence from grey literature and compared it with an earlier review on the same topic that included only published literature.55 The review evaluated the effectiveness and cost-effectiveness of strategies to improve immunisation coverage in developing countries. The authors observed that the quality and nature of evidence differed in some aspects between these two sources of evidence, and that the recommendations about the most cost-effective interventions would differ based on evidence summarised in these two reviews (see Table 3). The authors of the review acknowledged that the grey literature was mainly identified from international organisations and, therefore, would not have covered evidence from national governments. In addition, their literature searches were limited to English keywords and, thus, literature written in other languages was not examined.
The third review52 assessed 23 associations between various measures of organisational culture, organisational climate and nurses’ job satisfaction. The author searched for and included doctoral dissertations in the review, and made comparison between evidence from the dissertations and that from published journal articles for 10 of the associations. Statistically significant differences in the pooled estimates between these two types of literature were found in two of the 10 associations. This review was chosen as the basis of case study 2 of WP 3 and, therefore, further information concerning the methods and findings of the review can be found in Chapter 5.
Findings from other systematic reviews of substantive HSDR topics
Of the 181 remaining systematic reviews of specific HSDR topics, 100 examined potential publication bias across included studies, using funnel plots and related techniques, and 108 assessed outcome reporting bias within individual included studies, generally as part of the risk-of-bias assessment. The methods used in these reviews and key findings in relation to publication bias and outcome reporting bias are summarised in Appendix 6. Approximately half (51/100) of the 100 reviews that attempted to assess publication bias found some evidence of small-study effects.
Among the 108 reviews that assessed outcome reporting bias, reviewers frequently reported difficulties in judging outcome reporting bias due to the absence of a published protocol for the included studies. For example, a Cochrane review on the effectiveness of interventions to enhance medication adherence included 182 RCTs and judged eight and 32 of the RCTs to be at high and low risk for outcome reporting bias, respectively, but the authors could not make a clear judgement for the remaining 142 RCTs, primarily because of unavailability of protocols.56 Comparison of outcomes specified in the methods section with those reported in the results section was a commonly adopted approach in the absence of a protocol for the included studies; and authors sometimes made subjective judgements on the extent to which all important outcomes were reported. All but one57 of the reviews that assessed outcome reporting bias used either the Cochrane risk-of-bias tool or bespoke tools derived from this. In the only exception,57 the authors undertook a sensitivity analysis by imputing zero effects (with average standard deviation) for studies that would have otherwise met the inclusion criteria except for not providing sufficient data on relevant outcomes and including them in the analysis. The authors found that the pooled effect was considerably reduced after including the imputed data, although it was still statistically significant.57