Key informant interviews and focus group discussion to explore publication bias in HSDR

This chapter presents findings from WP 5. As outlined in Chapter 2, the results are intended to develop the themes from earlier WPs and to explore views of those involved in all aspects of HSDR. Findings are presented under broad themes, including the commissioning and funding of HSDR; journal and publisher decision-making; the influence of research design and researcher conduct in HSDR; distinctive features of HSDR; and the impact of publication bias. The final section explores these themes from the perspective of patients and members of the public.

Is publication bias a problem in HSDR?

The first issue we explored with interviewees was to what extent they believe that publication bias was a feature of HSDR. A striking finding of the study is that many felt unable to respond with any certainty. This included researchers, for whom typical answers included ‘I don’t know the answer to the question’ and ‘I can’t tell really, to be honest’. Of the 16 interviewees who expressed a view, 11 believed that publication bias was prevalent and five that it was not. However, within each of these groups there was varying levels of confidence in this assessment. For example, although some argued it to be ‘rampant’, others were circumspect (‘I don’t think that it should be, but I think that it creeps in’, ‘I think it is important, but I don’t think it’s the central issue’). Those arguing against the prevalence of publication bias were also hesitant in their judgement (‘I don’t think publication bias is an issue, to be honest with you’). These findings appear to mirror the complexity emerging from previous work and further exploration helped to unpick the reasons behind this. Subsequent questioning focused on aspects or stages of the publication process of most relevance to the interviewee and focus group respondents.

Commissioning and funding of HSDR

Interviewees generally agreed that the circumstances in which HSDR is commissioned and funded influence the likelihood of publication bias. For example, many believed that highly formalised programmes of research funding were less susceptible. Some cited the NIHR HSDR programme as an example of commissioners adopting criteria-based decision-making by autonomous panels, based on external peer review:

With the NIHR or research councils, you know, you put in your research proposal and they let you get on with the research.

A key reason why such funders were seen as associated with low publication bias was the presence of formal publication requirements (including, for example, mandatory full reporting and making data publicly available), as well as conflict policies and adherence to study protocols. This in turn was associated with substantial research projects led by teams with a strong academic track record and motivation to produce published academic outputs. Commissioners in these circumstances were described as being largely ‘blind’ to considerations other than the rigour of research outputs.

This approach to the commissioning and funding of research was contrasted with end user sponsored research, characterised by high levels of interest and involvement of the commissioners in the conduct and reporting of results. One respondent, for example, felt that government bodies directly commissioning evaluations of their own programmes were often ‘desperate for those to be shown to be positive’ and another spoke of charitable foundations requiring striking findings in order to ‘feed their communications machine’:

A lot of the research will be funded by apparently objective, non-commercial funding agencies, but there will still be the management consultancies and charities, there will still be government involvement. So I don’t think we can be too complacent.

Many of the interviewees who worked as researchers in such circumstances nevertheless asserted the relative autonomy afforded to them, indicating that any subsequent bias was unlikely to be direct or overly coercive:

What I worry about is this subtle, indirect pulling of punches; so softening the edges of the messaging. Because you get comments coming back from policy officials saying things like ‘you’ve given us a kind of glass half empty story. Can’t you turn it round?’. And of course, often it is a matter of language and emphasis.

Interviewees indicated that pressure from funders was of a lesser magnitude than that generated in much clinical research, from their perceptions or experience, and that there were less likely to be major conflicts of interests:

There’s a danger of framing it in the same way as in medical research because the stakes are very high in medical research and there’s big commercial involvement through the pharmaceutical companies, who’ve been repeatedly caught ‘off side’, as it were. Whereas I think the same issues don’t apply, at least universally, in health service research.

I think because so much of the funding we have is competitive and most of it is won through committees that actually make it very difficult, I would imagine, for the result of the previous study to affect the funding. Whereas traditionally, you do a trial and the company wants to give you more money to do more research in the area. I’ve never been in that situation, [laughing] sadly.

Despite this, the frequent absence of a formal requirement for peer-reviewed publications from such government bodies was identified as a potential source of bias:

It effectively leaves the onus on getting peer-reviewed publications up to us which is exactly when publication bias is more likely.

Interviewees noted that locally commissioned evaluations and audits are rarely intended to be reported beyond the immediate organisational context, and that therefore it is inappropriate for them to be subjected to the rigours of formal peer review before publication. Publication bias was therefore seen as a result of publishing of such data beyond the intended scope and setting of the data collection exercise.

Many interviewees made a distinction between research and other forms of knowledge:

I think the boundary between research and service improvement is really blurred and if you’re just using someone’s routinely collect data, not purposely collected for a research study, but just kind of scraped off their systems, and you’re using it for a model which will help them improve their systems, then I don’t call that research personally.

I suppose then it becomes a question of what counts as publication bias and where the borderline is with something that actually really isn’t research, or shouldn’t be judged as such, shouldn’t be kind of assessed as such. I think it entirely depends on how the project’s been set up . . . many small-scale quality improvement projects are not generalisable analytically or statistically and therefore I’m not terribly sure why you’d try to publish them other than using them in a local setting.

One interviewee considered that ‘editorial control’ was the primary means of differentiating research and consultancy:

At [name of organisation] we did both consultancy and research, and the distinction was research meant that there was an explicit agreement up front, that we the researchers would be able to have editorial control so that we could seek publication in a journal. In other words, that the funder could not stop us publishing.

Publication bias in journal decision-making

Within the sample, researchers were most likely to argue that publication bias was pervasive in academic publishing and especially in editorial decisions over whether to accept or reject HSDR journal submissions:

I think [positive effects] are of interest to all the journals and some are more explicit about it than others . . . All of the journals, despite their veneer of academic impartiality, are highly news oriented and they are kind of pushing for space in that crowded market place.

You would never ever send something non-significant to a top journal, that’s my feeling.

You do quite often find editors will say, ‘This is too similar to another paper that’s already published’.

The responses of journal editors within the sample were more mixed. For example, some argued that authors were primarily responsible for any publication bias at the analysis and write-up phase of the research process (see Research design and researcher conduct in HSDR). However, most conceded that factors other than scientific quality influenced publication decisions, and many of these factors coalesced around the requirement for novelty of some form. One editor of a medical journal used the term ‘saturation bias’ to describe the diminishing likelihood of publication of papers once ‘the message is already out’, irrespective of scientific merit. Another identified study scale and certainty of results as being important:

If you have a very clear-cut answer on either side, positive or negative, obviously any journal will be very interested. We’re not shying away from a negative study by any means, if it’s definitively negative. I think it’s when it’s non-definitive because the study is too small, or inadequately powered. I think we’re a little bit more hesitant with those.

Editors of medical journals acknowledged that HSDR submissions, in general, were less likely to be accepted, especially when they did not follow a trial-based design. Editors of journals in subdisciplines of HSDR described a range of characteristics predisposing journal editors to accept HSDR manuscripts for publication. Along with methodological rigour and study scale, these included extent of methodological and theoretical innovation, and the political salience of the research topic or intervention:

I think [strength of effects] wouldn’t affect publication in an OR [operational research] journal, but then usually they would be looking for you to show that your method was better than somebody else’s method. And if in a particular case you had shown that: you know, you’d applied your method to some problem and it hadn’t really made any difference, I think that would be harder to publish.

I do think that there is a very strong relationship between, if you like, the size of the funded project and its chances of being published, irrespective of its findings.

In our kind of publishing, if you don’t see any change, for example, that’s not a very interesting study theoretically. Nobody is really very keen to publish articles that explain why things stay the same and so, to that extent, there is some bias in that we think we know why things stay the same.

These views and experiences were echoed in interviews with researchers:

There are some systematic biases there and I think people, especially editors and peer reviewers, are sort of denying them in a way. They’re looking for the best quality research and they’ve got this idea of what best quality research looks like. And they do privilege the very novel, you know, highly rigorous, high impact studies over those that are sort of more mundane but still important.

I think it’s absolutely right that poor or null effects are less interesting, but they’re only less interesting if it’s a topic where people really want it to work; if it’s a topic where people don’t want it to work, then our hypothesis is great and, you know, the [name of journal] will publish it. So I don’t think it’s a simple as just positive results.

These factors were seen as diluting the influence of the magnitude and statistical significance of quantitative findings. By comparison, the latter were generally attributed higher importance in medical journal decision-making. This suggests that although HSDR journals may be less likely to exacerbate publication bias through their editorial practices, they may be substituting this with other forms of bias associated with preference for novelty.

Responses were less consistent with regard to the influence of the editor and peer review roles in publication bias. Although some editors referred to peer reviewers as ‘human funnel plots’, other interviewees believed that publication bias was rarely a major consideration in the peer review process:

Certainly as a reviewer and as an editor I don’t see many questions or prompts asking about the presence of publication bias. I don’t see many reviewers’ comments about publication bias unless they’re aware of particular studies that have been missed.

However, the proliferation of peer-reviewed journals was seen as mitigating the threat of publication bias, for example in HSDR systematic reviews. This meant that the status of the journal was a secondary consideration:

I don’t think it matters a toss where it’s published as long as it’s published.

Obviously, there’s a certain level of prestige that comes through getting it in a high impact factor journal but, nowadays, everything is findable in PubMed so as long as it’s there, I’m much less bothered about where it goes.

Research design and researcher conduct in HSDR

Interviewee perspectives were again mixed in relation to the role played by researchers in publication bias. Although some argued that in trial-based research, behaviours, such as selective reporting and ‘data hacking’, were no longer possible, others believed this to be commonplace, especially in association studies:

The more subtle and more common type [of publication bias] is the issue around hypothesis testing when you’re the statistician doing the analysis and when you do a lot of what you would, in your mind, neatly label ‘preliminary analysis’; a lot of unofficial hypothesis testing – ‘see if this works’ – and subgroup analysis . . . That is a massive problem.

Others believed there to be a temptation to search for positive results when researchers were in some way connected to the intervention being evaluated:

It’s very hard to find adverse effects of service delivery and part of that is either that the people who are introducing those models are sort of strongly advocating them, you know, it’s sort of like airing your dirty washing. And I suspect the other thing is also it’s very hard to identify unintended consequences because they have a lens that focuses on the evaluation framework and don’t look more widely at knock-on effects. So that would be a variation of selective outcome reporting bias but one particularly unique to Health Service and Delivery Research.

It depends if you think bias is the absence of the results or it’s the change in what people report. So people may have been trying, for example, reduce [hospital infection] rates and if in fact, they stay the same, but they manage to cut readmissions for whatever reason, they may then report, ‘Oh, this is an effective intervention to cut readmissions rates’, although it wasn’t actually designed that way in the first place.

For example, when I evaluated [name of government initiative]. I didn’t care whether it worked or not but if I have been part of developing a particular service, so for example I’m developing a medical adherence intervention at the moment, there is going to be part of me invested in wanting that to be positive.

There was general support for the view that research teams might take longer to submit negative or null results for publication. This was seen to be related to general levels of ‘excitement’ associated with positive findings:

I’ve been involved in a couple of large implementation research trials and neither of them came out with positive or significant outcome findings . . . To motivate yourself to get that negative finding trial out there, it’s a real challenge. I think that there is something about not wanting to share something that you feel failed in some way, which is ridiculous.

The psychology is, you know, ‘we looked for something, we didn’t find it, that’s a bit disappointing. Can we be bothered to write it up? It may not get published anyway. Let’s go ahead and look at something else’, kind of thing. I think that’s understandable and human nature.

Some respondents identified a risk of publication bias when evaluators are responsible for developing the intervention, and many felt that publication bias was a danger within much service improvement research:

There’s strong movement within quality improvement, which is a very enthusiastic kind of movement, one that celebrates success in lots of ways. And some relatively large claims are made for some work that’s often done with relatively simple ways of looking at data . . . Now that is all maturing rapidly but my instincts are that because of that strong enthusiasm and the desire for things to be seen to work, there’s a risk.

This problem was seen by the two managers in our sample to be compounded by a health-care environment that rewarded ‘good news’ stories about health-care innovations and improvements:

It’s a fear of failure and therefore you cannot be seen to try something that doesn’t work because that’s not celebrated.

Table 13 summarises the factors seen to increase or decrease the risk of publication bias in HSDR.

It was clear from the interviews with researchers that, to the extent that they considered researchers guilty of publication bias, this was partly driven by the perceived expectations of journals and universities. Some argued that ‘job prospects’ are linked to high-impact journal publications. For example, junior researchers within our sample were more likely to believe publication bias to be widespread:

Even though we are all academics and we know about publication bias when it comes to our own work we’re thinking, ‘Well, which journal will be interested in these findings?’. If they’re positive ones it’s easier to get kind of excited about it, as it were, when thinking about likelihood of acceptance by a bigger journal. You know, if you’re evaluating some new policy and you find it doesn’t work then clearly that is vital but I think the threshold is higher for a negative finding.

A small number of respondents identified incentives to researcher-instigated publication that went beyond the individual’s interests:

Sometimes important parts of HSDR studies don’t get published because the organisations, universities, don’t see the value of that.

In the USA the amount of funding that’s available for health systems improvement work is quite limited so there’s definitely a competition aspect that plays in, which leads to issues with individual investigators; obviously they want to show that the work that they’re doing is worthwhile which increases the pressure to publish positive results.

For reasons outlined in the following section (see Distinctive features of HSDR), senior researchers were more likely to underplay the pressure to publish positive results:

Almost all of my studies show that whatever I’m looking at doesn’t work . . . The answer to your question is that, in my own personal research, it would not cross my mind not to publish something.

Distinctive features of HSDR

As noted, a significant minority of researchers in the sample felt that, in broad terms, HSDR was less susceptible to publication bias than clinical research. They attributed this to characteristics they considered distinctive and typical of HSDR. These included the complex nature of interventions, which generated uncertainty in evaluation, and the context-specific determinants of outcomes, which meant that definitive conclusions were often difficult to draw:

I wonder whether epistemologically they are doing different things; essentially within the [Health Technology Assessment] community it’s much more of an aggregative process coming up with, you know, pooled overall effects, etc. But if you assume that there will be a great deal of heterogeneity in health service delivery research then perhaps epistemologically you’re trying to do much more about making sense of options and mapping different models. And linked to that is the interplay of context so that within an HTA [Health Technology Assessment] context there’s almost the assumption that a drug will work in a relatively similar way across multiple contexts, whereas a service delivery intervention is very context-specific. So maybe people are bringing different quality markers or a different sort of epistemological view.

They further argued that this complexity and contingency often prevented study randomisation and replication:

The sort of research I do, it is rarely possible to randomise the intervention, so if you’re evaluating the [name of national government initiative] for example, you can’t persuade NHS England to randomise it to practices. So much of the research designs have been observational.

Allied to this was the ‘multifactorial’ and ‘multicomponent’ nature of much HSDR study design, and the simultaneous addressing of multiple research questions, so that strength and direction of findings was assigned less overall importance:

I think in a lot of clinical research the effectiveness question is the only question that people are really interested in. They still pretend to be interested in mechanisms of action, you know, why something didn’t work. There’s still a pretence that we’re interested in the cost-effectiveness but at the end of the day that effectiveness question is the absolute king. Whereas in health services research that’s less of an issue, I think. Usually, we do have an effectiveness question that we can rarely answer with a randomised control trial; we have to address with something lower down the hierarchy of evidence. And because those designs have less status than the RCT, there’s more of a balance in the status of the questions and the methods that are being used within health services research . . . You know, in meetings everybody’s talking about all the different components and how they’re going. You don’t have to fight to get any component onto the agenda.

However, some interviewees expressed frustration at the relative inattention paid to outcomes and the primacy given to implementation in HSDR publications, highlighting the challenges this posed for subsequent evidence synthesis:

We’ve done work on user engagement in major health service reforms and it’s kind of, ‘Well, what’s the outcome? Is it whether you actually got the change implemented or whether you did the right consultation?’. The fact that the change was or wasn’t implemented may not be the key outcome but that’s often the one that’s reported, with the description about how you got there. So you may not have information about what might be considered the key outcome.

The equivocal nature of many HSDR results was cited as an additional reason for reduced levels of external scrutiny and verification from, for example, media, government and industry. This was seen as a further check on the pressure to find positive associations and effects.

Tackling publication bias

The range of views on the prevalence of the publication bias ‘problem’ was reflected in interviewee comments on potential ‘solutions’. All of those who considered publication bias to be widespread (and many of those who did not) expressed support for pre-registration of studies and the strict application of study protocols containing statements of pre-set outcome measures:

It’s very easy to do and I think it’s also a mark of sound intellectual practice to actually say what you’re going to study and why you’re going to study it, so we don’t end up seeing fishing expeditions.

A subgroup of interviewees believed this would result in unwarranted and inappropriate constraints on researcher conduct (‘feet of clay’). As well as the practical challenges in large, multistrand studies, interviewees argued that sensitivity to changes in context during the lifetime of a research programme would be compromised:

You’re not dealing with nature, you’re dealing with people. So you want to be able to take advantage of what you learn, not just think, ‘Alright, we’ll stick with the state of knowledge we had when we wrote the proposal’. But you need to be quite transparent about why you’ve changed what you’ve done.

If you take a recent study which we’ve done in which we probably had about eight substudies. All using different methodologies. Are you going to register each of them? And difficult to register the programme as the whole, not least because the nature of the grant is that things change as your 5 years goes on. So I remain to be persuaded of the benefits of pre registration for HSR [health service research] trials principally because pre registration for drug trials is to get at the very clear problem which you’ve articulated, which I don’t think exists in quite the same way for us.

Other strategies identified for tackling publication bias included mandatory publishing and development of repositories of null findings; training and awareness raising in publication bias; a general improvement in the rigour of HSDR studies (especially in the field of service improvement); strengthening of research teams (e.g. through partnering with senior academics); and greater incorporation of publication bias checks in journal peer review.

Impact of publication bias in HSDR

As noted, some interviewees believed publication bias to be common in HSDR and cited the potential distorting effects of this on syntheses of the evidence, for example through systematic review. However, others identified the proliferation of journals publishing HSDR as partly counteraction to this trend.

Interviews (including with service managers at senior and local levels) revealed some apparent contradictions in relation to the prevalence and impact of publication bias. On the one hand, there was a widespread view that decision-making was often carried out without due regard to the published evidence, such that publication bias was less of a concern. Interviewees were at pains to emphasise the weak relationship between evidence and the decision-making they observed:

You have to consider just how poor the access of NHS managers to research evidence is. That is my experience and when I came to realise there’s this whole world out there of people doing health services research it was a revelation. But it was something I had to discover for myself. There is a very low awareness of research work generally, I mean, in the management class. That’s a very broad statement, but that’s from my experience of having worked in it. The way you made decisions is very much based on what is in front of us and our conversations with colleagues, rather than through reference to literature.

There might be a sad reason why publication bias would not be such a problem in health services research, because the decision-makers don’t pay attention to the evidence anyway.

On the other hand, respondents identified similar sources of bias generated from within the decision-making environment itself and perpetuated, for example, through professional periodicals and information sources:

It’s the same in my organisation. I can’t think of anyone who’s really stood up and said ‘We tried this and I’m gonna write it up and get it published because I don’t want anyone else going through this pain or spending energy on it’. We just don’t seem to have a culture of doing that . . . I think when you’re getting a decision from your governing body or your board to try something there is an expectation from them that it will work. My director says, ‘Oh, we’ve been asked for some good news stories’ and in my head I want to say ‘well the good news could be that we tried this and it didn’t work so we can’t go through it again’.

If you take something like [name of professional journal], most of the papers are subtitled ‘How my last two year project has transformed health care’ you know? They’re less frequently constructed as ‘How my last two year project proved that this didn’t work.

If we run something like the [national health services initiative] and we’ve committed many billions of pounds to what is notionally a pilot project for other people to copy, it’s quite hard for the taxpayer to understand that at the end of spending many millions of pounds on testing something, we came to the conclusion we shouldn’t do it anymore because it’s a waste of money. Now, there is an acceptance in the drug industry that you have to kiss a lot of frogs before you find a prince and that’s not really the culture in management. The culture in management is everything needs to work every time.

One researcher in the interview sample identified a further knock-on effect of this on the research community when deciding which interventions to evaluate:

There’s great reason to believe that the picture of what’s going on, for example, in the NHS is heavily influenced by reporting bias, in other words, people do a successful initiative and they inter-disseminate it. That doesn’t translate into quantitative bias but upstream if you think that that means that these interventions are then more likely to get funding to be explored, then that will lead to a sort of distortional bias in terms of interventions that are available.

Patient and public views: findings from the focus group

The focus group was structured so as to enable discussion in three phases, starting with general discussion of evidence, followed by discussion of publication bias in clinical research and, finally, discussion of publication bias in HSDR. Examples were used to illustrate publication bias in the second and third phases. The over-riding aim was to gather perspectives on the importance of this topic from a patient and public perspective.

Participants described receiving an ‘avalanche’ of information, but said that they would search for more detailed information on the stories that piqued their interest (i.e. if it affected them, a family member or a friend). There was acknowledgement that news stories as presented might not be an accurate reflection of the research evidence and the need for critical reflection. Participants described the need to be ‘critical’ of sources and to consult and listen to varied perspectives. Participants also discussed different sources of information that they would consult, including social media (e.g. Twitter; Twitter, Inc., San Francisco, CA, USA), books (written by authors they trusted), to lectures and articles in peer-reviewed journals.

Key points of consensus included:

• There is a need to identify ‘trustworthy’ sources that accurately summarise evidence.
• Some people accept what is reported in the mainstream media unquestioningly, with associated risks and harms.
• When reporting research findings the mainstream media typically provide only limited information on how the study is funded and conducted, making it impossible to judge extent of potential bias.
• Some level of bias is unavoidable (e.g. ‘bias starts when you decide what you’re going to study’), so transparency of reporting is of the utmost importance.

The recent clinical example of reboxetine6 (Edronax®, Pfizer Inc., New York, NY, USA) (for anxiety and depression) was discussed. Participants used words, such as ‘worrying’, in relation to the example and raised ethical objections. For example, participants noted that patients participating in studies might not be aware that drug companies were not publishing all of the evidence. Participants also discussed the dangers of drug companies funding trials and the importance of patient involvement in setting research aims and outcome measures.

A second example from HSDR, related to the use of mass mail-outs, was discussed (see Maglione et al.,53 also described in Chapter 3, see Table 3). This prompted discussion of the need for public resources to be used as efficiently as possible, reducing waste and the threat posed to this by publication bias of this kind.

Across the two examples, participants made the following observations:

• Much extra attention is given to new interventions (‘the latest thing’) that promise to solve major problems and the tendency, therefore, is to be biased towards positive findings for new interventions.
• There are dangers in measuring and reporting only short-term outcomes and neglecting longer-term effects, whether positive or otherwise.
• Participants were aware that researchers might be under pressure to publish interesting results and might be deemed a ‘failure’ if they fail to do so.

This led to a series of recommendations put forward by participants:

• Critical appraisal – participants emphasised the importance of educating people at a young age to be critical of information they receive, especially in popular and social media.
• Clear reporting of information about who funds should be compulsory, with any conflicts declared.
• Addressing the pressures within universities to ‘get results and publish’ was seen as a priority. People should be allowed to ‘make mistakes’ (i.e. embark on studies resulting in null results) because these present ‘opportunities for learning’.
• Participants argued that other aspects of studies (e.g. methods) might be of interest and warrant publication where results are unremarkable.

Participants also agreed that research should be able to be replicated and, therefore, all sources of data should be made fully available for future researchers:

Overall, participants argued that having strong PPI throughout a study is critical. This needs to be carefully planned and inclusive of all groups, including those typically under-represented. They argued that PPI advisors should have roles in designing studies and ‘challenging’ the experts so that bad practices associated with publication were less likely to happen.

This chapter summarises diverse views on publication and related bias obtained from our interviews with a wide range of stakeholders of HSDR, including researchers of varied seniorities and specialties, journal editors, research funders and service managers. In addition, we present findings from a focus group discussion, which was planned with PPI involvement and highlighted service users’s and the public’s concerns about the potential impact of publication and related bias in terms of research waste and the influence of evidence on health service decision-making. We discuss the implications of these findings in relevant sections in Chapter 8.