Alongside the WSs addressing the national cluster trial and the process evaluation, we planned activities to disseminate our findings. We originally proposed a series of structured stakeholder ‘roundtable’ meetings, involving, among others, clinical and management leads involved in the transfusion audits and other national audit programmes, to produce evidence-based resources to support national audit programmes to adapt and adopt the two feedback interventions developed and evaluated with our programme. As the programme progressed, we found that developing relationships and sharing feedback with a number of national audit programmes, as well as working as much as possible within existing networks, offered more fruitful approaches to engagement and dissemination than hosting further ‘roundtable’ events. We therefore changed our approach (highlighted in our 2017 progress report) to the activities outlined below, described in further detail in Appendix 5. Although we recognise that both feedback interventions (enhanced content and enhanced follow-on) were ineffective when evaluated in the trials, we have made relevant intervention materials available (see Report Supplementary Material 9–14) because they nevertheless cover important points for audit programmes to consider when designing interventions and delivering feedback.
Engagement with the Healthcare Quality Improvement Partnership and national clinical audits
We channelled the majority of our engagement activities with national audit programmes through the Healthcare Quality Improvement Partnership (HQIP), given that it is responsible for commissioning most national audits in the UK, holds regular update events and distributes guidance on audit methods. We participated in the HQIP Methodology Advisory Group from 2017 onwards. This allowed us to gain an understanding of the key methodological issues facing national audits (e.g. ensuring data validity, promoting local action following feedback). We shared emerging lessons from AFFINITIE and updates of evidence on effective feedback at HQIP seminars and with several individual audit programmes.
Audit of audits
We planned a series of audits of existing national audit programmes. We identified a baseline sample of national audit reports for 23 programmes listed on the HQIP website in November 2015. We applied a set of evidence-based and good practice criteria to these reports. We verified our assessments, where possible, with national audit leads and project managers. HQIP published Reporting for Impact Guidance, to enhance the impact of national audits in March 2016.38 We then repeated our assessment in January 2017 by applying the criteria to a follow-up sample of 20 re-audit reports (out of the original 23 national audit programmes).
We identified a range of improvements over time in the content of audit reports, for example in the identification of key audit standards, findings and recommendations, the definition of target groups for dissemination, the use of comparators and achievable benchmarks, and the presentation and specification of action plans. We also identified areas for improvement, for example reducing time intervals between data collection and feedback. We reported our findings directly to HQIP and shared them at international collaborator meetings. With further refinements, we consider that a criterion-based ‘audit of audits’ offers one efficient means of monitoring the quality of national audit reports.
The Audit and Feedback MetaLab
We are co-founders of this international collaboration, led by our co-investigator, Jeremy Grimshaw. The Audit and Feedback MetaLab (www.ohri.ca/auditfeedback/) is an international research and health-care community that aims to synthesise and share evidence on A&F, engage with health system partners and provide a trusted source of evidence and recommendations, and develop research capacity and practical expertise in A&F. We have held annual meetings to bring together researchers and audit leaders in Europe and North America since 2014. The 2017 Leeds meeting included presentations from HQIP and national audit programmes, evidence updates and discussions about challenges faced by national audits.
Joint seminar with the Healthcare Quality Improvement Partnership: what can national clinical audits learn about improving impact from the AFFINITIE research programme?
We asked HQIP to host the main dissemination seminar for AFFINITIE because we recognised that this would lend the programme credibility and extend our reach to national audits. The seminar aimed to identify lessons for national clinical audit programmes based on AFFINITIE findings and experience.
We invited national audit leads, members of the public and researchers. Ninety-nine delegates registered for the seminar, which took place in London in June 2019. We presented the trial and process evaluation findings, contextualised our work in the wider evidence base, and sought feedback on the materials and toolkits produced as part of AFFINITIE.
Participant suggestions for national audit programmes responses largely echoed findings from the intervention development work and the process evaluation (e.g. ensuring credibility of audit measures, delivering timely feedback, offering proactive support for local teams to act on feedback findings).
Patient and public involvement
Our patient and public involvement (PPI) panel contributed to the design of the research questions, shaped our application for funding, and influenced the design of the enhanced feedback interventions. As in other implementation research, there was limited opportunity for the panel to assess the appropriateness and feasibility of the interventions targeting health-care professionals. Panel members informed implementation recommendations for the wider clinical, policy and academic community.
The panel was chaired by Alan White (co-investigator), who represented the panel in Programme Management Group meetings and at a consensus meeting to design the enhanced interventions. Alan brought valuable organisational and policy experience as a member and past chairperson of the Royal College of Physicians Patient and Carer Liaison Group, past chairperson of an NHS trust, and a patient representative involved in the National Comparative Audit of Transfusion (conducted by the Intercollegiate Committee on Haematology).
Fellow members included Phil Willan (deputy chairperson of the PPI panel), member of the Royal College of Physicians Patient and Carer Involvement Steering Group and member of an expert advisory group for the Medicines and Healthcare products Regulatory Agency, who brought experience of having a transplant and blood transfusions; Graham Prestwich, a lay member of NHS Leeds North Clinical Commissioning Group Patient and Public Involvement Group; Pauline Bland, a community development worker in Bradford; and Ella Reeves, a Patient Experience and Involvement Manager at John Radcliffe Hospital. Liz Glidewell (co-investigator) supported the panel to engage with the WS leads.
Derek Calum contributed as an independent PPI representative member of the Trial Steering Group.
Significant contributions were made as identified in a framework for the role of patients and the public in implementation research39 (Box 1).
The panel welcomed the research and commented on their support for conducting research within further national audit programmes. Panel members continually expressed a willingness to contribute but reported that there were limited opportunities for PPI input throughout the programme of work given that audit standards were prespecified and patients were not the target of the intervention.
Conclusions
We have undertaken a robust evaluation of a national A&F programme in transfusion through the successful conduct of two linked national cluster-randomised trials. We identified that current blood transfusion A&F in England makes limited use of available theory and evidence about how to effectively design and deliver A&F. We designed and implemented two behaviourally modified interventions aimed at augmenting feedback by enhancing the content of the reports and follow-on support at hospitals. Modelling indicated that the enhanced feedback interventions could be inexpensive per hospital and acceptable to recipients, but they were found to be no more effective than standard feedback in increasing the acceptable use of blood.
The process evaluation explored reasons for the lack of impact of the interventions on the acceptable use of blood, and identified a lack of credibility of the audit standards and data validity, and low engagement with the audit and feedback cycle. Levels of fidelity to the enactment of feedback were variable after the initial phase, and often poor at hospital sites, with a lack of sustained interest in telephone support or accessing the follow-on materials. Other contextual issues included variable interest in transfusion that was highlighted by health-care professionals, including how to use the feedback to change the behaviour of remote transfusion prescribing clinicians. Overall, it seems likely that the lack of any effect of the enhancements was driven, in part, by factors outside the reach of the interventions.
National clinical audit programmes have a significant potential to change practice. However, the potential for feedback to deliver change depends on the integrity and validity of the upstream steps of setting standards and data quality. There is also a need to devote greater attention to understanding engagement and implementation at local levels. It may well be that our low-cost interventions have merit to enhance feedback in other contexts, but our robust assessment as successfully delivered was unable to identify any effect within a national audit of blood transfusion.
Limitations
Several factors limited the interpretation of the cluster randomised trials, including the number of participating clusters, the number of patient records audited per cluster, the effect of randomised clusters lost to follow-up, the use of complex audit data, and use of audit standards requiring the greatest change in practice applying to only small numbers of audited patients.
The assessment of fidelity at both the provider and the recipient level was a methodological strength of our process evaluation. Numerous reviews show that most fidelity assessments focus solely on delivery and not on receipt and engagement. The combined use of in-depth qualitative and broader quantitative methods to gather process data across the range of participating clusters was a further strength. We included objective measures (web analytics, content analysis of audio-recorded session transcripts) that are less prone to self-report and recall biases. Limitations included participant self-selection, reporting and recall biases in the interview data.
Reflections on the programme
AFFINITIE is an example of an ‘implementation laboratory’ that involves close collaboration between a health system delivering an implementation strategy at scale and a research team.31 Working with an existing national audit programme allowed us to efficiently conduct head-to-head trials comparing different feedback interventions that are needed but relatively sparse in the A&F literature.5 We invested considerable effort in integrating research within the NCABT and also appreciated its reciprocity in changing arrangements for the development of feedback reports and adjusting timing of data collection and feedback to align with the randomised trials. We would advise others not to underestimate the levels of communications and goodwill required to embed research within a national audit programme. AFFINITIE allowed a close examination of the strengths and limitations of the NCABT that may be relevant to other national audit programmes and the major challenges involved in a rolling programme that selects one or two new topics per year for national audits. This contrasts with several other national audit programmes that consistently focus on a core, limited set of indicators. The NCABT can address a wide variety of priorities, but perhaps at a cost of continually focusing on a more limited range of key, evidence-based audit criteria. This may underpin some of the issues with credibility of data and explain some of the limited progress made in recent years in reducing the number of unnecessary blood transfusions. We also noted that the algorithms used to assess appropriateness of transfusions were sometimes complex. Clarity of audit standards is fundamental to all stages of effective audit delivery, and in turn these have an impact on the design of algorithms applied to assess the appropriate use of interventions. It was not always clear how these algorithms were validated and, hence, whether or not they would consistently assign all transfusions as appropriate.
AFFINITIE also acted as a focus for discussions about how to improve impact with HQIP and other national audit programmes. The wide coverage of and high levels of hospital participation in the NCABT underpinned ‘real-world’ generalisability and the programme funding allowed a parallel mixed-methods process evaluation to provide critical and unique insights into the trials findings, which can inform further research and practice.
Implications for health care
We emphasise that there is still a firm evidence base underpinning A&F, including different approaches to enhance its effects on patient care on which national audits can draw. Our work has demonstrated ways of making feedback reports more accessible to recipients, but they appear unlikely to work in the absence of more favourable contexts, for example where audit data are perceived as more valid and reliable indicators of performance. Components of our interventions, such as providing three levels of enhanced content (a brief report highlighting comparative performance and recommendations for selected key audit standards; a longer report covering all audit standards; and the long, detailed standard report), might have broader applicability for national audit leaders. Our process evaluation highlights the need for national audit teams to define and enhance the key processes of setting clear audit standards, strengthen the accuracy and efficiency of audit data collection, and develop feedback interventions that can better support meaningful recipient engagement and enactment.
Future practice aiming to improve the effect of the NCABT could consider ways of strengthening the perceived credibility and relevance of feedback (e.g. by reducing the interval between data collection and delivery of feedback) and enabling more effective local responses.
We note from our economic modelling that the intervention costs per site were modest; this means that the intervention would need to have only a small impact on the volume of blood transfused for it to be cost neutral. However, it should also be borne in mind that, as shown in the trials, there was no evidence of a clinically or statistically significant effect on acceptable transfusions. Initiatives such as the international Audit and Feedback MetaLab provide a forum for the commissioners and providers of national clinical audits to interact with the research community and keep up to date with emerging research evidence.
Recommendations for research
- We have demonstrated the feasibility of embedding ambitious large-scale and rigorous research within national clinical audit programmes. Further head-to-head comparisons of different feedback interventions are needed within these programmes to identify cost-effective ways of increasing the impact that the interventions have on changing practice.
- Future studies could develop and evaluate interventions to promote meaningful recipient engagement and support focused local action in response to feedback.
- Pilot studies to ensure sufficient fidelity and identify likely effective ‘doses’ of feedback interventions may increase the likelihood of definitive trials being able to investigate cost-effectiveness robustly.
- Future health economics work could include a patient-level analysis of costs and QALYs to move beyond a process outcome such as percentage of acceptable transfusions (which does not translate as readily into meaningful economic benefit.) A future value-of-information analysis may be used to identify key uncertainties around cost parameters and adverse events associated with acceptable compared with not acceptable transfusions.