Background
The two trials (see Workstream 2) identified no significant differences between either of the enhanced feedback interventions and standard practice. Process evaluations aim to understand how interventions work in practice by exploring their implementation (i.e. fidelity, dose and reach), their mechanisms of impact (i.e. participant responses to and interactions with interventions) and the contextual factors associated with variation in outcomes.28,29 The main process variable of interest in the AFFINITIE process evaluations was fidelity,22 including the fidelity dimensions of delivery (i.e. whether or not the intervention was delivered by providers as intended), receipt (i.e. whether or not participants initially understood, recalled and engaged with the intervention) and enactment (i.e. whether or not recipients enacted target behaviours and applied the intervention as intended in their clinical practice).30
The pragmatic trials had been embedded in routine quality improvement practice and, therefore, a wide range of potential contextual factors may also have influenced implementation, engagement and outcomes, beyond the interventions themselves.29,31
Aims
Specific research questions included:
- To what extent were the feedback interventions delivered by providers as designed and intended? (fidelity of delivery)
- To what extent were the feedback interventions received, understood, and engaged with as intended by transfusion clinical staff in UK hospitals? (fidelity of receipt)
- To what extent was change implemented in the light of feedback? (enactment)
- To what extent did contextual factors external to the interventions influence clinical staff’s responses to feedback? (context).
Methods
Design
The full process evaluation protocol is reported elsewhere.32 In line with process evaluation guidance,29 we used mixed methods, balancing in-depth qualitative data collection in a subsample of participating clusters with quantitative data collection across all participating clusters. We selected which aspects of fidelity to investigate (delivery, receipt, enactment), and our methodological approach to assessing these, based on published fidelity frameworks and guidance.22,30 Researchers who analysed the process evaluation data were blind to trial findings.
Fidelity of delivery
We defined fidelity of delivery in terms of the extent to which the enhanced interventions (content and follow-on) were delivered as designed and planned. This was assessed using:
- Upload checks. Materials for the enhanced and standard interventions (i.e. reports, links to the toolkit) were electronically delivered to sites by the NCA team, who first uploaded materials to the NCA website and then e-mailed listed contacts at each site to notify them that feedback materials were available for download. Staff at sites then logged in to the NCA website to access their site-specific feedback. At the upload stage, for all participating sites, we checked that the correct combination of feedback materials had been uploaded/delivered in accordance with each site’s allocation to trial arm. Any errors were corrected at this stage.
- Content analyses of the final versions of the enhanced reports, web-based toolkit and delivered telephone support sessions:
- Enhanced content. We coded the enhanced feedback reports produced by the NCA enhanced writing group for the presence/absence of the six recommended theory- and evidence-based enhancements (see Workstream 1 Intervention development and piloting and Report Supplementary Material 10) and calculated the percentage that were present.
- Enhanced follow-on. We coded the final version of the toolkit using BCT taxonomy v1,7 and calculated how many (%) of 17 intended BCTs (see Report Supplementary Material 13) were present. This was to check that no intended components/BCTs were lost when computer scientists translated the toolkit from paper to web-based delivery.33
- Telephone support component of enhanced follow-on. We logged and calculated how many (%) sites in each trial arm randomised to enhanced follow-on received at least one initial telephone support call. We audio-recorded all telephone support sessions. A subsample of sessions (≈ 10%; 12 total; six per trial, two for each of the three facilitators delivering support) were transcribed and coded into component BCTs using taxonomy v1.7 The telephone support manual specified four BCTs to be delivered during all initial telephone support sessions (prompts/cues, credible source, social support practical, and information about consequences). We calculated the percentage of these delivered in each session.
In line with published guidance, < 50% of delivery of intended intervention components was classed as ‘low’ fidelity, 51–79% was classed as ‘medium’ fidelity and 80–100% was classed as ‘high’ fidelity.30
Receipt and enactment
We defined receipt30 as the extent to which hospital staff receiving the feedback interventions initially engaged with the intervention (i.e. downloaded the feedback reports, read them, logged in to the online toolkit and completed the tools), and understood and remembered the interventions and their content. Enactment30 was defined in terms of the extent to which intervention recipients engaged in four behaviours targeted by the feedback interventions: (1) disseminating feedback reports to colleagues, (2) setting localised goals, (3) developing action plans and (4) re-monitoring performance locally.
Receipt and enactment were assessed both quantitatively and qualitatively.
Quantitative assessment
Web analytics data were used to assess initial receipt. The NCA web page on which interventions were delivered was programmed to record the number of times each feedback report (enhanced and standard) was downloaded throughout the intervention period (8 months). For enhanced follow-on, the toolkit was programmed to record usage patterns (i.e. number of logins, page views, adding/deleting characters). Data were collected for all clusters participating in each trial (trial 1, n = 135; trial 2, n = 134). For clusters with multiple sites, site-level data were aggregated to cluster level. Web analytics data were summarised using descriptive statistics and compared across interventions and trials.
Qualitative assessment
In-depth semistructured qualitative interviews were conducted with 35 participants from 21 clusters in trial 1 and with 20 participants from 14 clusters in trial 2. Participants were hospital staff in the hospital transfusion team and/or the clinical specialty being audited in each trial (surgery and haematology; see Appendix 4). Participants were recruited via a study information sheet that was sent to the NCA listed contact at each site, who was asked to forward the invitation to potentially eligible staff. Interviews were conducted by trained researchers over the telephone approximately 6 months after intervention delivery. These interviews were semistructured and included questions to explore how hospitals had responded to feedback, particularly extent of receipt (i.e. whether or not the participant recalled receiving the intervention materials, how much of the feedback materials they read and ease of understanding), enactment of the aforementioned target behaviours, and factors facilitating and hindering this. The interview topic guide is available in Report Supplementary Material 20. Interviews were transcribed, anonymised and analysed using inductive thematic synthesis.34 Themes were compared across interventions, trial arms and trials. Analysis was primarily conducted by a researcher not involved in designing and delivering the enhanced interventions. However, interpretation and proposed themes were discussed with the process evaluation research team, some of whom had been involved in the intervention design.
Context
The interview topic guide (see Report Supplementary Material 20) included questions to explore contextual influences on responses to feedback. These included items related to the inner hospital setting35 (e.g. support and engagement from colleagues, competing priorities, available resources, required skills) as well as settings35 external to the hospital (e.g. national initiatives, policies). These data were also analysed using thematic synthesis.
Results
Delivery
Overall, fidelity of delivery was high for both enhanced interventions. Upload checks confirmed that 100% of clusters in each trial received the correct combination of enhanced/standard feedback materials according to random allocation.
Enhanced content
For enhanced content, 100% of the intended enhancements were present in at least one enhanced report in both trial 1 and trial 2 (see Report Supplementary Material 21). This provided evidence that it was feasible to deliver, with good fidelity, the proposed enhancements to the design and content of feedback.
Enhanced follow-on
For enhanced follow-on, 100% of the 17 intended BCTs were identified in the final version of the web-based toolkit (see Report Supplementary Material 13). This version was used in both trials. An initial telephone support call was delivered to 89% of clusters randomised to receive enhanced follow-on in trial 1 (n = 71), and to 90% of clusters in trial 2 (n = 77). Transcripts of trial 1 telephone support sessions contained on average 96% of manual-specified BCTs, whereas trial 2 sessions contained on average 86% of manual-specified BCTs (see Report Supplementary Material 22).
Receipt and enactment
Overall, initial intervention receipt was high for enhanced content, moderate for standard content and follow-on and low for enhanced follow-on. Enactment for all interventions was moderate.
The summary of findings below focuses on trial 1, as there was little difference in receipt, enactment and contextual influences between the trials. Any differences are described narratively.
Web analytics
In trial 1, most clusters downloaded the enhanced feedback reports (summary key and full findings versions) at least once (Figure 10) [57 (82.6%) clusters receiving enhanced content and 52 clusters (78.8%) receiving standard feedback]. This percentage difference was not statistically significant (χ2(1) = 0.317; p = 0.574).
FIGURE 10 Percentages of clusters downloading enhanced and standard feedback reports in trial 1. (Figure not reproduced here; see the original article.)
In trial 1, most clusters (59; 86.8%) logged in to the toolkit at least once (see Figure 3). This was lower in trial 2 (49; 73.1%). In trial 1, the highest number of logins occurred in months 1 and 2, with the number subsequently decreasing over the intervention period (Figure 11), except in month 6, when there was a spike in logins that coincided with data collection for interviews.
FIGURE 11 Number of logins in to the enhanced follow-on toolkit in trial 1. (Figure not reproduced here; see the original article.)
Almost all toolkit enactment metrics (page views, downloads, items edited/added/deleted) had median scores of 0, indicating low engagement with the toolkit. Metrics with median scores of ≤ 1 included page views for the introduction page (median = 1), dissemination cascade (0.5) and selecting standards tool (0.5). Full descriptive statistics on all enactment metrics across both trials are available in Report Supplementary Material 23.
Interviews
Themes and supporting quotations related to receipt, enactment and context across interventions are presented in Report Supplementary Material 24–26 for trial 1 and Report Supplementary Material 27 for trial 2.
Receipt
For enhanced content, receipt was high: most participants interviewed in trial 1 recalled receiving feedback about their performance related to the elective surgery audit. Participants reported that the enhanced reports were clear, well written, comprehensible and structured in a logical order, with helpful visual representations of performance data that enabled them to readily identify their hospital’s performance. Many participants expressed a preference for the enhanced reports over standard reports. Standard content receipt was mixed: all participants in the standard content arms recalled receiving feedback reports from this audit. However, many participants reported having great difficulties with reading and interpreting feedback from standard reports. Although not all participants experienced such problems, some reported receiving summary versions produced by their hospital’s transfusion practitioner. Although the recommendations for change in both standard and enhanced reports were considered clear, some felt that the recommendations were unrealistic, for example those around the use of cell salvage machines when there was a lack of available equipment and trained clinicians (see Report Supplementary Material 24).
By contrast, receipt of enhanced follow-on was low: only six participants recalled receiving the toolkit. Many did not comment further on their understanding of and engagement with the intervention. Those who did said that they required support to use the toolkit as they struggled to use it otherwise (see Report Supplementary Material 24); however, no participants called the telephone support line that was offered as part of enhanced follow-on.
Enactment
In both trials, all participants reported enacting key behaviours targeted by the interventions, with minimal variation across trial arms and interventions. All participants shared feedback materials with colleagues, typically at hospital transfusion committee meetings. There were concerns over feedback not being disseminated to front-line staff (e.g. junior doctors). Shorter summary enhanced reports were more likely to be disseminated widely. All participants reported conducting gap analyses to set localised goals based on where their performance was poorest relative to audit standards and other hospitals. Action plans were in turn developed by participants in all intervention groups, and often shared with relevant colleagues. Some hospitals did not report ongoing monitoring of practice locally. However, some re-monitored weekly or monthly using a mix of formal and informal audits. Although enactment of target behaviours was evident, participants did not explicitly refer to using the enhanced reports or toolkit to facilitate these processes (see Report Supplementary Material 25). It is, therefore, not possible to attribute enactment to the enhanced interventions.
Context
Web analytics
The NCA website tracked downloads of all documents in the document library for each site, including non-trial-related documents such as reports from previous audits and data collection tools for other ongoing or planned audits. Figure 12 shows the number of non-trial documents downloaded by all clusters during trial 1. This large number of downloads suggests that hospitals were often engaging with materials from other transfusion audits, representing potential competing priorities that could detract from engagement with the trial audits.
FIGURE 12 Number of downloads of non-trial documents for 8 months following intervention delivery. (Figure not reproduced here; see the original article.)
Interviews
Several contextual factors were identified during the interviews with participants from all trial arms that may have undermined perceptions of, and response to, the feedback (see Report Supplementary Material 26). Many of these related to the audit side of the NCA audit and feedback intervention. The enhanced interventions in AFFINITIE focused on the feedback component and did not intervene with or attempt to enhance the audit component, which was standardised across all trial arms (i.e. same audit standards and data collection procedures).
Many participants discussed how some audit standards were unrealistic and did not represent the complexity of blood transfusion practice, particularly for different types of surgery (trial 1). Audit standards were also perceived as inflexible and not accounting for clinical judgement and knowledge when decisions to transfuse were made. In trial 2, strong concerns were expressed about the lack of evidence underpinning certain audit standards. Participants argued that adherence to such standards would be inappropriate. Consequently, several participants reported low motivation to change their transfusion practice, despite performing poorly against these standards.
These issues were further compounded by perceived issues around the credibility of the audit data, because of findings based on a small number of patient cases, the inappropriate inclusion of certain patient groups (e.g. fractured neck of femur patients in elective surgery audit) and scepticism around how performance against standards was assessed. Some viewed the audit data collection process as complicated, burdensome and confusing, sometimes leading to inaccurate data being collected and resulting in erroneous performance feedback being delivered.
Many participants reported that colleagues had varying levels of interest in blood transfusion. Unsurprisingly, transfusion practitioners and haematologists viewed transfusion as a core part of their clinical role/specialty and, thus, more of a priority. However, for clinicians from other specialties (e.g. surgery, anaesthetics) transfusion was a relatively smaller part of their role and, thus, of lower priority. This translated to mixed willingness to change practice and transfusion practitioners feeling as though they had limited influence over the practice of their colleagues (see Report Supplementary Material 26).
One theme discussed by most participants was the influence of national initiatives on transfusion practice. In 2015, NICE published new guidelines for the assessment and management of blood transfusions. These guidelines were published before the feedback for the surgical audit in trial 1 had been delivered. When participants discussed how they were responding to the feedback, and what recommendations they were working on, most participants reported how this was in response to the NICE guidlines rather than the feedback received from the audit. NICE was generally held in high esteem by clinicians and seen as more credible. Interestingly, the feedback from the NCA was seen as evidence to support the changes that they were already seeking to make based on NICE guidelines, rather than the other way around (see Report Supplementary Material 26).
Conclusions
The process evaluation demonstrated that the interventions were delivered with high fidelity, suggesting that they are feasible to deliver. Both interventions had good initial receipt, but subsequent engagement, particularly with enhanced follow-on, was low. Enactment appeared good, with hospitals across all trial arms engaging to varying extents in the target behaviours in response to feedback. Further analysis suggested that responses were driven by contextual factors, including the dissemination of national guidelines, rather than the enhanced interventions themselves.
Therefore, one interpretation is that the trials were not valid tests of the intervention, owing to low fidelity of receipt and enactment, and contextual factors that may have interfered with response to feedback. The lack of significant differences between interventions is thus to an extent unsurprising. However, although contextual factors drive practice change, as these would influence all trial arms the trial results still do not indicate that the enhanced interventions have any effect. The observed low levels of engagement with the feedback interventions also suggest that the interventions as designed were not sufficiently engaging, not feasible enough and/or potentially burdensome to use in practice. There is a need for further research to explore different ways of designing and delivering feedback to facilitate and increase engagement. Examples include automating aspects of the response to feedback (e.g. goal-setting, planning) as much as possible to reduce reliance on input from health-care professionals who have limited time and competing clinical priorities.
Nonetheless, participants reported a preference for some of the interventions, particularly enhanced content reports over standard reports, suggesting that there may still be merit in implementing and evaluating changes to the format and content of standard reports to enhance the understanding and usability of NCA feedback. This includes providing three levels of enhanced content (a brief report highlighting comparative performance and recommendations for selected key audit standards; a longer report covering all audit standards; and the long, detailed standard report).
Low engagement with feedback was in part attributed by interview participants to limitations in the upstream audit processes. This highlights the need for future research and practice to investigate enhancing the process of setting of audit standards and facilitating more accurate and efficient audit data collection.
Limitations
A methodological strength of our process evaluation was the assessment of fidelity at both the provider and the recipient level. Numerous reviews demonstrate that most fidelity assessments focus solely on delivery, and few focus on receipt and engagement.36,37 A further strength is the combined use of in-depth qualitative and broader quantitative methods to gather process data across the range of participating clusters. This included objective measures (web analytics, content analysis of audio-recorded session transcripts) that are less prone to self-report and recall biases. Limitations include participant self-selection, reporting and recall biases in the interview data. A further limitation is that researchers involved in developing the enhanced interventions were also involved in collecting and interpreting process evaluation data and were not blind to trial arm allocation during analysis. Although this may have biased interpretation, all researchers who analysed fidelity data were blind to trial outcomes. Finally, the enhanced feedback interventions were compared against usual feedback practice by the NCABT, and, as an active control condition, this shared some elements with the enhanced interventions, which may have resulted in a loss of treatment differentiation between trial arms.30