Workstream 1: intervention development and piloting

Background

The National Comparative Audit of Blood Transfusion (NCABT) approach to designing and delivering audit and feedback (A&F) has largely remained unchanged since the organisation was established in 2003. Yet repeated audits have identified high proportions (20%) of unnecessary transfusions,4 raising questions around the effectiveness of the programme’s current approach and whether or not this could be improved. Various theories, in particular control theory (Figure 2), have been used to describe how A&F may operate.5,6 A number of behaviour change techniques (BCTs)7 are associated with each step in the iterative self-regulatory A&F ‘loop’ described by control theory.8

FIGURE 2 Adapted representation of the control theory6 self-regulatory loop with corresponding behaviour change techniques. (Figure not reproduced here; see the original article.)

Control theory is consistent with the finding from a Cochrane review9 that A&F is more effective when it is accompanied by explicit goals and action plans. This aligns with behavioural science principles that behaviours are more likely to be enacted if they are explicitly specified in terms of who needs to do what, differently, to whom, where and when [i.e. Actor, Action, Context, Target, Timeframe (the AACTT framework)].10

Enhancing existing interventions requires first specifying what is currently done.11 Investigating current practice can be facilitated through frameworks for identifying, characterising and standardising the reporting of intervention components (i.e. BCT Taxonomy v17) and for investigating factors influencing health-care professional behaviours [i.e. the theoretical domains framework (TDF)].12

Aim

We aimed to specify current feedback practice as a basis for developing two theoretically enhanced A&F interventions focused on the design of feedback reports (‘enhanced content’) and supporting hospitals to implement change in the light of feedback (‘enhanced follow-on’), respectively.

The objectives to inform intervention development were to:

• specify the content of existing A&F reports delivered by the transfusion National Comparative Audit (NCA) and examine the extent to which they include theory-based BCTs8 and evidence-based characteristics9
• investigate who currently receives feedback from the transfusion NCA, how hospital staff respond to feedback, and the barriers to and enablers of acting on feedback
• pilot the developed interventions in the hospital context to assess their feasibility and acceptability.

Methods

Based on detailed methods published elsewhere,13 workstream (WS) 1 comprised three mixed-methods sub-workstreams.

Workstream 1a: enhanced content

Design

The workstream design was a structured documentary content analysis.

Materials

We used 12 feedback documents from three recent transfusion NCA audits (four per audit): Red Blood Cells in Neonates and Children 2010 (John Grant-Casey, John Radcliffe Hospital, 2010, personal communication), Platelets in Haematology 2011 (John Grant-Casey, personal communication) and Medical Use of Blood 2012 (John Grant-Casey, personal communication). Documents included summary and full findings reports, action plan templates and presentation slides.

Coding framework

Drawing on available A&F evidence,9 theory6 and behavioural science frameworks,7,10 we developed a coding framework (see Report Supplementary Material 1) to specify current feedback reports in terms of:

• component BCTs, using the 93-item BCT Taxonomy v17 to identify and categorise BCTs
• behavioural specificity of each audit standard, feedback item and recommendation, coded according to the AACTT framework10
• seven evidence-based feedback characteristics, namely whether or not feedback was provided in multiple formats; was delivered by respected peers; was delivered more than once and closely following audit data collection; included explicit goals and action plans; targeted behaviours where baseline performance was low; included multiple comparators (e.g. achievable benchmarks, top 10%); and was supportive rather than punitive in tone.9

Procedure and analysis

The framework was applied to code each document, with three reports (one per audit) double-coded by two researchers. Inter-rater reliability was assessed using Cohen’s kappa.14 For each document, we assessed the presence or absence of all 93 BCTs in the taxonomy. We assessed whether 11 BCTs consistent with control theory (see Figure 2) were present or absent in each document. We calculated the proportion of audit standards, feedback items and recommendations for which AACTT were specified. We investigated how many of the eight evidence-based feedback characteristics were identified across documents from each of the three audits.

Intervention development

Evidence- and theory-based feedback components that were absent or identified infrequently were considered possible enhancements to feedback content. These enhancements were incorporated into prototype feedback reports and a ‘how to’ guidance manual for the transfusion NCA for designing feedback reports with theory- and evidence-based content. The WS1a findings, the potential feasibility, and the theoretical and clinical face validity of proposed enhancements were reviewed at a multidisciplinary workshop with behavioural scientists, clinicians, and PPI representatives.

Workstream 1b: enhanced follow-on

Design

We took a multiple case study approach, using semistructured qualitative interviews and observations of hospital transfusion committee meetings.15

Participants

Four hospitals in England that routinely participate in audits by the transfusion NCA were purposively sampled for variation in infrastructure (e.g. district vs. teaching hospitals). We purposively recruited five to eight health-care professionals from each hospital from roles involved in prescribing and administering transfusions, and implementing change in response to feedback.

Materials

The interview topic guide (see Report Supplementary Material 2) investigated how feedback is operationalised in hospitals, including four key behaviours corresponding to the second half of control theory: dissemination of reports, local goal-setting, action-planning and problem solving, and re-monitoring. Questions exploring the barriers to and enablers of acting on feedback were structured around 12 domains of the TDF16 (e.g. knowledge, social influences, beliefs about consequences, environmental context and resources). We developed an observation sheet (see Report Supplementary Material 3) to record field notes in Hospital Transfusion Committee meetings on when and how A&F was discussed, communication, staff engagement and group decision-making.

Procedure

Interviews were conducted one to one with consenting participants, and then audio-recorded and transcribed verbatim. Two researchers observed one Hospital Transfusion Committee meeting at each hospital, with the consent of the meeting attendees, and recorded field notes.

Analysis

Interview transcripts were analysed using combined deductive framework (based on the TDF) and inductive thematic analysis.17 The key domains were identified by considering frequency (≥ 60% participants) and participants’ expressed importance.17 Field notes were summarised thematically. All themes were reviewed by three researchers until consensus was reached. Each case (hospital) was analysed separately. Case 1 was first analysed in full, with data from subsequent hospitals (cases 2–4) analysed by using themes from previous cases while allowing new themes to emerge.

Intervention development

We selected potential BCTs to address the identified barriers/enablers within key domains by consulting matrices that map BCTs from Taxonomy v1 against domains from the TDF,3,18 plus those BCTs consistent with control theory. These were incorporated into a prototype toolkit, including resources to support hospitals to disseminate feedback, set goals, action-plan/problem-solve, and re-monitor. The proposed BCTs and toolkit were discussed at the same multidisciplinary workshop as described for WS1a.

Workstream 1c: feasibility piloting

Four hospitals in England consented to participate in a pilot audit on the medical use of blood. Data were extracted from patient notes based on the audit standards from the 2012 Medical Use of Blood audit (John Grant-Casey, personal communication) and used to develop enhanced feedback reports using the prototype templates. Two researchers independently coded the draft enhanced report and toolkit to ensure that the intended BCTs and enhancements were present (i.e. delivered with fidelity). The prototype enhanced feedback reports and toolkit were then delivered and explained by two members of the research team to the hospital transfusion team during a 1-hour face-to-face training session.

Three months later, qualitative interviews were conducted with staff at each hospital. Staff were purposively sampled as per WS1b. We conducted one-to-one semistructured and think-aloud19 type interviews, which commenced by asking participants if they recalled receiving the feedback reports or toolkit. If yes, a semistructured interview was conducted to explore feasibility and acceptability of these.20 If no, we conducted a think-aloud interview, whereby staff were presented with the reports and toolkit and asked to verbalise their immediate reactions to the interventions. Both topic guides are available in Report Supplementary Material 4 and 5.

Interviews were audio-recorded, transcribed verbatim and analysed using thematic analysis. The WS1 research team (FL, NG, JF, LG and SJS) then considered potential ways of refining the interventions to address any themes indicating threats to feasibility and acceptability (e.g. adding or removing components, modifying format and modes of delivery). Potential refinements were then discussed at a wider multidisciplinary meeting to reach consensus on which to implement.

Key findings

Workstream 1a: enhanced content

Average inter-rater coding reliability was high (κ = 0.81, range 0.80–0.96). Overall, existing feedback documents incorporated a limited number of theory-and evidence-based components.

Behaviour change techniques

Documents contained on average 8 of the 93 BCTs in Taxonomy v1 (range 3–14) (see Report Supplementary Material 6). They included, on average, half (5.6, 51%) of the 11 BCTs associated with control theory (range 2–7). The most frequent, identified in at least one document from each of the three audits, corresponded to the first half of the control theory loop: ‘goal-setting’ (i.e. audit standards), ‘feedback on behaviour’ and/or ‘outcomes of behaviour’ and ‘discrepancy between behaviour and goal’. Only one BCT, ‘action-planning’, from the second ‘adaptive response’ half of control theory was identified from all three audits (see Report Supplementary Material 6). There were no identified BCTs encouraging review of audit standards, setting of localised goals or ongoing self-monitoring.

Behavioural specificity

Overall, behavioural specificity of audit standards, feedback and recommendations for change was low. Although the action (e.g. measuring pre-transfusion haemoglobin) and target group (e.g. neonates) were often stated, the actor (e.g. consultant haematologists), context (e.g. ‘paediatric ward’) and time (e.g. within 3 days of transfusion, preferably same day) were rarely stated (see Report Supplementary Material 7). Across documents, on average only half (range 0–100%) of the feedback related to behaviours specified in audit standards, indicating that there was a high volume of extraneous feedback.

Evidence-based characteristics

We identified only two of the eight evidence-based feedback characteristics (see Report Supplementary Material 8). Feedback was always provided electronically, in writing, and, rarely, also graphically or using other modalities. The stated feedback provider was always a regulatory body, rather than a respected peer. Feedback was delivered only once, typically 12 months after data collection. Average national performance in relation to audit standards was 72% (range 64–86%), suggesting that baseline performance for targeted behaviours was reasonably high, with limited room for improvement. Although the feedback provided was not necessarily punitive, supportive BCTs such as ‘social reward’ and ‘social support’ were delivered infrequently.

Although the two evidence-based characteristics ‘include explicit goals and action plans’, and ‘use multiple comparators’ were identified from all three audits, these were not fully operationalised. Documents included goals, and some recommendations for change, but these were not explicit, as evidenced by the limited AACTT specification. All audits provided comparative feedback on peer performance (nationally and/or regionally). However, no audit compared practice against achievable benchmarks of care (see Report Supplementary Material 8).

On this basis, six recommendations for enhancing the content of feedback reports were identified and agreed with multidisciplinary stakeholders:

1. include at least one BCT corresponding to each step of control theory
2. ‘be specific’ – phrase audit standards, feedback and recommendations in terms of who/what/where/when
3. ‘be relevant’ – include only feedback related to audit standards and take a graded-entry approach by producing feedback reports with varying levels of detail, from key findings to supplementary reports
4. include multiple comparators
5. include positive messages recognising good practice
6. improve feedback document presentation (e.g. provide feedback visual format, such as graphs, make writing legible, use a consistent layout, personalise feedback).

Enhanced prototype reports and the enhancement guidance manual are available in Report Supplementary Material 9a–c and Report Supplementary Material 10a–b.

Workstream 1b: enhanced content

Full results are published elsewhere.15 In summary, we interviewed 25 participants, including nurses, junior doctors, registrars and consultants from haematology and other clinical specialties, and quality improvement managers.15

Who receives feedback?

Report Supplementary Material 11 depicts the dissemination pathway of feedback reports for each hospital. Dissemination was a key barrier to implementing feedback across all hospitals. Feedback was often initially received by the hospital transfusion team, but then not disseminated more widely to clinical staff from other specialties who prescribe transfusions or to more junior staff. The extent to which feedback from the NCA was discussed at hospital transfusion committee meetings also varied.15

What do hospitals do with feedback and what are the barriers and enablers?

There was considerable variation in how feedback was received, shared and responded to across hospitals. The key barriers to and enablers of feedback across all cases fell into eight theoretical domains (see Report Supplementary Material 12):

1. social influences – not sharing and discussing feedback with colleagues; lack of influence over practice change and support from peers; desire for feedback to be delivered from a familiar, respected colleague; the view that comparison against national averages is not useful for identifying areas for improvement
2. behavioural regulation – not setting goals and action plans as a team; need for support materials and tools to facilitate planning locally; having to amend feedback to make it relevant locally
3. social professional role/identity – lack of clarity about who is responsible for A&F
4. knowledge – variable and limited awareness of transfusion NCA
5. motivation – competing priorities and audit fatigue
6. environmental context and resources – lack of staff and resources to conduct re-audits and/or implement change
7. beliefs about consequences – A&F does change practice
8. memory attention decision-making – not recalling the feedback materials; noticing new information only when it is different or clinically relevant.15

The prototype enhanced follow-on intervention included 19 BCTs targeting these barriers and enablers. Report Supplementary Material 13 presents the full list of BCTs and associated TDF domains. These BCTs were delivered through a paper-based ‘toolkit’ to support staff in planning their response to feedback. Tools included a dissemination cascade to facilitate the sharing of feedback reports, a fishbone problem-solving worksheet, goal-setting and action plan templates, and ‘QuickAudit’ local re-monitoring template (see Report Supplementary Material 14). The intervention involved a 1-hour training visit during which a member of the research team met with each hospital transfusion team to talk them through the toolkit (i.e. its purpose and how to use it).

Workstream 1c: feasibility and acceptability

All intended BCTs and enhancements were identified in the prototype feedback reports and toolkit, indicating that these were feasible to embed in intervention materials with fidelity. Twelve staff members participated in semistructured interviews and 14 in think-aloud interviews, across all health-care professional groups (consultant haematologists, geriatricians, gastroenterologists and obstetricians; transfusion practitioners; laboratory managers; nurses; midwives; clinical audit and effectiveness managers).

Six overarching feasibility and acceptability themes were identified for both interventions (see Report Supplementary Material 15 and 16):

1. comprehensibility (i.e. clarity of the content and formatting of the reports and toolkit)
2. preference (i.e. whether or not staff liked the different intervention materials and their preferences for different modes of delivery)
3. usability (i.e. the perceived ease of use and utility of the interventions)
4. engagement (i.e. the extent to which feedback and tools are engaging and capture recipients’ attention)
5. intention (i.e. how likely staff are to read/share the reports and use the toolkit)
6. effectiveness (i.e. the likely impact that the interventions have on practice).

Responses from participants, including health-care professionals, within these six overarching themes were mostly positive (e.g. enhanced content: ‘the recommendations for my hospital are clear, general findings for clinical staff, different groups . . . it lays out quite clearly what we’ve got to do’; enhanced follow-on: ‘as I’m going through I think it’s a really good resource for the teams . . . as a refresher or a reminder or even if they’re not doing it properly in the first place’). We therefore made no refinements to the intervention.

Further improvement suggestions were recorded, including modifiable changes to the formatting and design of the report (e.g. font type, size, colour). Time was identified as a key barrier to feasibly delivering the toolkit in the trial in terms of the availability of hospital transfusion staff to attend the toolkit training session and researchers travelling to sites nationally to deliver this. Participants did not find the paper-based tools sufficiently engaging, interactive or facilitative of team working. Two major refinements were therefore made. First, as proposed by participants, the mode of delivery was changed to web based, which supported greater interactivity and data-sharing among colleagues. Second, the in-person training session was replaced with a telephone support co-intervention, whereby staff were prompted to log in to the toolkit and each tool was explained and demonstrated. A telephone helpline was made available for the duration of the trial. Final intervention materials are available in Report Supplementary Material 9a–c, 10a–b, 15 and 16. A Template for Intervention Description and Replication (TIDieR) intervention specification checklist is available in Report Supplementary Material 17.21

Conclusions

Current blood transfusion A&F in England makes limited use of the available theory and evidence about how to effectively design and deliver A&F. We developed two theory- and evidence-enhanced feedback interventions for evaluation in two national cluster-randomised trials (WS2 and WS3).

Limitations

The main limitation is the sample size of feedback documents, hospitals and health-care professionals analysed/participating in each sub-workstream. Although these sources were purposively sampled to ensure maximum diversity, findings from these may not reflect the full variation in practice across hospitals nationally. Qualitative interviews in WS1b and WS1c were also likely to be subject to social desirability and recall biases.