Workstream 2: trials 1 and 2 and cost-effectiveness analysis

Workstream 2a: trial 1

Study design and setting

We conducted a 2 × 2 factorial, cross-sectional, cluster-randomised controlled trial in elective surgical patients, embedded in the 2015–16 NHS Blood and Transplant (NHSBT) NCAs of patient blood management. The two feedback interventions were directed at clinical teams in hospital trusts and health boards across the UK, so randomisation at the cluster level was essential. Different patients were audited at baseline and follow-up, leading to a cross-sectional design. We adopted a 2 × 2 factorial design in which an enhanced content intervention was compared with the standard content intervention across levels of follow-on, and an enhanced follow-on intervention was compared with the standard follow-on intervention across levels of content. Although we recognised that there might be a small antagonistic interaction between the feedback interventions, interest was in the marginal (i.e. main) effects of those interventions, even in the presence of realistic interactions.

All trusts and health boards in the UK were invited to take part, with NHS permissions sought for eligible sites. We assessed outcomes for audited patients of randomised clusters at 12 months after cluster randomisation. In addition, we requested safety data from the Serious Hazards of Transfusion (SHOT) database and blood usage data from the Blood Stocks Management Scheme (BSMS) database.

Cluster eligibility

We based eligibility on the following:

• inclusion criteria –
  • providing an NHS service relevant to the audit topic
  • accepting invitation by the NCA to participate in the audit.
• exclusion criteria –
  • independent hospitals (as clinicians involved in transfusion decisions are likely to practice in multiple clusters, leading to potential contamination)
  • participating in the development of the interventions (also to prevent contamination).

Where at least one NCA hospital site in a cluster was eligible, we regarded the cluster as eligible.

Randomisation

The trial statistician undertook randomisation at a single point following receipt of the baseline audit database from the NCA. We randomised trusts or health boards on a 1 : 1 : 1 : 1 basis using a computer-generated minimisation program, incorporating a random element, balancing for trust size (large, medium or small) and regional transfusion committee. We randomised clusters to one of four feedback interventions: (1) standard content/standard follow-on, (2) standard content/enhanced follow-on, (3) enhanced content/standard follow-on or (4) enhanced content/enhanced follow-on. If trusts or health boards merged following randomisation, we continued to regard them as distinct clusters for intervention, data collection and analysis purposes.

Interventions

Current practice was the standard feedback delivered by the NCA following completion of an audit. Feedback is in the form of a written clinical audit-specific report to hospital sites, a regional Microsoft PowerPoint^® (Microsoft Corporation, Redmond, WA, USA) presentation and often action plan templates. Responses by clinical teams to receipt of feedback is not standardised. We requested that staff who received the feedback did not share with colleagues external to their own trust or health board.

Enhanced content

Enhanced content comprised feedback documents with content written to specifically deliver behaviourally specified feedback and the relevant theoretically consistent BCTs. These were delivered by the NCA programme via written and graphic feedback presented in multiple feedback documents and presentations.

Enhanced follow-on

Enhanced follow-on made use of targeted dissemination of feedback to relevant staff with discussion and agreement of action plans. Follow-on support comprised practical guidance for clinical teams on how to operationalise the process of responding to feedback, including materials for clinical teams to facilitate discussion and agreement of locally relevant goals and action plans based on feedback.

Monitoring intervention adherence

The assessment of fidelity was based on a fidelity framework proposed by the National Institutes of Health (NIH) Behaviour Change Consortium22 to investigate and report the extent to which the enhanced and standard feedback interventions were designed, trained, delivered, received and enacted as intended.

Data collection

Table 1 summarises the required data and collection time points. Baseline was from October 2014 to September 2015 and follow-up was from November 2015 to October 2016. We obtained data from NHSBT, SHOT and BSMS, supplemented with our own collection of trial process data (i.e. withdrawals and fidelity).

Outcomes

Text in the following section is reproduced with permission from Hartley et al.2 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Transfusion may occur preoperatively, intraoperatively or postoperatively. There may also be multiple transfusion episodes after surgery but prior to discharge. The prespecified primary outcome, measured at the patient level and taken from the NCA follow-up audit, was whether preoperative and/or first postoperative transfusions were categorised as acceptable or any preoperative or first postoperative transfusions were outside guidelines (binary).2 A clinical algorithm (see Appendix 1) was agreed by an independent panel, minimising the risk of detection bias, based on clinical relevance and baseline compliance.

Prespecified secondary outcomes comprised:

• total volume of allogeneic RBCs transfused (units at trust level, from BSMS summed over blood groups and clinical specialties)
• total volume of allogeneic RBCs transfused (units at patient level, from NCA summed over preoperative, intraoperative and postoperative periods)
• total number of incidents reported to SHOT (count at trust level, from SHOT summed over clinical specialties and events, near-misses and ‘right blood right patient’ incidents)
• number of definitely, probably or possibly preventable incidents reported to SHOT within clinical specialties targeted by the audit (count at trust level, from SHOT summed over events, near-misses and ‘right blood right patient’ incidents).

Prespecified supportive outcomes comprised:

• preoperative transfusion (acceptable/outside guidelines)
• postoperative transfusion (acceptable/outside guidelines)
• individual NCA audit standard met (1, 2, 3, 4 and 8; yes/no)
• preoperative volume of allogeneic RBCs transfused (units at patient level)
• postoperative volume of allogeneic RBCs transfused (units at patient level)
• total volume of RBCs issued (units at trust level, from BSMS data)
• total volume of RBCs transfused (units at trust level, from BSMS data).

All other outcomes (i.e. intermediate NCA outcomes, number of relevant near-miss or ‘right blood right patient’ SHOT incidents, total number of unpredictable SHOT incidents, total number of possibly preventable SHOT incidents and BSMS total volume of RBCs wasted) reported were exploratory.

Sample size

There were two comparisons of interest (enhanced vs. standard content and enhanced vs. standard follow-on), relating to the two main effects of the factorial design under effect coding (–1,1 rather than 0,1). Assuming that 20% of transfusions are outside guidelines at follow-up, that the intracluster correlation coefficient will be 0.05 and that cluster sizes will vary from 17 to 68 with a mean of 45, we required 152 clusters to detect a minimally important reduction of 6% (to 14%) in the presence of, at most, a small antagonistic statistical interaction (i.e. main effects of 5%) with 80% power using logistic regression models, with a random-intercept for cluster, and a two-sided 2.5% significance level, for each comparison.

Statistical analysis

No interim analyses were planned or conducted. Wherever possible, we undertook primary data summaries and analyses on the intention-to-treat (ITT) sample, defined as all randomised clusters analysed as randomised. The proportion of missing data was anticipated to be non-trivial. Therefore, mechanisms for missing data on key variables were explored and multiple imputation was used based on 100 imputations and the full imputation model, under the assumption that data were missing at random (MAR). Sensitivity analyses assessed whether or not the conclusions were robust across approaches to handling missing data. A random intercept model accounted for clustering arising from cluster randomisation, as model convergence was unreliable with more complex structures. Reflecting interest in each main effect, an overall two-sided 5% significance level was used; 97.5% CIs are also presented for the main effects, according to a Bonferroni multiplicity adjustment.

We compared the primary outcome using multilevel logistic regression, adjusting for design factors (trust size, regional transfusion committee) and trust-level proportion of acceptable transfusions at baseline, with effect-coded contrasts for enhanced versus standard content, enhanced versus standard follow-on and their interaction. The patient-level secondary end point of volume of blood transfused is reported descriptively, as are the trust-level secondary outcomes (volume transfused, SHOT incidents). The supportive outcomes act as sensitivity analyses for primary and secondary outcome analyses. We conducted analyses in the statistical package software SAS^® version 9.4 (SAS Institute Inc., Cary, NC, USA. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ^® indicates USA registration).

Results

Screening and recruitment

Prior to cluster randomisation (15 October 2015), we screened 189 NHS trusts and health boards. A total of 152 (80.4%) were included in the audit, and 135 of these (88.8%) were randomised. Reasons for exclusion, when given, were mainly mergers or ineligibility, so the included clusters were typical of the UK as a whole. The baseline audit comprised a total of 2714 patients, 40% of target; 66 clusters were allocated to standard content, 69 were allocated to enhanced content, 67 were allocated to standard follow-on, and 68 were allocated enhanced follow-on. An average cluster size of 20 patients was observed, with a coefficient of variation of 0.7 (Figure 3).

FIGURE 3 Trial 1: NHS trust/health board and patient CONSORT flow diagram. (Figure not reproduced here; see the original article.)

Over half of the clusters received the enhanced (62%) and standard reports (56%). Receipt of the toolkit was lower (31% overall), with more clusters allocated to standard content receiving the toolkit (36%) than those allocated to enhanced content (26%). A total of 23 out of 135 clusters (17%) were lost to follow-up, this proportion being higher among those allocated to enhanced content (22% vs. 12%) and enhanced follow-on (21% vs. 13%). Overall, two clusters had no cases to audit, eight clusters provided audit cases but did so too late, and 13 declined to take part. Among the 112 clusters taking part in the follow-up audit, there were a total of 2222 audit cases (32% of target). Again, an average cluster size of 20 patients and a coefficient of variation of 0.7 were observed. We included all 112 clusters and 2222 patients in our primary analyses. Table 2 shows baseline patient-level characteristics. (Other baseline summaries are given in Appendix 2, Table 10.) These were generally well balanced, as would be expected given that patients were ascertained prior to randomisation. Notably, 30% surgical procedures were for a fractured neck of femur and, as anticipated, these patients would not typically be able to attend a preoperative clinic. In total, 249 (9%) received a preoperative, 363 (13%) an intraoperative and 2560 (94%) at least one postoperative transfusion.

The health-care professional making the decision to transfuse was usually the consultant or other doctor, with this being more commonly a junior doctor postoperatively.

Appendix 2, Table 11, shows that patient-level characteristics in the follow-up audit were generally well balanced. Recruitment bias following cluster randomisation was therefore minimal, with imbalances likely to result from missing clusters. The characteristics of patients at baseline and at follow-up are similar.

Outcomes at baseline

Patient-level outcomes at baseline are provided by content and follow-on in Appendix 2, Table 12. The proportion of missing data for the primary outcome was 7%, well balanced by follow-on but less well balanced by content. Around 80% of patients had a pre- or postoperative transfusion outside guidelines: 82% of patients received a preoperative and 78% a postoperative transfusion outside guidelines. On average, 2.2 units of blood were transfused per patient by content and follow-on.

A large number of patients were excluded from the audit standards: 31% from standard 1, 91% from standard 2, 95% from standard 3, 91% from standard 4 and 6% from standard 8. Feedback given to clusters was therefore limited to specific patient groups, limiting the ability of clusters to make changes at scale. A significant group of patients could not be classified for standard 1 (28%) and standard 8 (27%). Approximately 26% of patients had a transfusion classified as not meeting standard 1, 7% had one classified as not meeting standard 2, 2% had one classified as not meeting standard 3, 6% had one classified as not meeting standard 4 and 64% had one classified as not meeting standard 8. Thus, the greatest room for improvement was for standard 8, and missing data were substantial. As expected, patient outcomes at baseline were well balanced across content and follow-on.

Outcomes at follow-up

Patient-level outcomes at follow-up are provided by enhanced content and follow-on groups in Table 3 (other outcomes are summarised in Appendix 2, Table 13). The proportion of missing data for the primary outcome was 11%, balanced across randomised groups. Approximately 73% of patients had a pre- or postoperative transfusion outside guidelines; again, the proportions were similar between randomised groups. Overall, 62% of patients receiving a preoperative transfusion and 72% receiving a postoperative transfusion received one outside guidelines.

The proportions were similar across randomised groups in postoperative transfusions, and these contributed to a greater extent to the primary outcome than preoperative transfusions. On average, 2.1 units of blood were transfused per patient, similar across randomised groups. Similar numbers of patients were excluded from audit standards at follow-up. Again, a significant group of patients could not be classified for standard 1 (22%) or standard 8 (33%). Approximately 27% of patients had a transfusion classified as not meeting standard 1, 6% had one classified as not meeting standard 2, 2% had one classified as not meeting standard 3, 5% had one classified as not meeting standard 4 and 57% had one classified as not meeting standard 8. The greatest difference observed between randomised groups was for standard 8, but the number of missing data was substantial.

Primary outcomes

Table 4 shows the primary outcome results (with the main sensitivity analysis in Appendix 2, Table 14). Across 100 imputations, the unadjusted proportion of acceptable transfusions was 18% for those allocated to standard content and to enhanced content, and the adjusted risk difference was –1% (95% CI –7% to 4%). There was no evidence of a clinically or statistically significant effect. The unadjusted proportion of acceptable transfusions was 18% for those allocated to standard or enhanced follow-on, and the adjusted risk difference was 1% (95% CI –5% to 6%), again providing no evidence of a statistically significant effect. For the follow-on intervention, a clinically important effect size of 5% (in the presence of an interaction) was not ruled out. The interaction between content and follow-on is given for information. The conclusions were unchanged regardless of the method adopted for handling the missing data. The estimates were also similar, indicating that this result is robust.

Secondary outcomes

Appendix 2, Table 17, describes the cluster-level volume of RBCs transfused (BSMS) across baseline and follow-up (and in 3-month periods). Overall, seven and six clusters, respectively, were lost to follow-up. Data were skewed; interquartile ranges (IQRs) are similar across randomised groups at baseline and follow-up. There is an indication of a steady overall reduction in blood use over this period. Appendix 2, Tables 12 and 13 summarises the patient-level volume of RBCs transfused (NCA), in each case separately for preoperative and postoperative transfusions. Imbalances are observed at baseline by randomised group, expected to be related to imbalances in missing clusters.

Appendix 2, Table 16, describes the cluster-level total number of SHOT incidents and number of relevant errors. There were no missing SHOT data, as all 135 clusters provided data, summarised across baseline and follow-up (and in 3-month intervals). Interpretation is limited because of sparse data relating to the number of relevant errors. The median number of relevant events across the baseline and follow-up periods was zero across the randomised groups. IQRs were the same across baseline and follow-up. Therefore, there is no evidence that the trial interventions increased or decreased secondary outcomes reported to BSMS, NCA or SHOT.

Appendix 2, Table 15, reports supportive outcomes.

Workstream 2a: trial 2

Methods

Study design and setting

The trial 2 methods were similar to those in trial 1. We conducted a second 2 × 2 factorial, cross-sectional, cluster-randomised controlled trial, embedded in 2016/17 NCABT, of RBC and platelet transfusions in haematology patients.

Cluster eligibility

Cluster eligibility was assessed separately for trial 2 using the same criteria as for trial 1. Where possible, cluster definitions remained the same across trials; however, in some cases, different hospital sites within clusters signed up to the audits.

Randomisation

As previously, the trial statistician undertook re-randomisation at a single point following receipt of the NCA baseline audit database. We independently randomised trusts or health boards on a 1 : 1 : 1 : 1 basis, as before balancing for trust size and regional transfusion committee, but also for the previous treatment allocation where clusters had been entered into both trials.

Interventions

The interventions remained the same across the trials, although the content of the feedback reports was tailored to the audit topic.

Monitoring intervention adherence

As before, the assessment of fidelity was based on a fidelity framework proposed by the NIH Behaviour Change Consortium.22

Data collection

The required data and collection time points for trial 2 mirrored those in Table 1, but baseline was from July 2015 to June 2016 and follow-up was from July 2016 to June 2017.

Outcomes

Transfusion may occur for RBCs or platelets; all patients will have had one or more transfusions. The primary outcome, measured at the patient level and taken from the NCA follow-up audit, was to compare how many RBC and/or platelet transfusions were acceptable with how many transfusions were carried out outside guidelines. The clinical algorithm (see Appendix 1) was agreed by an independent panel based on clinical relevance and baseline compliance. No clinical judgement was required at a patient level to categorise transfusions.

Secondary outcomes comprised:

• total volume of RBCs transfused (units at trust level, from BSMS summed over blood groups and clinical specialties)
• total volume of RBCs transfused (units at patient level, from NCA)
• total volume of platelets transfused (units at patient level, from NCA)
• total number of incidents reported to SHOT (count at trust level, from SHOT summed over clinical specialties and events, near-misses and ‘right blood right patient’ incidents)
• number of definitely, probably or possibly preventable incidents reported to SHOT within clinical specialties targeted by the audit (count at trust level, from SHOT summed over events, near-misses and ‘right blood right patient’ incidents).

Supportive outcomes comprised:

• RBC transfusion (acceptable/outside guidelines)
• platelet transfusion (acceptable/outside guidelines)
• individual NCA audit standard met (1, 2, 3, 6 and 7; yes/no)
• total volume of RBCs issued (units at trust level, from BSMS data)
• total volume of RBCs transfused (units at trust level, from BSMS data).

As in trial 1, all other outcomes (i.e. intermediate NCA outcomes, number of relevant near-miss or ‘right blood right patient’ SHOT incidents, total number of unpredictable SHOT incidents, total number of possibly preventable SHOT incidents, BSMS total volume of RBCs wasted) reported are exploratory.

Sample size

As previously, we required 17–68 patients (mean 45) from 152 clusters in order to detect a minimally important reduction of 6% (to 14%) in the presence of, at most, a small antagonistic statistical interaction (i.e. main effects of 5%) with 80% power using logistic regression models, with a random-intercept for cluster, and a two-sided 2.5% significance level, for each comparison.

Statistical analysis

No interim analyses were planned or conducted. A similar analysis strategy was adopted for trial 2, except that, when adjusting for design factors, previous allocation was added.

Results

Screening and recruitment

Prior to cluster randomisation (6 July 2016), we screened 187 NHS trusts and health boards, covering the whole UK. A total of 141 (75.4%) were included in the audit, and 135 of those (95.7%) were randomised (although one was randomised in error, so 134 were entered). Reasons for exclusion were mainly ineligibility or declining to take part in the audit, so included clusters were typical of the audit as a whole. The baseline audit comprised a total of 4372 patients, 64% of target; 68 clusters were allocated to standard content, 66 were allocated to enhanced content, 67 were allocated to standard follow-on and 67 were allocated to enhanced follow-on. An average cluster size of 33 patients was observed, with a coefficient of variation of 0.5 (Figure 4).

FIGURE 4 Trial 2: NHS trust/health board and patient CONSORT flow diagram. (Figure not reproduced here; see the original article.)

Overall, half of the clusters received the enhanced (50%) and standard reports (47%). Receipt of the toolkit was lower (30% overall), with the proportion of clusters receiving the toolkit lower among those allocated to standard content (24%) than among those allocated to enhanced content (36%). In total, 12 out of 135 clusters (9%) were lost to follow-up; similar proportions had been allocated to standard and enhanced content (10% vs. 8%) but a higher proportion had been allocated to standard follow-on than to enhanced follow-on (12% vs 6%). Overall, four clusters had no cases to audit and eight clusters declined to take part in the follow-up audit. Among the 123 clusters taking part in the follow-up audit, there were a total of 3886 audit cases. We included all 123 clusters and 3859 patients (56% of target) in our primary analyses; 27 patients were excluded because they had received a platelet transfusion only for therapeutic reasons, making them ineligible. An average cluster size of 32 patients and a coefficient of variation of 0.5 were observed at analysis.

Appendix 2, Table 18 shows the baseline patient-level characteristics. (Other baseline summaries are given in Appendix 2, Table 19.) These are generally well balanced, given that they were ascertained prior to randomisation. Overall, 1387 (31%) patients received a RBC and platelet transfusion, 2781 (63%) patients received only a RBC transfusion and 271 (6%) patients received only a platelet transfusion. For this reason, the majority categorised were RBC transfusions for medical or chronic anaemia.

Patient-level characteristics in the follow-up audit are given in Appendix 2, Table 20. These are as balanced as they were at baseline, providing no indication of recruitment bias after cluster randomisation. Characteristics of patients at baseline and follow-up are also comparable, except for the overall proportion of patients receiving a RBC transfusion undergoing a chronic transfusion programme (33% vs. 25% at baseline).

Outcomes at baseline

Patient-level outcomes at baseline are given by content and follow-on group in Appendix 2, Table 21, together with detailed definitions. The proportion of missing data for the primary outcome was 12%, balanced by content and follow-on. Around 26% of patients received a RBC or platelet transfusion outside guidelines. Overall, 11% of patients receiving a RBC transfusion and 45% receiving a platelet transfusion received one outside guidelines. On average, 2.0 units of RBCs and 1.0 units of platelets were transfused per patient across content and follow-on groups.

As in trial 1, a large number of patients were excluded from the audit standards, 6% for standard 1, 75% for standard 2, 99% for standard 3, 90% for standard 6 and 71% for standard 7, owing to the specific subset of patients considered in each standard. As such, the feedback given to clusters was focused on specific patient groups, thereby limiting the ability of clusters to make changes that would apply to all patients. Few patients could not be classified (0% for standards 1, 3 and 7; 2% for standard 2; and 7% for standard 6). Of those eligible, 6% of patients had a transfusion classified as not meeting standard 1, 73% as not meeting standard 2, 47% as not meeting standard 3, 42% as not meeting standard 6 and 38% as not meeting standard 7. Thus, the greatest improvement was required in standards 2, 3, 6 and 7, which focused on specific patient groups. As expected, patient outcomes at baseline were well balanced across content and follow-on groups.

Outcomes at follow-up

Patient-level outcomes at follow-up are given by content and follow-on in Table 5 (other outcome summaries are given in Appendix 2). The proportion of missing data for the primary outcome was 9%, balanced by randomised groups. About 25% of patients received a RBC or platelet transfusion outside guidelines, and the proportions were similar between randomised groups. Overall, 11% of patients receiving a RBC transfusion and 45% receiving a platelet transfusion received one outside guidelines. Again, proportions were similar between randomised groups. On average, 2.0 units of RBCs and 1.0 units of platelets were transfused per patient, the same across randomised groups.

Similar numbers of patients were excluded from each audit standard at follow-up. Again, only a few patients could not be classified, with the exception of standard 3, for which 12% could not be classified. Of those eligible, 6% of patients had a transfusion classified as not meeting standard 1, 72% as not meeting standard 2, 21% as not meeting standard 3, 42% as not meeting standard 6 and 36% as not meeting standard 7. Differences between enhanced and standard content were indicated for standards 2, 3, 6 and 7, favouring enhanced content for standards 3 and 7 and favouring standard content for standards 2 and 6. Differences between enhanced and standard follow-on were minimal across all the standards.

Primary outcomes

Table 6 shows primary outcome results (sensitivity analyses are given in Appendix 2, Table 23). Across 100 imputations, the unadjusted proportion of acceptable transfusions was 74% for those allocated to standard content and 71% for those allocated to enhanced content; the adjusted risk difference was –4% (95% CI –9% to 2%). There was no evidence of a clinically or statistically significant effect. The unadjusted proportion of acceptable transfusions was 74% for those allocated to standard follow-on and 72% for those allocated to enhanced follow-on; the adjusted risk difference was –1% (95% CI –6% to 5%), again indicating no evidence of a statistically or clinically important effect. The interaction between content and follow-on is again given for information. The conclusions were unchanged regardless of the method adopted for handling the missing data. The estimates are also similar, indicating that this result is robust.

Secondary outcomes

Cluster-level volume of RBCs transfused (BSMS) is summarised descriptively across baseline and follow-up (and in 3-month periods) in Appendix 2, Table 26. Overall, three clusters were lost to follow-up. Data were again skewed; IQRs are again similar across randomised groups at baseline and follow-up. Patient-level volume of blood components transfused (NCA) is summarised descriptively in Appendix 2, Tables 21 and 22, in each case separately for RBC and platelet transfusions. The conclusions are consistent. Cluster-level total number of SHOT incidents and number of relevant errors are summarised descriptively in Appendix 2, Table 25. There were no missing SHOT data, as all 134 clusters provided data, summarised across baseline and follow-up (and in 3-month intervals). Again, sparse data relating to the number of relevant errors limit the interpretation of these. The median number of relevant events across randomised groups was similar at baseline and follow-up; IQRs were also similar. As in trial 1, there is no evidence that the trial interventions changed secondary outcomes reported to BSMS, NCA or SHOT.

Appendix 2, Table 24 reports supportive outcomes.

Limitations

Four main factors affected the interpretation of the cluster-randomised trials. First, the number of clusters participating in each audit and in the trials, as well as the number of patient records audited in each cluster, was smaller than projected. This compromised statistical power, increasing uncertainty around our estimates of intervention effects. However, the results appeared sufficiently consistent across the trials, the two interventions, and the range of primary and secondary outcomes for us to conclude that it is highly unlikely that the enhanced interventions had any important effects. Second, randomised clusters lost to follow-up were not included in analyses. Further sensitivity analyses are required to explore the impact of this. Third, the audit data contributing to trial outcome measures were more complex and required more review and cleaning than anticipated. Fourth, the audit standards that required most improvement were relevant to only a subset of patients included in the audits. This limited the ability of the trial to detect change in practice at scale, as it diluted the underlying effects of the interventions.

Workstream 2b: cost-effectiveness analysis

Aim

The economic modelling evaluated the costs and benefits of the two AFFINITIE A&F interventions in two trials, and aimed to assess the interventions’ cost-effectiveness.

Methods

Design overview and model

The analysis was conducted using decision-analytic modelling from the perspective of the NHS. For each of trials 1 and 2 we compared the costs and the outcomes of the ‘enhanced content’ with ‘usual content’ arms, and the ‘enhanced follow-on support’ with ‘usual follow-on support’ arms (Figure 5). We explored uncertainty around the parameters used in the model using sensitivity analysis.

FIGURE 5 Cost-effectiveness analysis model for trials 1 and 2. (Figure not reproduced here; see the original article.)

The primary outcome of the trials was the proportion of transfusions given that were acceptable. For ease of understanding the incremental cost-effectiveness ratios (ICERs) presented in this analysis, we multiplied those proportions by 100 so that they were percentages. Therefore, the primary outcome of this analysis was the percentage of acceptable transfusions. Secondary outcomes comprised the volume of blood transfused and the number of SHOT-reportable incidents. The time horizon for this analysis was 1 year, during which two rounds of audits took place for each trial. We applied no discounting.

Unit of analysis

The unit of analysis for this evaluation was per site, reflecting the fact that the intervention was delivered at secondary care sites, and that many of the cost data available were for local, site-level costs. This represented a change from the original protocol, in which the unit of analysis was ‘per patient.’

Analysis overview and changes to protocol

The analysis presented follows the original proposal, with the following amendments:

• The unit of analysis was originally ‘per patient’ but was changed to ‘per site’, as the costs were collected at trust and whole intervention level, and the interventions were delivered at the site level.
• The number of transfusion-related adverse events used in the economic analysis was taken from the results of the statistical analysis and was not modelled in the long-term, reflecting the rest of the analyses’ time horizon.
• The analysis was amended to undertake four two-arm comparisons, rather than two four-arm comparisons, to be appropriate to the overall study design.
• The protocol intended quality-adjusted life-years (QALYs) to be estimated from SHOT events. However, the number of these events was small and there was no significant difference in SHOT events between the comparison groups, meaning that a cost–utility analysis would not yield meaningful results.

Costing exercise methodology

The costs of the intervention were collected from a combination of top-down (gross-costing) and bottom-up (microcosting) methods, depending on the type and the quality of data available. An extensive costing exercise was undertaken to estimate the resources required to deliver the A&F interventions and the standard practice audits. The resource components included in this exercise were collecting the audit data, developing and delivering the audit reports, delivering the follow-on-support programmes, additional NHS activity in response to receiving the audit reports and the volume of blood transfused. The costs associated with the volume of blood transfused were far higher than the other costs combined. There was a large degree of uncertainty around these cost estimates, particularly around the amount of time NHS staff spent collecting and inputting audit data and using the online toolkit, and the volume of blood transfused per site. This uncertainty is explored in the sensitivity analyses. The costing exercise methodology and results are detailed in Report Supplementary Material 18.

Unit costs

The cost used for purchasing a unit of blood components was £128.99 and £185.86 for RBCs and platelets, respectively.23 We applied a cost of £51.32 for transfusing 1 unit of blood components. This was obtained using the value given for subsequent units of blood transfused in the costing statement of the National Institute for Health and Care Excellence (NICE) guidelines for blood transfusion and management,24 and inflating the difference for 2017/18.

Therefore, the unit cost of one unit of RBCs and platelets was £175.78 and £229.51, respectively. Unit costs of staff time were drawn from the NHS pay scale 2017/18,25 the British Medical Association pay scales for junior doctors 2017/18,26 and the British Medical Association pay scales for consultants 2017/18.27 Non-clinical staff costs were drawn from the health-care or research organisation employing the relevant staff member (i.e. NHSBT; University of Leeds; or City, University of London).

Outcome parameters

Outcome data were obtained from the outputs of the trial analyses (see Tables 4 and 6). Where necessary, these outcomes were converted from mean-per-cluster to mean-per-site to conform to the unit of analysis for this study. Table 7 presents the parameters used in the analysis for the primary and secondary outcomes for trial 1, and Appendix 3, Table 28, describes those for trial 2.

Cost-effectiveness analysis methodology

We calculated ICERs to measure the cost-effectiveness for each comparative analysis for trials 1 and 2. The ICER is interpreted as the additional cost required to produce one unit of benefit. For the primary outcome, benefit is defined as one additional percentage of blood transfused acceptably. For SHOT it is one fewer SHOT-reportable event. We treated the volume of blood used as a resource consumed and not as a health consequence, and thus included it only on the cost side of the ICER calculations.

The ICER calculation shown in Appendix 3, Tables 27 and 28, demonstrates how the different cost components were included in the calculation of the ICER for the primary outcome.

Sensitivity analysis methodology

To explore the uncertainty around the parameters used in our model, we conducted probabilistic sensitivity analysis (PSA) and deterministic sensitivity analysis (DSA). Each parameter in the DSA was varied by 20% above and below the mean value. For the PSA all cost parameters were characterised using a gamma distribution, as was the volume of blood transfused and the number of SHOT events. The proportion of acceptable transfusions was characterised using a beta distribution.

We ran 1000 iterations for each PSA comparison and presented the results in terms of mean incremental difference and associated uncertainty intervals (UIs) on a cost-effectiveness plane. A cost-effectiveness acceptability curve (CEAC) was plotted for each analysis. We found that PSA iterations fell into all four quadrants of the cost-effectiveness plane. To calculate the ICER relative to willingness-to-pay (WTP) thresholds for the CEAC, we took the following approach:

• All iterations in the north-west quadrant were never considered to be cost-effective.
• All iterations in the south-east quadrant were always considered to be cost-effective.
• Iterations in the north-east quadrant had to fall below the WTP threshold to be considered cost-effective.
• Iterations in the south-west quadrant had to be above the WTP threshold to be considered cost-effective.

Budget impact and cost-neutral analyses

We calculated the effect of each intervention on all 185 participating sites in trial 1 and 194 sites in trial 2 to estimate the effect on the NHS budget at a national level. We then adopted a cost-neutral framework by asking ‘How many units of blood would need to be prevented from being transfused in order for the interventions to be cost neutral (i.e. to break even)?’. For this analysis we evaluated the cost of each intervention in terms of the equivalent cost in units of RBCs. We did this by calculating the incremental costs of each pair of interventions (enhanced minus standard) for each trial, and dividing the result by the cost of a unit of RBCs. We used sensitivity analysis to explore the impact that the intervention costs and unit costs of RBCs had on the results.

Results

Intervention costs: budget impact analysis

Table 8 presents the cost of each arm for both trials. The highest costs across all arms were those of the blood transfused. The lowest costs were those of the feedback interventions.

Excluding the cost of blood transfusions, in trial 1 the incremental cost of the enhanced compared with the standard content intervention was £219 per site, and the incremental cost of the enhanced compared with the standard follow-on support intervention was £18 per site. For trial 2 these figures were £248 and –£198, respectively. Owing to the cost of additional NHS activity, enhanced follow-on support cost less per site than standard support.

Cost-effectiveness analysis results: primary outcome

Trial 1

The cost per site of standard content and enhanced feedback content was £492,619 and £628,189, respectively. Enhanced feedback content cost £135,570 more per site than standard content, and was associated with a 0.8% decrease in acceptable transfusions. Therefore, the base-case ICER was –£169,462 and the enhanced intervention was dominated by the standard intervention.

The cost per site of standard follow-on support and enhanced follow-on support was £506,982 and £561,826, respectively. Enhanced follow-on support cost £54,826 per site more than standard follow-on support, and was associated with a 0.1% decrease in acceptable transfusions. Therefore, the base-case ICER was –£548,261 and the enhanced intervention was dominated by the standard intervention.

Trial 2

The cost per site of the standard content and enhanced feedback content was £675,173 and £689,873, respectively. Enhanced content cost £14,700 more per site than standard content, and was associated with a 3.0% decrease in acceptable transfusions. Therefore, the base-case ICER was –£4900 and the enhanced intervention was dominated by the standard intervention.

The cost per site of standard follow-on support and enhanced follow-on support was £582,805 and £617,084, respectively. Enhanced follow-on support cost £34,278 per site more than standard follow-on support, and was associated with a 1.8% decrease in acceptable transfusions. Therefore, the base-case ICER was –£19,044 and the enhanced intervention was dominated by the standard intervention.

Hence, in every case, the enhanced intervention was dominated by the standard intervention. This was also the case if we included only the cost of the interventions and related activity (i.e. not the cost of blood transfusions).

Cost-effectiveness analysis results: secondary outcome (number of SHOT events)

Trial 1

Enhanced feedback content saw a reduction in SHOT events of 0.2 per site compared with standard content, producing a base-case ICER of £735,927 per one-unit reduction in SHOT events. Enhanced follow-on support saw an increase in SHOT events of 1.3 per site compared with standard follow-on support, producing a base-case ICER of –£43,455 and resulting in the enhanced intervention being dominated by the standard intervention.

Trial 2

The results for trial 2 can be found in Report Supplementary Material 19.

Probabilistic sensitivity analysis: primary outcome

For each of the following analyses there is considerable uncertainty in the estimates produced, indicated by the wide uncertainty intervals, which include zero.

Trial 1: percentage of transfusions acceptable (enhanced versus standard content)

The PSA results suggest that the mean incremental cost of the enhanced versus standard intervention was £129,449 per site (95% UI –£1,092,049 to £1,350,946 per site). The mean incremental change in percentage of acceptable transfusions was –0.81% (95% UI –3.1% to 1.4%). This indicates that the enhanced intervention was more costly and less effective than the standard intervention.

Figure 6 shows the simulated outputs shown on a cost-effectiveness plane (CEP).

FIGURE 6 Simulated outputs on the cost-effectiveness plane for trial 1 enhanced vs. standard content for the primary outcome. Each blue diamond represents one individual simulated output of the model. (Figure not reproduced here; see the original article.)

Figure 7 shows the probability of the enhanced intervention being cost-effective at increasing WTP thresholds. The slight downwards-trending CEAC is due to more iterations falling in the south-west quadrant than in the northeast quadrant (32% vs. 14%, respectively), and as the WTP threshold increases more iterations in the south-west are excluded from being cost-effective than iterations in the north-east are included as being cost-effective.

FIGURE 7 The CEAC for trial 1 enhanced vs. standard content for the primary outcome. The navy line represents the probability that the intervention is cost-effective at ascending WTP thresholds. (Figure not reproduced here; see the original article.)

Trial 1: percentage of transfusions acceptable (enhanced versus standard follow-on support)

The PSA results suggest that the mean incremental cost of the enhanced versus standard intervention was £64,319 per site (95% UI –£1,003,034 to £1,131,673 per site). The mean incremental change in percentage of acceptable transfusions was –0.11% (95% UI –2.38% to 2.16%). Therefore, the enhanced intervention was more costly and less effective than the standard intervention.

Figure 8 shows the simulated outputs on a cost-effectiveness plane (CEP).

FIGURE 8 A CEP for enhanced vs. standard follow-on support for percentage of acceptable transfusions. (Figure not reproduced here; see the original article.)

Figure 9 shows the CEAC holding fairly constant at 45% probable cost-effectiveness. This is because the number of iterations in the south-west quadrant that are found to not be cost-effective is roughly similar to the number in the north-east quadrant that are found to be cost-effective and this remains the case as the WTP threshold increases. See Appendix 3 for further results tables on trial 2.

FIGURE 9 The CEAC for enhanced vs. standard follow-on support for percentage of acceptable transfusions. (Figure not reproduced here; see the original article.)

Deterministic sensitivity analysis results

Varying all of the intervention components had a negligible effect on the model, with the exception of the cost of blood transfused. Varying the cost per site of RBCs and platelets had a large effect on the model, as did varying the percentage of appropriate transfusions and the number of SHOT events. The detailed results of the DSA are presented in Report Supplementary Material 19.

Cost-neutral analysis

For trial 1, the mean incremental cost of the enhanced content intervention compared with the standard content intervention was £219 per site and of enhanced versus standard follow-on support was £18 per site. Hence, the interventions would need to reduce the volume of unacceptable blood transfusions by 1.2 and 0.1 units of RBCs per site, respectively, to be cost neutral.

In trial 2, the mean incremental cost of the enhanced content intervention compared with the standard content intervention was £248 per site and the mean incremental cost of enhanced compared with standard follow-on support was –£198 per site. Hence, the interventions would need to reduce the volume of unacceptable blood transfusions by 1.4 and 0.0 units of RBCs per site, respectively, to be cost neutral. The incremental costs and equivalent units of RBCs are presented in Table 9 (comparable results for trial 2 are shown in Appendix 3, Table 29).

Limitations

The economic modelling was limited by missing data around costs and also did not include costs of adverse events reported to the national haemovigilance system (SHOT). We also used site rather than patient as the unit of analysis to approximate costs and benefits at NHS trust or health board level; we judged that this would best inform national and local decision-making on the relative value of the interventions. The analysis time horizon was very short. No significant difference was found in the number of SHOT events, and the number of events was small, precluding a meaningful cost–utility analysis. Had the intervention been successful in this regard, a cost-per-QALY analysis would have been a valuable method of estimating the effect of serious harm caused by unnecessary blood transfusions on patient health outcomes. Data on additional activity that were collected by means questionnaires sent to NHS staff suffered from being poor quality and a large number of data were missing. The cost of additional activity was much higher than the costs associated with auditing and feeding back; however, the missing cost data had little impact on the model overall, as the cost of the volume of blood transfused dominated all other costs.