Synopsis

Background and rationale

Material throughout this section and the abstract has been adapted from the study protocol by Walker et al.1 and Llewelyn et al.2 These are Open Access articles distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Antimicrobial resistance (AMR) is a global public health risk comparable to malaria and HIV, with an estimated 4.95 million deaths in 2019.3 AMR places increased demands on healthcare systems, leading to mounting healthcare costs, treatment failure and deaths.4–6 Human antibiotic consumption is a major driver of AMR,7 with greater use driving resistance at both a population level and an individual patient level.8 It is increasing year-on-year, particularly in common gram-negative bacteria.4,9 The only way to reduce the selective pressure on bacteria to develop resistance is to reduce antibiotic use.10

Antimicrobial stewardship (AMS) aims to minimise resistance selection by ensuring antibiotics are only prescribed when clinically indicated and that narrow-spectrum agents (active against limited species and strains and thus less likely to select for AMR) are used whenever appropriate.11 Efforts have been made to improve AMS and drive down antimicrobial use in the NHS since at least 2002 when the UK government published its strategy for infectious diseases ‘Getting Ahead of the Curve’.12

In primary care, where antibiotic prescribing decisions are typically in patients earlier on in their illness course, AMS strategies which restrict antibiotic use, such as avoiding or delaying antibiotics in respiratory tract infection, can be safe and effective.13 These approaches are supported by well-developed implementation tools, specifically within Treat Antibiotics Responsibly, Guidance, Education and Tools (TARGET) resources.14

While hospital prescribing accounts for only ~15% of antibiotics consumed in England, ~55% of hospital prescriptions are for ‘broad-spectrum’ agents, compared with ~4% of antibiotic prescriptions in primary care. Thus hospital prescribing accounts for ~65% of broad-spectrum antibiotic use: these antibiotics have the greatest potential to drive future resistance. Hospital antibiotic prescribing decisions can be more challenging as prescribers often have to consider the potential harms of withholding antibiotics in situations where there is diagnostic uncertainty given the clear evidence that prompt initiation of effective antibiotic treatment in severe bacterial infections saves lives.15 Hospital AMS interventions which focus on improving prescribing are therefore more acceptable than the restrictive strategies used in primary care, and can reduce overuse and length-of-stay without compromising mortality.16

The need to ensure patients with serious bacterial infections are treated promptly with antibiotics, before a diagnosis can be confirmed, means that an ongoing ‘review and revise’ approach, incorporating diagnostic information and ongoing clinical assessment, is needed to safely minimise antibiotic overuse. In England, the Department of Health and Social Care’s guidance ‘Start Smart, then Focus’ sets out how this process should be undertaken with ‘review and revise’ of antibiotic prescriptions every 48–72 hours.17,18 In the USA, the analogous term ‘antibiotic timeouts’ is used, but Centers for Disease Control and Prevention guidance revised in 201919 prioritised pharmacist-led audit and feedback to prescribers, as evidence is lacking that prescriber-led reviews reduce overall consumption.20 Of 95 hospital AMS interventions included in a Cochrane review in 2013,21 only 8 considered decisions after the initial antibiotic prescription. Two large studies targeted duration of surgical antibiotic prophylaxis, while the remaining six were small (5 recruiting < 300 participants), were mostly limited to specialised populations (paediatrics, intensive care, ventilator/community-acquired pneumonia) and used varying interventions including introducing guidelines/policies and education/feedback. Interventions to reduce antibiotic duration were not evaluated as a specific category, but relevant studies generally found reductions in antibiotic consumption (not always statistically significant). One study (n = 462)22 found an intervention incorporating face-to-face pharmacist feedback significantly reduced antibiotic durations for lower respiratory tract infections in intensive care. However, an earlier study (not in the review) found no impact of prescription review by telephone,23 illustrating the importance of specific intervention components.

Following the introduction of ‘Start Smart then Focus’ in 2012, in 2013 we conducted a national survey of antibiotic practice at NHS hospitals. This found that hospital antibiotic policies were recommending broad-spectrum antibiotics for empirical therapy (‘Start Smart’) but that this could have the unintended potential to increase the use of broad-spectrum antibiotics and risk of Clostridioides difficile infection (CDI) without adequate ‘review and revise’.24

In 2016, the NHS in England introduced a Commissioning for Quality and Innovation for antimicrobial prescribing which incentivised acute hospital trusts to reduce antimicrobial overuse through increased compliance with ‘review and revise’. Surveying AMS specialists in NHS hospitals about perceptions of ‘review and revise’ and its ability to successfully control antibiotic overuse identified widespread concern about lack of practical tools to implement ‘review and revise’ and lack of confidence in its effectiveness and safety.25

Perhaps unsurprisingly given this, antibiotic use in NHS hospitals continued to rise in the decade following the guidance’s publication. When the Antibiotic Review Kit (ARK)-Hospital programme began (2016), hospital antibiotic dispensing (corrected for clinical activity) had increased by 7% since 2013,26 among the highest increases in Europe,27 and in stark contrast with primary care where implementing practical approaches within TARGET were associated with year-on-year reductions in prescribing.26

Wide-ranging educational and practice resources exist to support hospital antibiotic stewardship28 but knowledge, system and behavioural factors combine to make it difficult for doctors to take active decisions to stop antibiotics.29 The challenge is developing feasible, sustainable and generalisable interventions to change decision-making behaviour at the ‘review and revise’ decision, given the complexity of hospital antibiotic prescribing. Doctors need to balance the risks of under-treating infection in their patient against the risks of antibiotic-resistant infections in the future, making practical choices about specific antibiotic drug, dose and duration given clinical indication, within the cultural context of multi-disciplinary secondary care.

The research programme

The overarching aim of the ARK-Hospital programme was to reduce the incidence of serious infections caused by antibiotic-resistant bacteria in the future, by substantially and safely reducing antibiotic use in hospitals now through effective implementation of ‘review and revise’.

The research was conducted between 1 April 2016 and 31 August 2023. It comprised six work packages (WPs) designed to provide the underpinning data (WPs 1–3) for design and piloting of a multi-component ‘review and revise’ behaviour-change intervention for inpatients/carers and healthcare professionals (WP 4), evaluated for effectiveness and safety (WP 5), and cost-effectiveness (WP 6), summarised in Figure 1.

FIGURE 1 Research pathway diagram. HEE, Health Education England; BSAC, British Society for Antimicrobial Chemotherapy. Note: WP shown in dark grey, peer-reviewed publications in purple, non-peer reviewed publication in dark coral, outputs in turquoise, ARK intervention components in light grey, and pathway to impact in light coral. (Figure not reproduced here; see the original article.)

The research proceeded broadly as planned: the only minor differences [approved by the Programme Steering Committee (PSC)] were:

• Activities with patient and professional stakeholders in WP 3 to understand beliefs and concerns around the ‘review and revise’ process and in WP 4 to co-design the intervention happened in parallel to maximise learning and feedback, and so are presented together in the report. The WP 4 feasibility evaluation resulted in very minor modifications to intervention materials.
• In WP 5, non-randomised pilot sites (n = 3) and randomised main trial sites (n = 36) were analysed together to maximise power, as per the approved protocol, since the intervention did not change between the trial’s pilot and main phase.
• The original ARK acronym was ‘Antibiotic Reduction and Conservation’; however, many people found this confusing and therefore it was amended to ‘Antibiotic Review Kit’.

Work package 1: Overview of systematic reviews

The idea that for antibiotic treatment ‘shorter is better’30 is now well established, in part due to the ARK-Hospital programme. But when the project began, it was clear from our previous research24,25 that there were significant professional concerns that reducing antibiotic durations could compromise patient outcomes. Recommendations for antibiotic duration have historically been syndrome-specific and do not account for differences between bacterial pathogens causing the same illness (e.g. Streptococci and Mycoplasmas causing community-acquired pneumonia), between patients or between agents. They were also set historically when prevalent concerns were that under-use of precious and scarce antibiotics could drive resistance, and before widespread availability of antibiotics and emergence of concerns about overuse and resistance.

By 2016, a growing number of studies had found shorter antibiotic courses to be non-inferior to longer antibiotic courses across a range of common bacterial infections, with a few exceptions. However, this evidence was fragmented and contained within condition-specific reviews, such as ventilator-associated pneumonia31 or urosepsis.32

Research aims

In WP 1, we therefore aimed to summarise the systematic review evidence for the clinical effectiveness of shorter versus longer duration antibiotic treatment for bacterial infections in secondary care across indications. This overview of systematic reviews was reported in Onakpoya et al.33

Methods

The protocol for the overview of systematic reviews was registered with PROSPERO (CRD42016046907). The review included searches of MEDLINE, EMBASE, Cochrane database of review of effects and CINAHL, from inception to May 2016, without age or language restrictions. The bibliography of included studies was also searched.

Reviews were included if they assessed the clinical effectiveness of short- versus long-course antibiotics to treat bacterial infections in hospitalised adults or children. Reviews were excluded if:

• they focused on antibiotic treatment in primary care
• antibiotic prescriptions were not fixed in duration but instead varied based on clinical progress and biomarkers
• the effects of different topical agents, doses, antibiotic classes, combinations of antibiotics, or treatment delays were compared, rather than duration
• antibiotics were used to treat tuberculosis or gastroenteritis.

‘Short’ treatment duration was defined as ≤ 7 days, and ‘long’ duration as > 7 days.

The primary outcome was clinical resolution, as defined in the review studies. Secondary outcomes included mortality, development of AMR, treatment adherence, intensive therapy unit (ITU) admission, adverse events, symptom duration, complications, and microbiological cure.

We assessed the quality of included reviews using Assessing the Methodological Quality of Systematic Reviews (AMSTAR) criteria and of the evidence using Grading of Recommendations, Assessment, Development and Evaluation (GRADE) criteria.

Findings

We screened 641 titles and abstracts and included 6 systematic reviews in the final analysis, representing 3162 participants in 26 randomised trials.

Each review addressed a specific syndrome and population: hospital-acquired bacteraemia,34 meningitis in children,35 community-acquired pneumonia in children,36 hospital-acquired/ventilator-associated pneumonia in adults,31 acute pyelonephritis/septic urinary tract infection32 and pregnant women with amniotic fluid infection.37

• Overall quality of the underlying evidence was rated from very low to moderate.
• There was no overall evidence of differences between shorter versus longer durations across the systematic reviews.
• Among the 26 underlying trials, only one comparison in 1 trial provided low quality evidence favouring longer treatment (microbiological cure in bacteraemic pyelonephritis with 21 vs. 14 days beta-lactams), with all other comparisons finding no evidence of differences or favouring shorter treatment.

Key limitations

The main limitations relate to the relatively low quality and high heterogeneity of the underlying data. Overall, there were little data of variable quality, with insufficient or no data on some important outcomes.

Although our overview was aimed at providing our targeted audience (clinicians, microbiologists, researchers and policy-makers) with up-to-date evidence, an inherent limitation of the approach taken was that some of the included reviews did not include more recent primary studies.

Inter-relationships with other work packages

Despite these limitations, WP 1 findings contributed to the research’s success by collating disparate underlying evidence for the safety of shorter antibiotic treatment durations, which was included within the educational module developed within WP 3/WP 4. It also supported network development by building confidence among AMS practitioners.

Work package 2: Observational data analyses

To address AMR, NHS England sets antibiotic reduction targets for acute hospitals, for example the 1% year-on-year reduction in the 2021–2 NHS Standard Contract.38 Although these targets are relatively modest, AMS leads have nevertheless expressed significant concerns about the safety of achieving them,25 likely due to fears around systematic under-treatment leading to patient harm.

To provide reassurance that substantial reductions in antibiotic use could be achieved safely, WP 2 included two observational analyses using hospital-level and patient-level antibiotic data, respectively, to investigate the impact of reduced antibiotic use on clinical outcomes in acute/general medical inpatients. Completely adjusting for confounding in such analyses is very difficult; the goal was to assess whether there was strong early evidence of harm (greater rates of treatment failure/mortality associated with less antibiotic use), when confounding would have lesser impact.

Research aims

1. To determine the extent to which the substantial variation in hospital-level antibiotic prescribing39,40 was associated with patient-level clinical outcomes after adjusting for case-mix in acute/general medicine admissions in England, in order to estimate what system-wide reductions might theoretically be safely achieved if higher-consuming hospitals replicated prescribing of lower-consuming hospitals. This analysis was reported in Budgell et al.41
2. To investigate the causal effect of using shorter versus longer antibiotic courses using patient-level antibiotic and clinical outcome data from one hospital group (University Hospitals Birmingham NHS Foundation Trust). This analysis has not been previously published and is reported in Appendix 1.

Methods

For the first analysis, inpatient admissions April 2010–March 2017 were obtained from Hospital Episode Statistics (HES) for 135 acute NHS Trusts (denoted ‘hospitals’ below), supplied by NHS Digital. Vital status 14 and 30 days after admission was obtained through death registry linkage performed by NHS Digital. Antibiotic defined daily doses (DDDs) were available quarterly from April 2014 to March 2017 at the hospital level (not by specialty and not before this) from IQVIA through the UK Health Security Agency (UKHSA), as were annual bed-days and admissions. Acute/general medical inpatients may be treated under different adult specialties, and how these are coded varies by hospital,42 meaning general medicine codes (300) likely reflect only a subset (i.e. perfect specificity, but sub-optimal sensitivity). The study population was therefore defined using multiple consultant specialty codes reflecting specialties most commonly used to admit adult acute/general medical inpatients.

We estimated all-cause 30-day mortality risk using a separate Poisson model for each hospital, adjusting for case-mix using 16 admission factors43 and 9 interaction terms. We then used random-effects meta-regression to assess whether heterogeneity in these marginal effects was associated with hospital-level antibiotic DDDs/1000 bed-days or admissions and multiple other metrics of antibiotic use including WHO Access, Watch, and Reserve (AWaRe) classifications.44

The second analysis used de-identified electronic health records from 141,492 adult (≥ 16 years) acute/general medical inpatients admitted to the University Hospitals Birmingham NHS Foundation Trust 1 January 2010–31 December 2015. Date of death was obtained through linkage with the national death registry performed within the underlying NHS data sources (i.e. not as part of this study). Patient-level antibiotic data included dispensing in hospital and at discharge. Analyses included patients admitted or transferred to general medicine within 24 hours of admission, who initiated antibiotics within 1 calendar day of admission, had non-missing deprivation score and had specific test results at admission, many reflecting underlying infection and hence essential to try to adjust for confounding by indication.

We assessed the causal effect of stopping versus continuing inpatient antibiotics on all-cause 30-day mortality by treating time-varying antibiotic use as sequence of non-randomised nested ‘trials’45 up to 10 days after starting antibiotics (see Appendix 1, Table 2; details of methods in Appendix 1). By comparing mortality risk in patients who stopped versus continued, regardless of whether they subsequently stopped or restarted, the estimated causal contrast was analogous to an intention-to-treat (ITT) effect in a sequentially randomised controlled trial. Since patients may stop antibiotics when death is considered imminent (leading to reverse causation between stopping antibiotics and death), each model was modified by lagging the primary exposure (stopped vs. continued) by 1 day. Each ‘trial’ was adjusted for the main exposure (stopped vs. continued antibiotics), and, irrespective of statistical significance, 13 admission factors included in a previous analysis,43 4 additional admission factors, and baseline and the most recent values of 15 laboratory test results and physiological measurements.

Findings

In the first analysis, the study population included 36,124,372 acute/general medical admissions. The adjusted probability of 30-day mortality varied threefold by hospital [0.58–1.75% at the reference category; median: 1.12%, interquartile range (IQR): 0.96–1.28%]. Hospital-level antibiotic use also varied widely, with DDDs/1000 bed-days varying 15-fold (median: 1814; IQR: 1624–2080; range: 266–4006). After extensive adjustment for hospital case-mix, the probability of 30-day mortality changed by −0.010% [95% confidence interval (CI) −0.064 to +0.044] for each 500 higher hospital-level antibiotic DDDs/1000 bed-days. Despite the wide variation, there was no evidence of association between hospital-level antibiotic use and adjusted 30-day mortality for most (22/24) antibiotic consumption metrics; the two models with some evidence identified effects in opposite directions. Most (20/24) metrics of antibiotic use also showed no evidence of association with 14-day mortality, with the remainder showing both positive and negative associations.

These findings suggest that hospitals could, in theory, be benchmarked against their lower prescribing peers (following risk adjustments) to drive safe and substantial reductions in antibiotic consumption. If hospitals with antibiotic consumption above the 25 percentile (1624 DDDs/1000 bed-days; 101 hospitals) reduced their use to this level, total DDDs would drop by 21.6%. If hospitals with use above the 10 percentile (1454 DDDs/1000 bed-days; 121 hospitals) or 5 percentile (1282 DDDs/1000 bed-days; 128 hospitals) dropped consumption to this level, total DDDs would decline by 27.0%, or 34.4%, respectively.

The second analysis included 16,948 adult general/acute medical inpatient admissions, with median age 73 years (IQR: 56–83) and pneumonia the most common primary diagnosis group based on clinical classifications software (CCS) (19.7%, n = 3340) (see Appendix 1, Table 3). The median antibiotic duration was 5 days (relatively small proportions stopped during the first 4 days, Appendix 1, Figure 2), 15.4% re-started inpatient antibiotics within 30 days of stopping and overall 10.5% died within 30 days of starting antibiotics (see Appendix 1, Table 4).

The causal effect of stopping versus continuing inpatient antibiotics on each of the first 10 days after initiation showed no evidence of harm from earlier stopping (see Appendix 1, Figure 3). There was some evidence for lower risk of 30-day mortality in those stopping versus continuing from day 8 onwards, and on day 2, potentially due to residual confounding (see Appendix 1, Figure 4).

Key limitations

The main limitation of the first analysis was being ecological rather than associational or causal; it cannot address how some acute hospitals achieved similar clinical outcomes with much lower antibiotic use, and there could be multiple confounding factors. A second limitation was that antibiotic data were not available at the specialty level. The proportion of antibiotic DDDs attributable to acute/general medicine inpatients (in which we assessed mortality) likely varied by hospital, though our use of a very broad definition of general medicine may partially offset this limitation.

The main limitation of the second analysis was that at least some residual confounding by indication is likely to remain from less sick patients stopping antibiotics quickly and being at lower risk of death. Possible sources of unmeasured confounding include information regarding antibiotic use in the community before admission and history of multi-resistant infections, neither of which were available. The diagnosis codes in electronic health records also pose challenges, as they are based on discharge summaries and are primarily collected for administrative and reimbursement purposes, reflecting the main condition managed in an episode. Consequently, the clinical indication for prescribing antibiotics may be unrelated to the conditions coded at discharge, and adjusting for these diagnoses may effectively use the future to predict the past in some patients. It is not possible to resolve this using routine health records.

Inter-relationships with other work packages

Not demonstrating harm associated with lower antibiotic use in observational analyses was a prerequisite for developing the ARK intervention, since this is necessary, but not sufficient, for lack of harm given the plausible direction of confounding (whereby sicker patients get prescribed more antibiotics, and therefore greater antibiotic use should be associated with higher mortality if confounding is not adequately accounted for). Published findings were included within the educational module developed within WP 3/WP 4.

Work packages 3 and 4: Intervention development and piloting

Parts of this section has been adapted from Cross et al.46 This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The text below includes minor additions and formatting changes to the original text.

In primary care, safe and effective strategies to reduce overuse include ‘restrictive’ interventions, for example avoiding or delaying antibiotics for respiratory tract infections,47,48 though this may require careful targeting of low-risk patients.49 However, as described above, in hospitals clinical urgency and diagnostic uncertainty mean antibiotics are often prescribed rapidly and empirically, and strategies to reduce overuse instead depend on prescribers reviewing and stopping antibiotics when appropriate, as recommended in ‘Start Smart, then Focus’.18 ‘Enablement’ interventions, such as providing decision support and feedback to prescribers, can safely reduce hospital prescribing16 but relatively little research has focused on stopping antibiotics when they are no longer needed, rather than the original decision to prescribe.

Research aims

The overall goal of WPs 3 and 4 was to develop a feasible, generalisable and sustainable multicomponent healthcare professional and inpatient/carer intervention to support antibiotic ‘review and revise’, informed by qualitative studies, relevant theory, existing evidence, and WPs 1–2 findings, and using the person-based approach to intervention development.

Iterative stakeholder consultations and in-depth qualitative studies aimed to deepen understanding of perceptions and experiences of inpatient antibiotic prescribing (particularly focusing on antibiotic duration, not previously addressed in qualitative research in hospitals). We also aimed to identify barriers to and facilitators of change that needed to be addressed in implementation guidance. This iterative process, leading to the initial ARK intervention, was reported in Santillo et al.50

The resulting intervention was then tested using mixed methods in a feasibility study in one NHS Trust in order to test whether it should proceed to definitive evaluation in a non-randomised internal pilot and then a stepped-wedge cluster-randomised controlled trial. This feasibility study was reported in Cross et al.46 (available here51) with patient experiences reported in Mowbray et al.52

Methods

We co-designed the complex digital, behavioural and organisational ARK intervention using the theory-, evidence- and person-based approach.53,54 Intervention development followed three stages, each using a stakeholder workshop with healthcare professionals across different grades, cadres and specialties working in acute/general medicine:

1. Planning and developing the initial intervention.
2. Optimising the intervention as an online educational, motivational and decision support tool.
3. Optimising the implementation of the intervention.

All stakeholder workshops collated potential barriers and facilitators to stopping antibiotics at ‘review and revise’ and how these might be managed. Additional ‘think-aloud’ interviews probed target users’ challenges, issues, needs and context in greater detail. The intervention was described following the Template for Intervention Description and Replication (TIDieR) guidance,55 mapping intervention components onto a generic theoretical taxonomy, and the hypothesised causal relations between key intervention components/techniques and outcomes described using a logic model. The patient leaflet was co-designed using a similar iterative process including the ARK patient and public involvement (PPI) group.

In the feasibility study, acceptability of the different intervention components was assessed over 12 weeks in terms of online tool uptake, adoption of the decision aid into prescribing practice and rates of decisions to stop antibiotics at review (assessed through repeated point-prevalence surveys). Patient perceptions of the information leaflet were assessed through a brief questionnaire and a qualitative interview study involving semistructured, think-aloud, telephone interviews in a convenience sample of acute/general medical inpatients.

Findings

Stakeholders reported that the high workload and burden of existing e-learning modules would reduce the acceptability of a long education module. Attending seminars was also seen as difficult; incorporating training and discussion within existing structures, for example handover meetings and ward ‘huddles’ was preferred. A key barrier to stopping antibiotics was the initial prescription and subsequent ‘review and revise’ often being done by different people. Absence of information about the original prescriber’s rationale for, and degree of certainty about, the need for antibiotics, led to concerns that the different healthcare professionals making decisions about stopping could be ‘missing something important’ and hence antibiotics were often continued ‘just in case’. Increasing transparency around the initial prescription decision therefore became an integral component of the ARK intervention. Using a standardised proforma was not attractive due to challenges incorporating it in diverse hospital systems. Participants felt that ARK needed to integrate with existing processes, with a selection of tools to help implementation, particularly as some hospitals were using electronic prescribing but others were still using paper drug charts.

This led to development of a flexible decision aid to both capture and communicate the certainty surrounding the initial antibiotics prescription (Table 1). The education module was reduced to 7–10 minutes and peer-led seminars replaced with regular, short discussions within routine clinical team meetings. A selection of resources was developed to help each ARK Champion (site lead) and their team implement ARK as appropriate in their hospital. This content was refined using ‘think-aloud’ interviews with 20 diverse healthcare professionals from 6 hospitals, in particular adding clarity around the key roles of non-prescribers and the need to provide more evidence about the safety of short courses. Further stakeholder workshops refined the implementation plan, taking local teams through the necessary steps to implement ARK, including piloting and testing, and developing new resources including spreadsheets for local data collection and quizzes to test understanding of the ARK decision aid.

The developed ARK behaviour change intervention, comprising four components (see Table 1), was then evaluated in a feasibility study in one hospital to assess whether it could support stopping antibiotic treatment when it is safe to do so among hospitalised patients.

All intervention components were successfully introduced into practice. One hundred and thirty-two front-line staff completed the online tool (19.4 healthcare professionals per 100 acute beds), including 97% (32/33) of those pre-specified by the local site team as essential for its uptake (target 70%). Of 588 prescription charts included in 7 point-prevalence surveys, 82% (76–90% across surveys) used the decision aid (target 50%). More prescriptions were categorised as ‘probable’ (335/588, 57%) rather than ‘possible’ (135/588, 23%) using the decision aid, but the proportion of ‘possibles’ increased over the 12-week implementation [odds ratio (OR) possible vs. probable = 1.11 per week (95% CI 1.04 to 1.17); p = 0.001]. Overall prescription review rates increased from 91% (69/76) at baseline to 99% (450/457) post implementation [OR post vs. pre implementation = 6.52 (95% CI 2.22 to 19.16); p = 0.001]. Antibiotic stop rates at 72 hours increased significantly to 36% overall (156/450) (range: 29–40% across surveys) from 9% (6/69) pre implementation [OR = 5.57 (95% CI 2.36 to 13.16); p < 0.001] and were higher for prescriptions categorised as ‘possible’ (48%, 51/106) than categorised as ‘probable’ (29%, 75/260) [OR possible vs. probable = 2.29 (95% CI 1.44 to 3.64); p = 0.001], with a similar trend for those where the decision aid was not used (36%, 30/84) [OR possible vs. not used = 1.67 (95% CI 0.93 to 3.00); p = 0.09].

Patients rated the leaflet highly on providing all the information they needed [mean score (maximum 7): 6.5; standard deviation (SD) 0.52], being helpful (6.4; SD 0.63), trustworthy (6.5; SD 0.64), and something they would recommend to others (6.2; SD 0.86), with 13/15 (87%) stating that they would be likely or extremely likely to recommend the leaflet to friends and family taking antibiotics in hospital.

Ten women and five men, aged 50–91 years, participated in the patient interview study. Overall, the leaflet was viewed positively, raising some information that patients were not previously aware of that they could then discuss with the clinical team, friends and family. Patients had positive experiences of being prescribed antibiotics initially and then understanding why antibiotics were changed. In some cases, patients felt that the leaflet had helped them to make sense of their experiences.

The patient leaflet was revised to clarify points raised in these interviews. The study highlighted the importance of testing messages with the target audience. Both the ethics committee and some stakeholders were unclear regarding the usefulness and responsibility of providing this kind of information to patients, fearing it might increase concerns about antibiotic treatment or resistance. We found that patients were keen to receive proactive, information, allowing them to feel more confident and invested in their care.

Key limitations

A limitation of any behavioural intervention is that one programme may not fit all needs; however, providing a flexible toolkit and behaviour change support that addresses potential barriers to allow the local context to drive implementation is one solution. One important issue when translating interventions into practice is balancing intervention fidelity and the need for adaptation to local contexts. ARK made adaptation an integral part of its design to support translation into practice, a critical design issue highlighted by stakeholders given substantial variation across different hospitals.

Some metrics used in the feasibility study were arbitrary, particularly the number of staff completing training per 100 acute beds. However, for practical reasons training was rolled out across the entire clinical area, and staff change continuously, making assessing either the number of staff working in areas where ARK was implemented or invited to do the online learning challenging.

The patient interview study was based on a convenience sample of patients admitted during the feasibility study. In total 125 patients were approached, 25 provided consent to be contacted, but 10 dropped out, either because they no longer wanted to take part by the time of interview or because they could not be contacted, leaving only 15 study participants who could be included.

Inter-relationship with other work packages

These WPs developed and tested the feasibility of the ARK intervention components. Showing that these were a feasible and acceptable way of supporting stopping antibiotics safely at post-prescription reviews allowed their further evaluation in the pilot and main phase of the ARK-Hospital trial.

Work package 5: Inpatient intervention evaluation

Parts of this section has been adapted from Llewelyn et al.2 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Assessments of antimicrobial reduction strategies often suffer from study design weaknesses that limit the translation of evidence into clinical practice,56–60 with many studies being non-experimental, underpowered and failing to report clinical or microbiological outcomes.57 Consequently, the quality of evidence to support specific stewardship objectives is low and heterogeneous.59

The ARK programme’s intervention design process and feasibility assessment, described above, and the internal pilot and full trial evaluation, described here, were designed to overcome these common design weaknesses. The trial evaluation included a qualitative process evaluation and a separate sub-study to assess whether the faecal resistome of medical admissions could be used to directly measure harms associated with antibiotic usage, a direct measure of AMR.

Research aim

The aim was to definitively evaluate the ARK intervention developed through WPs 1–4 and assess whether it could safely reduce hospital antibiotic use. By supporting clinician-led decisions to stop antibiotics at review, the intervention aimed to reduce treatment duration, rather than targeting decisions to start antibiotics, as this approach has the potential to reduce overall use without delaying the rapid administration of empirical antibiotic treatment in acutely unwell patients. The ARK-Hospital trial protocol was published in Walker et al.1 The results of the trial were reported in Llewelyn et al.2 Additional details of trial results are provided in Appendix 2.

As well as assessing multiple clinical and antibiotic outcomes in the main trial, a parallel quantitative and qualitative process evaluation aimed to investigate the mechanisms through which the ARK intervention affected antibiotic prescribing in more detail. The quantitative analysis was reported as part of the main trial report. The qualitative analyses have not been previously published and are reported in Appendix 3.

Finally, the ultimate goal of shortening antibiotic treatment is to reduce the pressure for resistance to emerge, but directly measuring the AMR burden carried by individuals is challenging. We therefore assayed total resistance gene burden (the gut ‘resistome’) in discarded faecal samples taken from acute/general medical inpatients using metagenomics to detect and quantify AMR genes. We aimed to test the hypothesis that differences in antibiotic use across three different hospitals in the UK would lead to differences in the resistome of individual patients. The results of this study were reported in Yokoyama et al.61

Methods

The safety and effectiveness of the ARK intervention were assessed in a non-randomised internal pilot in 3 acute NHS hospitals (the first 3 to volunteer), immediately followed by a stepped-wedge, cluster-randomised controlled trial (with hospital as the unit of randomisation) in 36 acute NHS hospitals (or their equivalent in the devolved administrations) across all UK nations. Hospitals were randomised to an implementation date at 1- to 2-week intervals over approximately 18 months using computer-generated sequences in seven calendar-time blocks. Each block contained six hospitals, who were randomly allocated over six implementation dates within the block, pre-specified before the trial started. Some implementation dates were shifted by a week to avoid school holidays and implementation was paused over summer holidays given complexities of rolling out an intervention across acute/general medicine when many staff were on leave. Twenty-four hospitals (four blocks; randomly allocated) had already been recruited before randomisation started. The remainder joined during the early part of the trial, but could not have been randomised within the earliest blocks. To avoid contamination, complete information about the intervention was concealed until randomisation (12 weeks before implementation), when local preparations were designed to start. The trial (including feasibility, pilot and cluster-randomised phases) was registered as ISRCTN12674243.

To be eligible, hospitals needed to admit adult acute/general medical inpatients, be able to provide the required study data, and have a local ‘Champion’ willing to lead the intervention implementation. The study population was identified from electronic health records using consultant specialty codes, as in WP 2 above (see Appendix 2, Figure 5).

The trial had two coprimary outcomes: (1) total monthly hospital-level antibiotic DDDs per adult (≥ 16 years) acute/medical admission (superiority) and (2) patient-level all-cause 30-day mortality (non-inferiority margin 5%, meaning to declare non-inferiority the upper 95% confidence bound for the relative immediate mortality change associated with the intervention needed to be below a 5% increase). Secondary antibiotic (superiority) outcomes included total DDDs per acute/medical bed-day, and DDDs/admission for the following antibiotic categories: oral and parenteral antibiotics, narrow-spectrum and broad-spectrum agents, carbapenems and the UKHSA’s interpretations of the WHO’s AWaRe antibiotics.44 Secondary clinical (non-inferiority) outcomes included intensive care unit admission, hospital length of stay, 90-day mortality, emergency re-admission to any specialty within 30 days of discharge, and CDI or colonisation within 90 days of admission. All de-identified data were collected electronically from local hospital systems.

Statistical analysis

The protocol pre-specified that the Acceptance Checklist for Clinical Effectiveness Pilot Trials62 would be used to decide whether to combine data from the pilot and main trial: there was no change to the intervention following the internal pilot so this was done. We used an interrupted time series analysis to estimate an immediate implementation effect and a sustained year-on-year effect post versus pre implementation in each site, based on the randomised implementation date. This analysis followed an ITT approach that ignored any differences in when the intervention was implemented or how well, but was modified to exclude seven sites that withdrew after randomisation but before implementation or data collection began in whom the intervention effect was zero by definition (sometimes called modified ITT; using the estimands framework, this corresponds to a ‘principal stratum’ strategy regarding the attempt to implement the intervention and a ‘treatment policy’ strategy for the implementation itself). An overall weighted mean of the site-specific effects was derived using random-effects meta-analysis. Random-effects meta-regression was used to assess how pre-specified potential effect modifiers were associated with greater or lesser impacts of the ARK intervention on the coprimary outcomes. With at least 36 sites, the study had > 85% power to detect an immediate 15% relative reduction in antibiotic use and exclude a 5% relative increase in 30-day mortality associated with implementation.

To enable outcome assessment before and after implementation, electronic study data were collected from February 2016 to October 2020 (see Appendix 2, Figure 6). Antibiotic end points were modelled using negative binomial regression. Binary patient-level clinical end points were modelled using logistic regression. Subhazard regression was used to model hospital length of stay (days) censored at 90 days, with in-hospital deaths as a competing event and admission durations < 1 day set to 0.1 days. All clinical end points were modelled with a robust variance adjustment by patient.

Since the pandemic greatly affected the study end points, all models adjusted for COVID-19 as a binary indicator for March–June 2020, with the exception of models for C. difficile as trend estimates for this outcome were not visibly impacted by COVID. Sensitivity analyses excluded data after February 2020, including 12 sites with < 12 months post implementation data as a result. Antibiotic models also adjusted for seasonal effects by fitting day of year as a sin() + cos() function. Models of patient-level clinical end points were also adjusted for admission characteristics, regardless of statistical significance (based on43), including:

• age
• sex
• Index of Multiple Deprivation (IMD) percentile
• Charlson comorbidity index, and its interaction with age
• immunosuppression
• admission method
• admission source
• admission specialty
• patient classification
• admission day of week (weekend vs. weekday)
• admission day of year and time of day (both modelled as a sin() + cos() function), including an interaction between time of day and admission day of week
• number of overnight admissions in the past year
• any overnight complex admission in the past year (counting admissions with > 1 consultant episode, excluding episodes in the emergency department and rehabilitation)

Ethnicity was not adjusted for as it was missing in median 8.8% admissions (IQR: 4.5–18.4%) across sites.

Oversight

The PSC was chaired by Alison Holmes (Professor of Infectious Diseases, Imperial), with independent members Philip Gothard (Consultant Physician, University College London Hospitals NHS Foundation Trust), Karla Hemming (Professor Biostatistics, University of Birmingham, an expert in stepped-wedge cluster randomised trials) and Cliff Gorton (patient and public representative). An independent Data Monitoring Committee (DMC) reviewed outcome data from the main trial three times during the trial by the DMC, which used a Haybittle–Peto statistical rule for early stopping. The DMC was chaired by Neil French (Professor of Infectious Diseases, University of Liverpool), Tom Rogers (Professor of Clinical Microbiology, Trinity College Dublin) and James Lewis (Professor of Social Sciences, University of Cardiff).

Qualitative process evaluation

For the qualitative process evaluation we analysed audio-recorded and transcribed semistructured telephone interviews lasting 30–60 minutes with 58 ARK-Hospital Champions and core team members from 35 trial sites between 5 February 2018 and 4 February 2020. The interviews were conducted at different points after implementation: 22 were carried out at 2–4 months, 24 at 5–6 months and the remainder at 6–12 months. Research questions centred around healthcare professionals’ perceptions of the ARK intervention; impacts of ARK Hospital on antibiotic prescribing; perceived benefits and challenges of implementation and sustainability. The initial coding focused on healthcare professionals’ perceptions and experiences of the ARK intervention as well as benefits and challenges to intervention implementation and sustainability. A pragmatic rapid analysis approach was adopted,63 organising data around the principal themes evident in the data that were relevant to our research questions. Initial analysis and coding was led by Fiona Mowbray, an experienced qualitative researcher, but was iterative in nature and supported by discussion of emerging themes among the wider study team. Further detailed analysis was precluded by COVID-19 (see Appendix 3 for details).

Faecal resistome

Three acute Trusts taking part in the ARK-Hospital study with differing rates of antibiotic use and CDI sent randomly selected completely anonymised faecal samples from acute/medical inpatients that were to be discarded after C. difficile testing over a period of 9–15 months (determined by the period of the site’s involvement in the ARK-Hospital trial and availability of clinical laboratory staff to process samples, varying between July 2017 and October 2019). The only metadata available for samples was the date of sampling and hospital site. This approach was taken to avoid the burden of, and biases inherent in, taking informed consent. Metagenomic DNA was extracted from these samples and sequenced using an Illumina NovaSeq 6000 platform to a minimum depth of 10 million reads [including human (discarded) and non-human]. Taxonomic composition and the presence of AMR genes was determined from the resulting sequencing reads using Kraken2 and a published method implemented within ResPipe (v.1.5.0),64 providing ‘AMR genes per bacterial cell’ for each sample, using random subsampling to 10 million sequences per sample to address uneven sequencing depth across samples.

Findings

Three pilot sites implemented the intervention between 25 September 2017 and 20 November 2017; 43 further sites were randomised to implement between 12 February 2018 and 1 July 2019, of which 7 withdrew before implementation and were excluded from analyses as no data were collected (see Appendix 2, Figure 7) leaving 36 randomised sites. The 39 included sites (pilot + randomised) were distributed across the UK, with 32 in England, 4 in Northern Ireland, 2 in Wales, and 1 in Scotland (see Appendix 2, Table 5). In the 12 months before implementation, antibiotic use varied widely across sites, with Reserve antibiotics accounting for 0.3–5.7% of total DDDs, Watch 4.8–44.1%, and Access 30.7–85.2%. Antibiotic prescribing was paper-based in 25 (64%) sites at implementation. Twenty-one (54%) sites implemented the decision aid with a hard stop, meaning prescribers had to stop initial prescriptions within 72 hours unless already finalised, while 9 (23%) sites implemented as a soft stop, meaning prescribers were reminded to stop or finalise prescriptions within 72 hours, and the remaining 9 (23%) sites did neither.

Implementation fidelity was varied, with sites achieving a median 6 of the 8 pre-specified fidelity criteria (IQR: 5–7) and 9 (23%) sites achieving ≤ 4 criteria. Implementation was delayed in 9 (23%) sites, by a median 7.4 weeks (IQR: 6.1–13.0, range: 5.3–25.0). Over the first 12 weeks after implementation, sites aimed to have ≥ 70% of essential staff complete the online training, but 16 (41%) sites did not meet this target (see Appendix 2, Figure 8). A median 52% (IQR: 31–76) of audited antibiotic prescriptions were categorised using the decision aid, and prescription review rates were high at implementation (median 91.0%) and 12 weeks later (median 89.9%). Prescriptions were more often stopped at review in the 12 weeks after implementation than at baseline (median 16.2% vs. 12.7%, Wilcoxon-matched pairs p = 0.006), and increases were greatest in sites with lower baseline rates.

Assessment of the coprimary end point, total DDDs per admission, included post implementation follow-up spanning a median 23 months (IQR: 18–28, range: 14–37) across 38 sites (1 site excluded from analyses of antibiotic use as unable to provide data) (see Appendix 2, Figure 6). Implementation was associated with an overall adjusted −1.0% (95% CI −4.0 to +2.1) (p = 0.54) immediate change in total DDDs/admission and a sustained −4.8% per year (95% CI −9.1 to −0.2) (p = 0.042) change in trend post versus pre implementation (see Appendix 2, Figure 9) (incidence rate ratios from negative binomial regression), but with considerable heterogeneity among sites (see Appendix 2, Figure 10).

Assessments of 30-day mortality included post implementation follow-up spanning a median 22 months (IQR: 19–30, range: 16–38) across 39 sites (see Appendix 2, Figure 6). Among 7,160,421 admissions, 4.4% (n = 314,313) died within 30 days (range across sites: 2.6–7.2%) (see Appendix 2, Table 6). Implementation of the ARK intervention was associated with an overall immediate change in adjusted 30-day mortality odds of −2.7% (95% CI −5.7 to +0.3) (p = 0.079), which was consistent with non-inferiority (see Appendix 2, Figure 9) (ORs from logistic regression). There was weak evidence of an overall year-on-year increase in adjusted 30-day mortality odds of +3.0% (95% CI −0.1 to +6.2) (p = 0.060), with considerable heterogeneity among sites (see Appendix 2, Figure 10). However, this increase was not found after excluding admissions after the onset of the COVID-19 pandemic in March 2020 [year-on-year change post vs. pre implementation −0.9% (−4.0 to +2.4) (p = 0.60)]. Also, we found no evidence sites with greater decreases in total DDDs/admission (vs. smaller decreases) had greater immediate (r = 0.044, p = 0.79) or sustained year-on-year (r = 0.011, p = 0.95) increases in 30-day mortality.

Implementation was associated with immediate reductions in DDDs/admission for broad-spectrum agents (−6.3%, 95% CI −9.5 to −3.0) (p = 0.0003), Watch antibiotics (−9.5%, 95% CI −13.2 to −5.6) (p < 0.001) and quinolones (−15.5%, 95% CI −21.9 to −8.5) (p < 0.0001), and immediate increases in Access antibiotics (+ 4.4%, 95% CI +0.3 to +8.8) (p = 0.037) (see Appendix 2, Figure 9). There were also sustained year-on-year reductions in Access (−5.3%, 95% CI −10.0 to −0.4) (p = 0.033), Watch (−11.0%, 95% CI −17.1 to −4.5) (p = 0.001), narrow-spectrum (−5.2%, 95% CI −9.4 to −0.9) (p = 0.019) and oral antibiotics (−6.4%, 95% CI −11.2 to −1.4) (p = 0.12) post versus pre implementation. There was no evidence of a sustained change in broad-spectrum (−2.6%, 95% CI −8.5 to 3.6) (p = 0.40) or parenteral antibiotics (−0.9%, 95% CI −5.2 to +3.6) (p = 0.70), but there was evidence of year-on-year increases in carbapenems (+12.3%, 95% CI 2.3 to 23.2) (p = 0.015), with similar trends for Reserve agents (+7.3%, 95% CI −1.5 to 17.0) (p = 0.11), albeit from low levels and absolute increases were small.

As with 30-day mortality, there was some evidence of a sustained year-on-year increase in 90-day mortality odds of +3.9% (95% CI +0.5 to +7.4) (p = 0.023), but this was not observed after excluding data from 1 March 2020 (−3.9%, 95% CI −8.3 to +0.7) (p = 0.18), and there was no evidence of implementation impact on:

• ITU admission (immediate effect: +2.3%, 95% CI −1.9 to +6.7; p = 0.28; sustained effect: −5.9%, 95% CI −12.8 to +1.6; p = 0.12).
• Emergency 30-day re-admission to hospital (immediate effect: −0.1%, 95% CI −2.6 to +2.5; p = 0.96; sustained effect: −1.5%, 95% CI −4.6 to +1.6; p = 0.32).
• Length of stay [immediate relative change in subdistribution hazard ratio (SHR) per additional day in hospital: −0.3%, 95% CI −1.0 to +0.5; p = 0.48; sustained change: +0.1%, 95% CI −0.8 to +1.1; p = 0.77].
• CDI (immediate effect: −4.6%, 95% CI −16.6 to +9.0; p = 0.49; sustained effect: +5.8%, 95% CI −8.1 to +21.9; p = 0.43) or colonisation (immediate effect: −5.6%, 95% CI −16.9 to +7.2; p = 0.37; sustained effect: −6.9%, 95% CI −17.5 to +5.0; p = 0.24).

Thus, the weak, inconsistent intervention effects on mortality are likely a result of the COVID-19 pandemic.

Point estimates for immediate and sustained implementation effects are summarised in Appendix 2, Table 7, while trend estimates before and after implementation are summarised in table S8 in Llewelyn et al.2

Quantitative process evaluation/implementation effect modifiers

There was no evidence overall implementation fidelity was associated with effects on total DDDs/admission (immediate effects per additional fidelity criteria achieved: −0.5%, 95% CI −2.7 to +1.7; p = 0.64; sustained year-on-year effects: −1.4%, 95% CI −4.9 to +2.3; p = 0.45) or on 30-day mortality (immediate effects per additional fidelity criteria achieved: +1.3%, 95% CI −6.8 to +3.3; p = 0.19; sustained year-on-year effects: −0.1%, 95% CI −2.1 to +1.9; p = 0.93) (see Appendix 2, Table 8). However, specific implementation fidelity criteria were associated with intervention effects on the coprimary end points, suggesting overall fidelity may be averaging across elements, only some of which are drivers of change. For example, sites with processes for ongoing audit and feedback in place by implementation had greater immediate reductions in DDDs/admission (by −16.6%, 95% CI −28.5 to −2.8, interaction p = 0.022) than those that did not. There was also weak evidence of greater sustained year-on-year reductions in total DDDs/admission among sites that introduced ARK categories into the prescribing process by their randomised implementation date (by −11.5%, 95% CI −22.9 to +1.7, vs. those that did not, interaction p = 0.083), and among sites with higher uptake of the online learning (i.e. training completed by ≥ 20 people per 100 acute beds) (by −9.9%, 95% CI −19.7 to +1.1 vs. lower uptake, interaction p = 0.075).

Qualitative process evaluation (Appendix 3)

Overall, Champions and core team members viewed the ARK intervention as important and worthwhile, with many noting its good fit with existing antibiotic guidelines and stewardship activities. Many Champions also spoke about positive experiences of both the planning and implementation phases, citing the various training materials and resources that they found useful. Each Champion spoke about their experience of engaging both their core team and the wider clinical teams with the intervention, citing the importance of specific ARK kick-off meetings followed up by regular, brief communication at all stages of the study. Although Champions spoke about the challenges of meeting frequently with staff, those who were able to provide feedback face-to-face tended to portray overall engagement at their hospital more positively. Several Champions reported that providing staff with a comparison of their implementation performance, both over time and across wards or departments, was beneficial and resulted in a greater sense of ownership and involvement in the study.

Although Champions all recognised the importance of meeting regularly with staff, this was often a challenge due to competing time pressures and frequent staff changes. Champions at all sites discussed their experiences of changing the drug chart to include the ARK decision aid categories. While the process was straightforward for some, for others this was more challenging. The IT systems in place for electronic drug charts were sometimes inflexible and paper drug charts often could not be changed in time for the randomised start date, even within the 12-week implementation phase. In these cases, solutions such as stickers were used to indicate categories, though this was time consuming and not a long-term solution.

There was often concern about the risks of implementing the ARK recommendation that if the likelihood of infection was only ‘possible’ rather than ‘probable’ a short course of antibiotics should be prescribed, to be reviewed after 48 hours. This was considered particularly problematic before weekends/public holidays, when fewer senior staff would be available to review the prescription. Staff shortages and changeovers were also considered a threat to the review process, leading some hospitals to adopt a more cautious approach to prescribing short courses. Individual prescribers were sometimes motivated to overuse the ‘probable’ ARK category to justify prescribing antibiotics when anxious about the potential consequences of prescribing a short course.

Several Champions discussed the impact of implementing ARK only in the acute medical unit (AMU). Limiting the study to AMU was intended to ensure the scale of implementation and data collection activities was manageable. However, several sites found that many patients prescribed antibiotics before reaching AMU would have an initial antibiotic prescription, but no category indication, making it difficult to implement the ARK review and revise process. Other sites reported that patients moved on quickly after initial prescription and there was little chance to follow up their care pathway, meaning that antibiotic review might not happen following ARK guidelines and posing difficulties for the data collection required for the audit and feedback element of ARK. Some sites chose to address this challenge by extending the ARK intervention to other wards or departments, encouraging more staff to take part and giving ARK a broader reach than initially anticipated.

The ARK kick-off meetings and requirement for all staff to complete the online training at the start of implementation gave an impetus to ARK implementation that could be difficult to sustain in the longer-term. Persistent effort was needed to ensure that new staff completed the initial ARK online training and maintain audit and feedback to front-line staff.

Faecal resistome

Six hundred and eighty-three faecal metagenomes were generated, a median of 18 per site per calendar month (IQR: 15–22, range 0–42). The number of reads which mapped to AMR genes ranged from 272 to 179,190 per sample (IQR: 4022–24,068, median 9510). We found huge variation between individuals in terms of taxonomic diversity (Shannon Index range 0.10–3.99) and carriage of AMR genes (genes per cell per sample range 0.03–48.55; median 1.50). We found no evidence of differences in diversity across the three acute hospitals [median Shannon index 2.16 (IQR 1.71–2.56), 2.15 (IQR 1.62–2.50) and 2.26 (IQR 1.55–2.51), respectively] nor in carriage of AMR genes [median 1.37 genes/cell/sample (IQR 0.70–3.24), 1.70 (IQR 0.70–4.52) and 1.43 (IQR 0.55–3.71), respectively]. There was no evidence that the percentage of samples with genes conferring resistance to each major antibiotic class varied across the three sites. We also found no evidence of differences in either diversity or carriage of AMR genes over time within each hospital group.

Key limitations

Classical analysis of stepped-wedge cluster randomised controlled trials is based on comparing those units randomised to control versus intervention in each time period (or ‘step’), averaging across time periods to obtain an overall estimate of the intervention effect over the period during which it is rolled out to obtain an overall estimate of the intervention effect. Randomisation ensures unbiasedness of this classical stepped-wedge estimator in expectation – but not in one realisation of such in a single trial. Any kind of heterogeneity between randomised units, with regard to factors other than the intervention, or the effect of the intervention itself or underlying event rates varying over time, increases the chance of large imbalances across the steps, increasing variability and reducing the ability to detect intervention effects. Hospitals are extremely diverse in terms of their structures, and so, with a relatively small number of units, chance imbalances were considered likely were we to rely on the standard analysis above. This was compounded by the huge pressure on AMS over the period over which the trial ran, with financial penalties for failing to achieve specific targets, meaning that independently of the randomised intervention, hospitals were trying numerous other approaches to stewardship, leading to additional variability both between hospitals and over time. One solution would have been to increase the number of hospitals randomised, to reduce the impact of random variability, but this was not feasible within either the funding envelope or the healthcare system (we ended up including around one in four acute hospitals in the UK in the trial, and approached nearly all). Given these constraints, a key question was how to reduce variability across hospitals – a crossover design is a well-recognised method to achieve this, leading to our choice of analysis method. This further allowed us to assess the longer-term trends in rates after the intervention was implemented, which is not possible in a classical stepped wedge analysis, but is of key importance to AMS, where we often expect the impact of interventions to increase as they become more closely embedded in routine practice. Although it was not a factor in our decision, which was pre-specified in the original protocol, exploiting the crossover design also more easily accommodated the fact that only 24 of the 36 trusts who were eventually randomised had agreed to join the trial when it started.

Although the trial included over a quarter of acute hospitals in the UK, some time-varying confounding factors may remain unbalanced, despite randomisation, such as the COVID-19 pandemic (which began after implementation in all sites) or other changes in hospital stewardship, case-mix, or AMR rates (which were unknown but may vary by site and over time). It is also possible participating sites had more effective AMS teams not representative of all UK acute hospitals, although this could either increase or decrease intervention effects. Since we relied on a broad list of specialty codes to define the patient population (see Appendix 2, Figure 5), some post implementation prescribing decisions were likely not exposed to the intervention (e.g. in outlying surgical patients), diluting the implementation effect on antibiotic use. Regarding effect modifiers, it was not possible to assess patient-level mediators of impact as patient-level antibiotic data was only provided by four sites.

Regarding the qualitative process evaluation, the main limitation was that we were unable to undertake a systematic and comprehensive analysis and triangulation of the mixed-methods data due to re-deployment of study team members with the COVID-19 pandemic. We are therefore unable to provide more detailed insights into what elements of ARK may have been most useful or problematic, and we cannot make confident statements about the reasons why some hospitals were more successful in implementing ARK than others. The views of our participants also principally reflect those of the Champions and some core team members, who are likely to have been the most positive and engaged staff. Staff who felt less positive about ARK may well have been more reluctant to be interviewed. Interview and focus group participants may have felt social pressure to express positive views to the ARK team. However, the research team found that during the development phase of the ARK intervention hospital staff were forthcoming with criticisms of aspects of ARK that needed to be improved.

Regarding the faecal resistome sub-study, if samples were taken from patients recently admitted to hospital then the results may potentially be more likely to reflect community antibiotic use, for which data were not available. Our study was relatively small; pooling larger numbers of samples might better reflect hospital-level prescribing. Our design meant we had no metadata on factors such as age, gender, ethnicity, medical history, previous antibiotic exposure and time from admission, all of which may influence AMR gene burden. Finally, the sensitivity of metagenomic data for profiling AMR genes is strongly associated with sequencing depth;64 we may therefore not have detected AMR genes present at very low frequencies but nevertheless clinically relevant, and ready to emerge under antibiotic selection.

Inter-relationship with other work packages

Work package 5 comprised the main definitive evaluation of the intervention developed in the earlier WPs, both quantitatively (including on resistance) and qualitatively, and formed the basis for the subsequent health economic evaluation.

Work package 6: Health economic assessments

Up to 20–30% of initial prescriptions among acute medical admissions could be stopped safely through ‘review and revise’ ~48–72 hours after a patient starts antibiotic treatment.65 However, in routine practice, typically < 10% are stopped. Sociocultural and behavioural factors, such as fear of adverse outcomes for patients, being uncomfortable with uncertainty, professional hierarchy and conflicting antibiotic prescribing guidelines, likely play a role in the decision to stop treatment.66 Quantitative evidence on the relative importance of these and other factors could support the design and implementation of efficient stewardship interventions, by directing interventions to target the most influential factors. Complementing qualitative studies exploring individual views in depth, survey designs such as discrete choice experiments can provide quantitative evidence on the relative importance of such factors across a population.

An important part of the evaluation of the ARK-Hospital intervention was to estimate whether it was cost-effective; consideration of the costs of providing the intervention and its consequences may also provide information on how the intervention could be rolled out across the NHS if shown to be cost-effective.

Research aims

The first part of the health economic assessments involved undertaking a choice experiment to quantify the relative importance of different factors (attributes) affecting antibiotic prescribing, particularly decisions to continue or discontinue antibiotics at the ‘review and revise’ stage in acute care. This study was reported in detail in Roope et al.67

The second part involved conducting an economic evaluation, to estimate the cost-effectiveness of rolling out the ARK intervention across the NHS. The intention was to conduct a within-trial cost-effectiveness analysis of the ‘review and revise’ intervention, taking an NHS perspective, based on data collected within the main trial, and modelling the differences in costs and outcomes following the implementation of the ARK intervention, compared to standard (counterfactual) care without it. Note: this analysis has not been previously published and is reported in Appendix 4.

Methods

In the first part, we conducted a choice experiment using an online a survey method to elicit preferences for antibiotic prescribing. Acute/general hospital prescribers in England were asked if they would continue or discontinue antibiotic treatment in 15 hypothetical scenarios described according to six attributes: (1) the patient’s presenting symptoms; (2) the patient’s premorbid condition; (3) whether discontinuation would conflict with local prescribing guidelines; (4) risk of significant harm arising from continued antibiotic treatment; (5) risk of significant harm arising from discontinuing antibiotic treatment and (6) the level of external pressure to continue antibiotic treatment. We estimated a conditional logistic regression model, including a fixed effect at the level of the respondent to capture any unobserved heterogeneity between respondents.

The economic evaluation was undertaken from the perspective of the UK NHS and personal social services. The time horizon for the analysis assumed that, once implemented, the intervention activities continue for 33 months, reflecting the duration of available study data, and also expectations of the duration of effect of an AMS intervention, before further investment is required to re-invigorate, update or replace the intervention.

Costs of implementing the intervention included resource use (i.e. time) for the Champion and the core team (e.g. for ARK project kick-off meetings, audits) estimated from a secondary analysis of interview transcripts with the ARK Champions performed as part of the process evaluation (WP 5), time for training staff to implement the ARK intervention taken from records of online training, estimated additional consults that would be required at the end of a week to avoid missing the 72-hour target for prescription review that would fall later in the weekend, and £500 per hospital for general costs such as printing of materials and audit awards. Costs per hour for relevant job roles and levels were taken from the Personal Social Services Research Unit (PSSRU) database of Unit Costs of Health and Social Care 2018.68

Costs of hospital admissions were estimated from Healthcare Resource Group (HRG) codes assigned to each hospital episode using NHS grouper software. Costs associated with these HRG codes were obtained from the 2018–9 NHS cost collection, including costs of treatment, excess bed-days, and unbundled HRG codes. Excess bed-day costs were not available in the 2018–9 data set, so 2017–8 costs were used and inflated using PSSRU inflation rates. Episode costs were then summed for each admission.

Antibiotic costs per DDD for each dose and formulation were obtained from the pharmaceutical electronic market information tool (eMIT) for the 2018 and applied to the bulk antibiotic dispensing data provided for the wards that implemented the ARK intervention (most were unable to provide patient-level electronic prescribing data) to provide a mean per-admission antibiotic cost per month per hospital.

We used modelling and probabilistic sensitivity analysis to quantify the extent of uncertainty in estimates of the costs and outcomes from the study data, and its effect on the cost-effectiveness results. We fitted a statistical model to each outcome (e.g. admission costs), and then allowed all the inputs to vary simultaneously, to generate a population of cost estimates consistent with the real data. These models adjusted for confounders (see Appendix 4 for details). Next we took the mean of these estimates, and the distribution of the estimates indicated how confident we were in the mean. The modelling also allowed us to control for confounding factors (differences between hospitals that could affect their results) and to allow for the different duration of intervention for each hospital, due to the trial’s stepped-wedge design.

The effectiveness of the intervention in terms of different costs was estimated using a counterfactual approach: that is, we used a model fitted to the data to predict what would happen if all hospitals had implemented the ARK intervention simultaneously, compared to what would have happened if none of the hospitals had implemented the intervention using Monte Carlo simulation, assuming that coefficients were asymptotically multivariate normally distributed, using the modified model matrix, the coefficients, and the variance–covariance matrix of the fitted regression. These counterfactual analyses were evaluated from the date the first site implemented the intervention (September 2017) up to the date when the first site stopped providing data (end of June 2020), meaning that implicitly we assumed that effects – if present – would disappear after approximately 33 months. For 90-day mortality, we assumed that each patient not observed to die during the trial period lived the expected number of life-years for their age at admission, at the average age-related quality of life for those years based on McNamara et al.69

After estimating the intervention implementation costs and potential changes in total antibiotic and admission costs, as well as potential effects on quality-adjusted life-years (QALYs) through changes in 90-day mortality, we estimated the incremental net monetary benefit (INMB) of implementing the intervention from the NHS perspective. This analysis directly incorporates estimates from the regression models above, and hence implicitly assumes – without re-design of the intervention – no effect beyond 33 months. When estimating the INMB, cost and QALYs were discounted using a rate of 3.5%.

Findings

One hundred UK respondents (hospital prescribers) completed the choice experiment. Respondents were much more likely to continue antibiotics when discontinuation would ‘strongly conflict’ with local guidelines (average marginal effect (AME) on the probability of continuing +0.194, p < 0.001). Respondents were also more likely to continue antibiotics when presenting symptoms more clearly indicated antibiotics (AME of urinary tract infection symptoms +0.173 (p < 0.001) vs. unclear symptoms) and when patients had severe frailty/comorbidities (AME +0.101, p < 0.001). Respondents were less likely to continue antibiotics when under no external pressure to continue (AME −0.101, p < 0.001). Decisions were also influenced by the risks to patient health of continuing/discontinuing antibiotic treatment, but a higher risk from discontinuing affected decisions about twice as much as the same higher risk from continuing. Being a consultant was associated with being less likely to continue prescribing antibiotics. However, we did not find evidence for modification of the effects of the attributes on continuing prescribing by being a consultant versus other clinical grades (interaction p > 0.06 for all attribute levels).

For the main trial cost-effectiveness analysis, the average cost per hospital site of implementing the ARK intervention was £27,440 (SD £3792), although there was considerable variation between sites, with estimates ranging from £16,230 to £37,353. These costs included both the initial costs (such as training, kick-off meetings), as well as ongoing activities throughout the 33-month period (training and audit). The total intervention implementation cost across all 39 sites was estimated to be £1,070,147 (see Appendix 4, Table 9). The estimate was most sensitive to the assumptions regarding the number of hours spent on audit and feedback to clinical teams on their implementation (a central part of the ARK programme), the number of hours assumed for core team meetings, and the size of the core team.

Additional consults added a further £580,127 to the total cost, assuming that these continued at the same rate throughout the 33-month period considered in the cost-effectiveness analysis. This is a conservative assumption, as it is likely that practice would adjust with familiarity, so fewer additional consults would be needed to avoid missing the 72-hour review window.

If all ARK study sites had implemented the intervention at the same time as the first site implemented the intervention, and had been followed up for 33 months, this would have resulted in an estimated saving in total antibiotic costs of £3,635,492 (95% CI −£1,427,888 to £8,887,271), compared to the counterfactual situation where none of the sites had implemented the intervention.

In total, there were 4,765,836 admissions during the 33 months included in the analysis. Observed average costs per admission were £3111. Implementation of the intervention was estimated to provide savings of £124 (95% CI £101 to £145) per admission compared to not implementing the intervention, resulting in estimated total savings of £593,118,060 (95% CI £488,509,723 to £688,750,179) in admission costs across all 39 sites.

There was no strong evidence that implementation of the intervention led to an increase in mortality, with a non-significant estimated increase in the number of deaths of 5997 (95% CI −2369 to 15,012) during the 33 months following implementation of the intervention across all 39 hospitals (likely affected by incomplete adjustment for the impact of COVID-19, see above). This equates to a non-significant 1.1 additional deaths per hospital per week (95% CI −0.5 to 2.8), and a resulting non-significant estimated QALY loss of 43,125 (95% CI −15,867 to 104,796).

Combining these estimates, implementation of the intervention was estimated to reduce healthcare costs, but using a standard cost-effectiveness threshold of £20,000 per QALY, implementation of the intervention was not the optimal choice, as the savings did not outweigh the (statistically non-significant) reduction in QALYs (mean INMB: −£267M). The ARK intervention would therefore be the optimal choice if the reduction in antibiotic consumption led to future cost savings of £267M or greater, due to avoidance of consequences of antibiotic use, such as selection for resistance and associated costs.

However, under the assumption that the weak association between implementation of the intervention and 90-day mortality in the trial was due to a combination of chance (95% CI includes zero) and bias (e.g. the effect of the COVID-19 pandemic), implementation of the intervention would be clearly the optimal decision from a cost-effectiveness point of view, given the relatively low costs of implementing the intervention in all 39 hospitals (~£1M) compared to estimated reductions in antibiotic costs (~£3.6M) and admission costs (~£593M).

Key limitations

There were several limitations to our choice experiment. Firstly, as in the ARK-Hospital trial, the setting for this study was the NHS in England, and the factors driving early discontinuation of antibiotics may vary elsewhere. Secondly, reported actions in a survey may not reflect actual choices in clinical practice, though asking respondents to make trade-offs rather than state absolute preferences may partially mitigate this limitation. Thirdly, our sample size may be considered small, but choice experiments typically require smaller sample sizes than many other types of regression-based analyses (because comparisons are primarily made within respondents across different choices rather than between respondents, reducing variability); sample sizes of 100 or lower are not unusual. Fourthly, deciding which factors to include in the choice experiment inevitably required an element of judgement, and the factors we used to characterise the ‘review and revise’ decision were necessarily a simplification.

In terms of the cost-effectiveness analysis, the main limitations relate to the pragmatic assumptions that had to be made. First, we estimated the amount of time and resources consumed to implement the intervention. However, it is not entirely clear what the opportunity costs of the person-time invested are. For example, if individuals would otherwise spend time on less structured – potentially ineffective – antibiotic stewardship activities, there may not be much opportunity cost. If this was the case, we may have underestimated the cost-effectiveness of implementing the intervention.

Second, the assumptions made to group and cost admissions data may have introduced bias, especially if the data were not missing at random. In particular, the four HES variables describing admission and discharge characteristics were missing for over 500,000 episodes (around 5% of the total), including all of the Northern Ireland patients due to between-country differences in coding schemes. Notably, the admission method variable determines whether the episode is assigned a cost as an elective or non-elective procedure; our imputation of the modal code meant that all missing data would have been costed as non-elective, which could have biased cost upwards. A further important assumption was to impute the median cost for the ~1% episodes that remained ungrouped or where no associated cost was available. The ungrouped includes episodes where no valid primary diagnosis code was available; if these were missing for reasons related to the diagnosis, or to the timing (e.g. during the COVID-19 pandemic), this could introduce bias into the costing.

Third, the estimated average reduction in costs per admission is larger than anticipated a priori, and could indicate bias due to the profound effect the COVID-19 pandemic had on the types of patients being admitted, as well as potential length of stay of patients, including those that remained COVID-19 free during their admission. Although the magnitude of this bias likely varies between different outcomes considered, if we believe it is present for cost per admission, it likely also affects other outcomes, including estimated effects on mortality and antibiotic costs.

Fourth, we did not have data on antibiotics used in primary care for recurrences of infections; therefore, we may have underestimated resource use, which may have been slightly increased under the intervention implementation scenario. However, when focusing on more severe events such as hospital re-admissions, there was no evidence that the intervention had a negative impact.

Finally, our analysis does not explicitly account for the benefits due to reducing antibiotic use, including reduction in selection for resistance. Estimating the impact of marginal (one extra unit) changes in antibiotic consumption on future resistance is challenging.70,71 We have estimated the cost saving that would be required for the ARK intervention to be cost-effective, given the study data; these additional cost savings would be a summary of all health-economic consequences of antibiotic use not already explicitly captured by the health-economic model, including antibiotic resistance selection, and also consequences such as gastrointestinal side effects, and microbiome dysbiosis leading to increased susceptibility to colonisation by potentially pathogenic bacteria.72 Further research is needed to provide tools for decision-makers to determine the likelihood of such savings being realised.

Inter-relationship with other work packages

The design of the online survey, in terms of the selection of attributes likely to be important in ‘review and revise’ decisions, was informed by rich data from qualitative interviews with healthcare professionals in WP 3. Its findings highlight particular areas that need to be considered in implementing the ARK intervention (particularly the role of local guidelines) and in developing future interventions and policy to support continued reduction in antibiotic consumption through review of initial prescribing decisions.

The economic analysis used the same cleaned data set developed by the study statistician for the published main trial analysis as described in Work package 5: Inpatient intervention evaluation. The models used to estimate mortality and costs were also the same as those used in the primary analysis. The estimate of resource use to implement the intervention was based on secondary analysis of qualitative work carried out as part of the implementation process evaluation described in Work package 5: Inpatient intervention evaluation.

Patient and public involvement

The aim of PPI within the ARK-Hospital programme was to ensure that the research addressed the needs and concerns of acute/general medical inpatients receiving antibiotics in hospital, and of the wider public, and that its results were appropriately communicated, by providing input at all stages of the research, including co-design of patient-facing materials.

The grant proposal was developed in collaboration with patient representative and co-applicant Mr Clifford Gorton, as well as a wider patient and public group. Mr Gorton continued to participate in and provide significant insights for programme meetings throughout the research and also represented the public on the PSC. When the programme started in April 2016, we also invited additional members of the patient and public group involved in developing the ARK application to formally join the programme’s Community Advisory Board (CAB), which was chaired by Mr Gorton. Two kindly accepted and the CAB met 6-monthly through the programme until the main trial entered its follow-up phase (electronic data collection only). At this point, with less frequent updates the CAB became part of a broader PPI Steering Committee for the Modernising Medical Microbiology group at the University of Oxford (containing ~6–8 active members at any one time and meeting approximately 3 times a year) who continued to be regularly updated regarding progress with ARK.

The CAB critically reviewed the design of materials to be used to recruit patients into the qualitative study in WP 3, the planned recruitment process, and the patient leaflet that formed a key component of the intervention to which they made invaluable contributions. They also reviewed findings from interviews with patients and contributed to their interpretation. The CAB also reviewed the protocol for the feasibility, pilot and main trial. We developed a quarterly newsletter and provided this information to ‘trial participants’ [here hospitals and the (non-research) staff using the intervention] and CAB members.

As part of our intervention development, it became very clear that the myth ‘complete the antibiotic course to avoid resistance’ was highly prevalent among members of the public as well as healthcare professionals. We therefore wrote an opinion piece for the BMJ (Llewelyn et al.,73 freely available here74), coauthored with Mr Gorton, highlighting that this myth is not based on evidence and is actively unhelpful as we try to reduce antibiotic use. Although fairly widely misreported, this led to a large amount of public and media interest (altmetric score 3598), and a number of internet events with major journals highlighting issues around stopping antibiotics and the lack of evidence for most commonly prescribed course lengths. There have subsequently been two major funding calls from the NIHR covering research to address uncertainty about antibiotic course length: 20/109 shorter versus longer fixed-course antibiotic treatments and 23/99 interventions to reduce AMR.

We also kept the broader public informed of the research through a website with both healthcare professional and public facing content (no longer available to due ongoing maintenance costs).

Reflections on what was and was not successful

A major success of the programme was collaboration with a large number of NHS Trusts and hospital organisations across the UK to establish the ARK Champions network. Despite huge staffing challenges and ongoing day-to-day disruptions, they showed enormous energy, drive and creativity in finding ways to implement the ARK intervention. This group underpinned the successful delivery of the research and implementation of ARK especially in Wales and Northern Ireland, which has persisted beyond the research. Leadership from NHS consultants, rather than researchers alone, was essential for making sure firstly that the ARK intervention was rooted in clinical practice, and secondly could be turned into something that could be used in practice over the longer-term, rather than a research tool alone (further details below).

We also benefitted greatly from NIHR’s flexibility regarding no cost extensions. Specifically we had major challenges in obtaining data from NHS Digital, leading to several delays which we were able to accommodate only through NIHR’s support.

The COVID-19 pandemic caused major challenges to the final 6 months of data collection for the main trial, and also to subsequent sustainability with enormous other pressures on the health service. It meant unanticipated changes to the planned statistical analysis, and could not be adjusted for completely.

The potential for electronic health records to reduce the burden of data collection in clinical trials is an area of active research.75 However, the use of electronic health records also has important limitations, three in particular affecting the ARK programme: specifically how the intervention population was defined, how antibiotic use was assessed and how data were accessed.

• Firstly, the intervention targeted (adult) acute/general medical inpatients. These patients are managed through very different pathways in different organisations. Hence, part of the implementation toolkit involved mapping these patient flows and involving all relevant clinicians in training. However, identifying the relevant acute/general medical inpatients is not typically possible using specialty codes, requiring an approximation to use electronic health record data (see Appendix 2, Figure 5), and potential exposure misclassification.
• Each antibiotic has its own dose; WHO has proposed formulae for DDDs76 to standardise milligrams of different antibiotics into a unified unit reflecting a typical dose. DDDs are widely recommended for use in evaluating AMS interventions.77 One challenge is that reducing broad-spectrum antibiotics often involves replacing them with a combination of multiple narrower-spectrum antibiotics, that is one broad-spectrum DDD may be replaced with multiple DDDs of narrower-spectrum antibiotics, but this is probably better for future AMR. Very few organisations were able to provide individual patient-level antibiotic data from electronic prescribing systems, and therefore we were only able to analyse bulk-level DDDs in acute/general medical inpatients.
• The final challenge was getting the electronic data from individual participating organisations. Most hospitals do not have single unified systems for all their electronic patient data, meaning that antibiotic usage data (aggregate DDDs or individual level prescriptions), patient admissions and C. difficile tests were all kept on separate database systems from which the relevant data had to be extracted, combined together for the trial using patient identifiers and then de-identified by information technology departments before submission. This created significant challenges for many hospitals despite the programme providing substantial funding to each organisation for this.

Equality, diversity and inclusion

Infections affect individuals regardless of characteristics; however, those from the most deprived communities experience disproportionately greater rates of life-threatening infection78 and therefore are likely subject to greater rates of empiric antibiotic prescribing, as well as higher rates of comorbidities, increasing prescribing complexity. We deliberately aimed to involve hospital organisations from all four nations of the UK, and all geographical regions, ultimately recruiting a diverse range representing the breadth of healthcare provision into the main trial (see Appendix 2, Table 5). The relatively small number of units involved (39 hospital organisations) meant that we were not able to disaggregate these in analyses.

We were unable to purposively sample patients for our interview study, ultimately relying on a convenience sample of volunteers, but were able to achieve greater diversity in our interview studies with healthcare professionals. We deliberately considered the language used in all patient-facing and healthcare professional facing materials to ensure it reflected the concerns of different types of individuals. Our online choice experiment was advertised through the ARK forums and through word of mouth, so, while language was encouraging and inclusive, reflected those who wished to take part.

Our research team, oversight committees (PSC and DMC) and CAB were well balanced in terms of gender, including at all levels of seniority in the research team, but those of non-White ethnicity were under-represented. This reflected both those originally involved in formulating the research plan, and also applicants for post-doctoral positions. Junior members of the team were encouraged to take responsibility for parts of the project, attend training courses, present at conferences and write first drafts of manuscripts (see publications below) as development opportunities.

Limitations relating to the method or execution of the research

The main limitation of the programme was the COVID-19 pandemic. All key members of the research team were fully redeployed to pandemic management, due to our highly relevant expertise, which greatly affected the final stages of the programme and meant that some findings were not reported in peer-reviewed publications due to lack of time, funding or staffing (see Appendices 1, 3 and 4). Of note, all sites had fully implemented ARK by the time of the pandemic. In these hospitals, rather than focus on AMS, clinical efforts were devoted to managing COVID-19 patients, and behavioural interventions such as ARK were necessarily de-prioritised. Therefore, the impact of the pandemic was on our estimates of the long-term impact of the ARK intervention: we were likely not able to fully adjust for the impact of COVID-19 on mortality in the stepped-wedge cluster randomised trial, likely affecting our primary mortality and cost-effectiveness analysis.

As described in Work package 5: Inpatient intervention evaluation, classical analysis of stepped-wedge cluster randomised controlled trials is based on comparing those units randomised to control versus intervention in each time period (or ‘step’), averaging across time periods during which the intervention is rolled out to obtain an overall estimate of the intervention effect. Randomisation ensures unbiasedness of this classical stepped-wedge estimator in expectation – but not in one realisation of such in a single trial. Any kind of heterogeneity between randomised units, with regard to factors other than the intervention, or the effect of the intervention itself or underlying event rates varying over time, increases the chance of large imbalances across the steps, increasing variability and reducing the ability to detect intervention effects. Hospitals are extremely diverse in terms of their structures, and so, with a relatively small number of units, chance imbalances were considered likely were we to rely on the standard analysis above. This was compounded by the huge pressure on AMS over the period over which the trial ran (even before COVID-19), with financial penalties for failing to achieve specific targets, meaning that independently of the randomised intervention, hospitals were trying numerous other approaches to stewardship, leading to additional variability both between hospitals and over time. One solution would have been to increase the number of hospitals randomised, to reduce the impact of random variability, but this was not feasible within either the funding envelope or the healthcare system (we ended up including around one in four acute hospitals in the UK in the trial and approached nearly all of them). Given these constraints, the key question was how to reduce variability across hospitals – a crossover design is a well-recognised method to achieve this, leading to our choice of analysis method. This further allowed us to assess the longer term trends in rates after the intervention was implemented, which is not possible in a classical stepped-wedge analysis, but is of key importance to AMS, where we often expect the impact of interventions to increase as they become more closely embedded in routine practice. Although it was not a factor in our decision, which was pre-specified in the original protocol, exploiting the crossover design more easily accommodated the fact that only 24 of the 36 trusts who were eventually randomised had agreed to join the trial when it started.

Conclusions from the programme

The programme showed that combining qualitative and quantitative expertise across a multi-disciplinary team, adopting a theory-, evidence- and person-based approach,53,54 and using robust evaluation including feasibility, non-randomised internal pilot and randomised main trial phases, can both develop and test a highly complex behaviour change intervention designed to maximise the value of ‘review and revise’ 48–72 hours after initiation of typically broad-spectrum antibiotic treatment in acute/general medical inpatients.

Trial findings show the ARK intervention achieved safe, sustained reductions in hospital-level antibiotic use in adult acute/general medical inpatients. Though the antibiotic reductions were relatively modest (−4.8% per year), they may be clinically significant as the standard contract for acute hospitals in England only sought antibiotic reductions of 1% per year over the study period.79 The cluster-randomised approach allowed us to assess hospital-level effects and patient-level clinical outcomes while overcoming many of the study design weaknesses that have prevented the translation of previous research findings into clinical practice, particularly observational rather than experimental design and lack of reporting of clinical and microbiological outcomes.57,80 Our analysis method allowed us to estimate the impact of the intervention over the longer-term, which is not possible with classical stepped wedge analysis. Our findings are consistent with the latest Cochrane review evidence showing stewardship interventions can safely reduce antibiotic overuse,16 and the results of three much smaller trials of hospital stewardship interventions, which demonstrated the importance of education and audit and feedback to prescribers81,82 and intervention co-design with practitioners.83 The ARK tools are now freely available through the British Society for Antimicrobial Chemotherapy (BSAC)84 (further details under Implications for practice and lessons learnt).

The primary analysis of our coprimary mortality end point showed weak evidence for an immediate decline in mortality associated with the intervention (p = 0.079) and then subsequently for a small year-on-year increase (p = 0.060). First it is worth noting that many journals would explicitly ask that such results not be overinterpreted given the potential for type I errors. Second, there was no expectation that using fewer antibiotics could actually result in benefits for patients; therefore an overall null result (no immediate change, no long-term difference) associated with the intervention could be artificially reflected in a (false-positive) initial reduction followed by a (false-positive) subsequent small year-on-year increase (‘push–pull’ effect). Third, multiple findings from the trial indicate a causal link between the intervention and mortality is unlikely: specifically, the lack of association between the impact of the intervention on antibiotic use and mortality across individual hospitals, the lack of association between implementation fidelity and mortality and, most importantly, the lack of association between the implementation of ARK and mortality excluding admissions after the onset of the COVID-19 pandemic in March 2020 (well after all hospitals had implemented the intervention). Indeed, the fact that the COVID-19 pandemic started towards the end of the post implementation follow-up period in all hospitals is a more likely explanation. While we tried to adjust for the impact of COVID-19 on our primary mortality analyses, this was imperfect, with mortality impacts on individual hospitals varying in timing depending on when prevalence rose within different regions, inflating mortality rates at the end of the post implementation period. Excluding follow-up time after March 2020, there was no evidence of differences in mortality trends post versus pre implementation (p = 0.60) or any immediate effect (p = 0.36). No randomised controlled trials have shown increased mortality associated with shorter antibiotic courses: the vast majority have shown non-inferiority between shorter versus longer courses, or even benefits from shorter durations.30

This process evaluation provides encouraging evidence that ARK is likely to be acceptable and feasible for implementation in a wide variety of hospital contexts. Importantly for widespread uptake, the views of ARK expressed by staff were mainly enthusiastic and appreciative of its goals and methods. ARK was designed to be implemented flexibly and adapted to local constraints, and we found that there was indeed large variation in the ways in which it was implemented, the challenges to implementation and the solutions created by different hospital teams. Many of the facilitators of successful implementation of ARK are common to effective management of most hospital initiatives: energetic and committed leadership; good staff relationships and communication; adequate staff time.85 However, a major specific barrier to implementation of ARK was the use of drug charts or electronic prescribing systems that could not easily record the ARK ‘possible’ versus ‘probable’ categorisation of the reason for the antibiotic prescribing decision, and this is an important area for future research (see below). In addition, the restriction of the ARK implementation to AMU posed difficulties for applying the ARK process to patients who moved in and out of the unit, addressed by some sites applying ARK more widely across the hospital.

In terms of health economics, the best option is the most cost-effective strategy, that is the option with the most favourable mean INMB. While there is uncertainty in some areas such as the effect of the intervention on antibiotic expenditure, the conclusion would be that the optimal choice is to not implement the ARK intervention. However, this relies on believing that the statistically non-significant increases in mortality represent a true meaningful effect on the risk of death associated with the intervention. There are major challenges with this interpretation due to the possibility of residual confounding from the profound effects of the COVID-19 pandemic. However, if one believes the mortality findings reflect truly no effect on the risk of death, the optimal decision would be to implement the ARK intervention into NHS hospitals.

The ultimate goal of reducing antibiotic exposure through shorter durations in those who no longer need additional antibiotics at a 48- to 72-hour review is to reduce AMR, both in commensal flora in the individual patient being treated (which could lead to a resistant infection in that patient in future) and in the population as a whole through onward transmission. How such ‘resistance burden’ should be assessed however remains unclear. We found no evidence that evaluating the faecal resistome of < 1% of hospital inpatients using discarded samples submitted for C. difficile testing could be used to assess the impact of antibiotic practice on carried resistance at an hospital-level, despite substantial differences in antibiotic prescribing and CDI rates, suggesting that other sampling frameworks may be more suitable to determine whether organisational level differences in antibiotic use are associated with individual-level differences in burden of AMR carriage.

Recommendations for future research

The concerns about the ARK implementation process chiefly centred around the difficulties of embedding it in the hospital system. Since the ARK tools were designed, there have been important contextual changes in how AMS is conducted in the NHS. The clinicians who co-designed ARK tell us these are preventing its sustainable implementation and upscaling. In particular, electronic prescribing systems are replacing the paper charts which were in use when the ARK intervention was developed. These systems can create challenges for incorporating the ARK decision aid into the prescribing process, but also offer opportunities to facilitate sustainable implementation of the ARK audit and feedback tools, particularly if correct application of the ARK decision aid criteria can be included within e-prescribing systems. Solutions need to be created to implement ARK within both stand-alone and integrated electronic prescribing systems. Learning from ARK needs to be captured to support recommendations for AMS functionality more widely in electronic prescribing systems.

Hospitals will also need to ensure that all prescribers undertake the ARK training so that they understand how the criteria should be used and are convinced that they can be applied safely and effectively. The success of the ARK trial, which has been accompanied by national dissemination of the ARK intervention by BSAC (see below), should help to reassure those who are concerned about prescribing shorter courses of antibiotics. It may also be necessary to undertake further work to continue to inform and reassure all hospital staff about the safe and effective use of shorter antibiotic courses. Nevertheless updated national antibiotic prescribing policies and emerging evidence, including that from the ARK programme but also from an increasing number of trials investigating shorter versus longer antibiotic courses, should be used to update the ARK online tool and patient leaflet, also reflecting the contextual changes with e-prescribing.

Noting that some hospitals chose to manage anxieties about prescribing shorter courses by employing an alert recommending stopping antibiotics after a short course rather than a ‘hard stop’, a further topic for future investigation concerns the extent to which this adaptation may negatively impact antibiotic stop rates.

Given our incomplete understanding about the impact of shorter versus longer antibiotics on AMR gene burden, further studies investigating whether more detailed sampling, or pooled sampling, would be of benefit in detecting the impact of differential prescribing strategies on the ‘resistome’ should be explored. How to incorporate potential benefits due to reducing antibiotic use, including reduction in selection for resistance, into cost-effectiveness analysis remains unclear, given the low cost of most antibiotics and the fact that the major detrimental effects of resistance are likely to be in the future, and in patients other than the one being treated for an infection now. Estimating the impact of marginal changes in antibiotic consumption on future resistance is challenging,70,71 and further research is needed to understand how these ‘unknowns’ should be managed by decision-makers.

Finally, while a cluster-randomised stepped-wedge design was the only feasible mechanism of incorporating random allocation into our evaluation, given the limited number of hospital organisations in the UK, these still have recognised challenges. Secular trends and the influence of other external factors including policy change (or COVID-19) in individual hospitals are particularly pertinent, since the classical approach of assuming that randomisation will provide balance across measured and unmeasured confounders within each time step may not hold when the numbers of clusters being randomised is relatively small, and when there are numerous other policy changes and drivers in the background which different units will respond to differently. This is why we chose to analyse each hospital individually using an interrupted time series analysis, analogous to a crossover trial, and then pool these estimates using meta-analytic techniques. We relied on the randomisation to ensure that the dates of implementation were not influenced (confounded) by external factors. While our analysis might be criticised, the study design was experimental and we reported both clinical and microbiological outcomes, weaknesses identified in many studies in a previous review.57 Given that the mechanism of action is through behaviour change at the level of individual healthcare professionals, teams and organisations, further research could investigate whether there are aspects of how quality improvement is assessed, for example through ‘Plan, Do, Study, Act’ cycles86,87 responding to how organisations/individuals interact with tools in a rigorous, systematic, but iterative, manner, that could be valuable in evaluating organisational level behaviour change interventions, such as ARK. Another approach proposed by Tacconelli et al. in their editorial published alongside our main trial paper88 would be to establish a system-wide standardised data-collection platform, to support iterative evaluation of AMS interventions in terms of appropriateness of prescribing (i.e. matching the appropriateness of the prescribed antibiotic with the infection that prompted its use).

Implications for practice and lessons learnt

The key implication for practice is that acute hospitals should consider incorporating the ARK toolkit into their AMS activities, as part of efforts to safely decrease antibiotic overuse in acute/general medicine inpatients and reverse decades of rising antibiotic use in hospitals, leading to reductions in AMR over the longer term. The ARK intervention was developed for widespread application across the NHS, and was highly acceptable to clinicians and healthcare professionals, including many consultants with huge collective experience. COVID-19 has led to many challenges in antibiotic prescribing, but as its impact recedes, there is a unique opportunity to embed ARK in clinical practice.

The revised 2023 Department of Health and Social Care Start Smart Then Focus Antimicrobial Stewardship Toolkit for Inpatient Care Settings18 highlights ARK findings extensively and directs readers to the BSAC resources, promoting uptake.

However, the ARK toolkit was originally designed for evaluation within the research programme, and therefore it would not have been straightforward for hospitals to implement it ‘as is’. In collaboration with BSAC, we secured additional funding from Health Education England through the 2019 Innovation Education and Training Fund to develop ‘non-research’ versions of the ARK online learning and other resources for Trusts to continue to use after the implementation phase, and for new Trusts not randomised within the study to use. These are freely available at the BSAC Learning Hub89 together with the implementation resources. This enabled us to update the evidence base cited in the online learning and remove references to the ‘trial’ or ‘research’, but is otherwise unchanged. The new ARK resources website also includes a more user-friendly non-research focused forum with frequently asked questions (FAQs) based on previous forum postings in the trial. Importantly, it is Sharable Content Object Reference Model (SCORM)-compliant, meaning that individual Trusts can download the package and embed it within their local mandatory online training platforms, enabling them to track usage. It was launched in June 2019 and has been accessed by over 15,000 learners who have given it an average score of 8.3/10.

During 2020, the leading company providing ePrescribing systems in the NHS (WellSky) worked with the ARK-hospital Champions network and NHS Digital to incorporate the ARK-hospital decision aid into the antibiotic prescribing approach for their electronic medication prescribing system. Subsequently, WellSky was purchased by System C who reportedly supply 38% of NHS hospital electronic prescribing systems: their system ‘CareFlow’ has an antimicrobial prescribing module which from version 8.2 (now live) has included ARK functionality, specifically recording of indication using International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) codes and categorisation of the antibiotic prescription as developed within the ARK programme.

In terms of other facilitators and barriers, the largest potential barrier to safe early discontinuation of antibiotics in our online survey was early discontinuation ‘strongly conflicting’ with local guidelines. This barrier may be easier to address than others. Specifically, local guidelines that make didactic statements stipulating duration, for example ‘duration 7 days’, are not likely to be appropriate for many patients and may discourage safe discontinuation at review. Revising guidelines to be less prescriptive about duration, and instead more conditional on patient factors and treatment response, could help hospital prescribers safely discontinue antibiotics if diagnostic information suggests they are no longer needed.