Cluster randomised clinical trial process evaluation: methods and results

Introduction

The POPPI cluster RCT evaluated a preventive, complex (multicomponent) psychological intervention, delivered across multiple critical care units. Trials evaluating interventions that have several component parts can pose methodological challenges in understanding the effect of the intervention overall, as well as separating the effect of one component from another. In addition, when delivered across multiple units (such as in this case), they are susceptible to variation in how the intervention is implemented and delivered, potentially affecting patient outcomes and producing varying degrees of treatment effect between units. The complexity of the cluster RCT necessitated an evaluation of these processes. In the context of this process evaluation, we evaluated the three core elements (of which there are four components) of the POPPI intervention:

• Element 1: creating a therapeutic environment in critical care.
  • POPPI online training.
  • Translating knowledge from the online training into practice.
• Element 2: three stress support sessions for patients identified as acutely stressed.
  • Identifying acutely stressed patients with the IPAT, and delivering stress support sessions.
• Element 3: relaxation and recovery programme for patients identified as acutely stressed.
  • Take-home booklet and DVD offered to all patients identified as acutely stressed.

The first element of the intervention, creating a therapeutic environment in critical care, facilitated primarily by an online training course, was a cluster (site)-level intervention for all critical care staff to participate in; thus, all patients would be exposed to its potential effects. The second and third elements (stress support sessions and relaxation and recovery programme) were patient-level interventions, delivered to patients identified as acutely stressed (indicated by a score of ≥ 7 points on the IPAT). The timeline for exposure to the POPPI intervention is outlined in Figure 15.

FIGURE 15 Timeline for exposure to the POPPI intervention. R&R, relaxation and recovery programme; SSS, stress support session(s). (Figure not reproduced here; see the original article.)

Aims

The aims of the process evaluation were to:

1. assess the uptake and level of adherence to the POPPI intervention
2. understand factors affecting the successful delivery of the POPPI intervention and cluster RCT
3. uncover the relationship between the variation in the POPPI intervention delivery and the POPPI cluster RCT primary outcome.

The process evaluation also provides increased understanding of the barriers to and facilitators of the wider cluster RCT delivery and augments understanding of any observed differences between units. This chapter outlines the methods of how this evaluation was undertaken and presents the results of aims one and two. The results of aim three are reported in Chapter 7.

Methods

Objectives

Guided by Steckler and Linnan’s86 recommendations for six key process evaluation components (fidelity, dose, reach, implementation, context and recruitment), the following objectives were developed to address the aims of the process evaluation.

Aim 1: to assess the uptake and level of adherence to the POPPI intervention

1. To what extent is the POPPI intervention delivered as intended (fidelity)?
2. How much of the intended POPPI intervention is delivered and received (dose)?
3. What proportion of intended recipients actually participated in the POPPI intervention (reach)?

Aim 2: to understand factors affecting the successful delivery of the POPPI intervention and cluster RCT

1. Which aspects of the unit culture influenced how the POPPI intervention and cluster RCT were implemented (context)?
2. What procedures were used to ensure that patients were recruited to the cluster RCT (recruitment)?

Aim 3: to uncover the relationship between the variation in POPPI intervention delivery and the cluster RCT primary outcome

1. Is there a difference in the primary outcome from lowest to highest overall intervention adherence (implementation: an overall weighted composite score summing fidelity, dose and reach)?

Logic model

The logic model for the POPPI intervention is a diagrammatical representation of the theory of how the intervention is intended to work (see Figure 1). In the context of the cluster RCT, the logic model illustrated the main elements of the POPPI intervention, how they were intended to interact to produce change, the expected outcomes, and the resources and mechanisms needed to ensure successful delivery, implementation and, therefore, outcomes. Constructing a logic model is useful in that it may uncover weaknesses in the assumptions, identify at which points clinicians may have conflicting understandings of the intended intervention, or help evaluators to think critically about potential unintended consequences.87 A semistructured interview guide and structured observational data collection form were designed to gather information that could be used to test the assumptions of the logic model, alongside routinely collected cluster RCT data.

Data sources

The process evaluation adopted a mixed-methods approach drawing on data collected from a number of sources (Table 15).

The process evaluation involved field observation and discussion of the cluster RCT and the POPPI intervention with unit staff, so it was important that the researcher was sufficiently independent to minimise the introduction of bias into the evaluation, and for it to remain credible.87 This independence ensured that the external researchers would neither view the intervention too positively, nor be unduly critical. The relationship between the process evaluation team and the wider trial management team was defined at the planning stage.

The evaluation was conducted in three phases, spanning the duration of the cluster RCT recruitment period from the beginning to the end. It was necessary to collect baseline data from the beginning of the trial (phase I) as contextual factors have the potential to influence how an intervention is implemented and may act as barriers to or facilitators of intervention implementation and delivery.88 By understanding baseline contextual differences between units, it would be possible to elucidate how these may have influenced trial implementation and delivery. Collecting these data during the baseline period also ensured that it was not influenced by treatment allocation [as individual sites were randomised in their second month of recruitment (during the baseline period) – after the phase I site visits].

Collecting during-trial (phase II) data, after the intervention sites had transitioned to delivering the intervention, was key to understanding how the POPPI intervention and cluster RCT were being implemented and delivered within sites, and in identifying emerging themes regarding barriers and facilitators that affected this. Collecting during-trial data was also a fundamental step in the development of both the interview guides and the analytical framework to be used at the end of the cluster RCT.

It was essential that sites were then followed up at the end of the cluster RCT (phase III) to fully understand the uptake and the level of adherence to the POPPI intervention. This allowed for in-depth exploration with those working closely within the cluster RCT, to explore how and why the intervention was implemented, delivered and sustained over time (or not), and to uncover the relationship between the variation in intervention delivery and the cluster RCT primary outcome. Routinely collected cluster RCT data were collected throughout the entire trial trajectory and integrated with the qualitative data throughout.

Phase I

At the start of the patient recruitment period, during the baseline period, contextual data were gathered using individual interviews and field observation at all 24 sites. This information detailed the baseline psychological support for patients and environmental factors that may have affected a patient’s psychological well-being in the critical care unit.

Participants

Participants were the site PIs at all 24 sites. The PIs were chosen as they had primary responsibility for local management of the POPPI intervention and had in-depth knowledge of relevant local critical care unit practice and protocols. The PIs accompanied the researcher onto the unit to undertake a period of field observation.

Recruitment and consent

The PIs were informed of the process evaluation format by the trial management team at the site initiation visits (see Chapter 4 for details). PIs were told that the researcher would contact them directly to arrange the phase I visit. The researcher contacted all PIs, via e-mail, to invite them to participate in individual interviews at a time convenient to them and to request access to the unit to undertake the period of field observation. PIs were advised to check local NHS trust policies and inform the researcher of any pre-visit access checks/approvals that were mandated.

Individual interview data collection

The researcher used a semistructured interview guide to conduct individual face-to-face interviews with the PIs at each site (see Appendix 2 for the semistructured interview guide). Interviews lasted 45–60 minutes, during which all information was recorded in field notes. The data collected encompassed five categories:

1. Family care: what was usual practice in the unit regarding family/visitors? Were they included in patient care, and was there any information/support available to them?
2. Sleep: what steps were taken to manage patients’ sleep in the unit? Were there any non-pharmacological interventions or sleep protocols in use?
3. Pain, agitation, delirium, sedation (PADS): was there a PADS protocol in use? How was delirium managed, and what training did staff get?
4. Psychological support: were patients psychologically assessed on the unit, and was there any psychological support or follow-up available?
5. Management of serious incidents: how were patients protected from witnessing serious incidents on the unit, and were they debriefed afterwards?

Field observation data collection

A period of observation was conducted on each unit, which included observing part of a morning ward round. This lasted 45–60 minutes and observations were recorded using the data collection form, structured to capture relevant observable information (see Appendix 3 for a field observation data collection form). The observational data collected during Phase I encompassed five categories:

1. Patient orientation: what was being done on the unit to orientate patients to night and day?
2. Unit layout: were bed spaces cramped or cluttered, and was there anything of visual interest to patients?
3. Communication: how did staff address/include patients, and were patients encouraged to ask questions?
4. Unit noise: what was the noise level like on the unit (alarms, telephones, staff), and what efforts had been made to reduce noise?
5. Relaxation and distraction: was there any stimulation or distraction for patients (TV, radio, books, complementary therapies)?

Data analysis

Both the individual interview data and observational data collected from each unit were factual information collected in prespecified categories, so these were organised and tabulated for later cross-examination of baseline similarities/differences between sites. These data were used to understand how baseline differences between units affected the cluster RCT and the POPPI intervention implementation and delivery.

Phase II

To understand how the POPPI intervention and cluster RCT were being implemented in sites and to begin to identify barriers to and facilitators of successful delivery, during-trial exploratory telephone interviews were conducted for all intervention group sites. In addition, routinely collected cluster RCT data pertaining to patient recruitment and intervention delivery were summarised by the trial management team in the form of individual unit profiles and sent to the researcher to help understand how sites were engaging with the cluster RCT and the POPPI intervention. During-trial exploratory telephone interviews were also conducted for all usual care group sites, to explore barriers to and facilitators of cluster RCT implementation and delivery and to discover whether or not control sites remained engaged with the trial.

Telephone interviews

Participants

The trial management team identified a study driver at each site. The study drivers were PIs, POPPI nurses or research nurses, and had been involved in trial implementation, drove the project, and were involved in the POPPI intervention (at intervention group sites) and/or cluster RCT delivery. The study driver was chosen as the telephone interview participant, as they had a wealth of information about how the cluster RCT was accepted and worked in practice.

Recruitment

The researcher was provided with a list of study drivers for all units. At this stage of the trial, the researcher was familiar with staff across all units, both through meeting in person at the phase I visits and through observing monthly routine cluster RCT site teleconferences. The study drivers were contacted by the researcher via e-mail, inviting them to take part in a one-to-one telephone interview at a time convenient to them. They were provided with an overview of the topics that would be discussed in the interview, to enable them to reflect and make note of any relevant areas for discussion.

Data collection

Semistructured interview guides (see Appendices 4 and 5) were used to elicit information on emerging issues and themes, guided by the unit profiles and the cluster RCT logic model (see Figure 1). The interviews explored how the cluster RCT and the POPPI intervention had been received in the units, issues surrounding patient recruitment, how the cluster RCT and the POPPI intervention fitted into a given context (including fidelity) and the impact that it had on ‘routine’ clinical workload. Field notes were taken during the telephone interviews. The semistructured interview data provided qualitative information on the fidelity of intervention delivery across sites and explored the effect of contextual/external factors on implementation and identified barriers to and facilitators of adherence to the individual elements of the POPPI intervention.

During the telephone interviews, staff were encouraged to elaborate on both positive and negative aspects of the cluster RCT to date. This allowed key issues to emerge and enabled the preliminary identification of barriers to and facilitators of the intervention components. Cluster RCT implementation and delivery was explored during interviews with control site staff, including understanding any changes in engagement after sites were randomised.

Data analysis

A content analysis of the interview field notes was undertaken. Content analysis was chosen as it is a straightforward method of analysis, used to determine the presence of concepts within text by breaking that text down into manageable categories. In accordance with Krippendorf’s89 guidance, this began with familiarisation of the notes from the telephone interviews and making note of any relevant or interesting information in the margins. All information in the margins was then categorised, with descriptions given for each category. Categories were examined to see if there was any overlap, or if they could be merged into major and minor themes. The process was applied to each set of interview notes, after which the original notes were rechecked to ensure that all information had been categorised. The content analysis was double-checked by another independent member of the research team, after which the key themes were condensed and merged, resulting in a concise set of themes and subthemes. Once analysed, these data were used to inform the development of the final Phase III interview schedule, guiding appropriate areas for further exploration.

Routine cluster randomised clinical trial data

The trial management team provided the researcher with individual site profiles indicating performance and engagement within the cluster RCT prior to the conduct of telephone interviews. These included free-text comments detailing any emerging issues at the site, as well as considering team involvement and communication. For intervention group sites, the unit profiles also provided information to evaluate the dose and reach of the POPPI intervention. The unit profiles comprised the following:

• screening and recruitment figures for the baseline period and transition and intervention periods (when relevant)
• monthly patient recruitment figures
• uptake of the POPPI intervention online training over time
• number of patients assessed with the IPAT
• number of stress support sessions received by patients identified as acutely stressed
• number of patients receiving the booklet and/or DVD (as part of the relaxation and recovery programme).

Phase III

Towards the end of the cluster RCT patient recruitment, interviews were conducted at site visits with both PIs and POPPI nurses at all intervention group sites. Convenience samples of staff were also informally interviewed at all intervention sites. Routinely collected cluster RCT data pertaining to patient recruitment and intervention delivery were also used to explore indicators of unit engagement including uptake of the online training, and stress support sessions delivered. In addition, interviews were conducted at a purposively selected sample of usual care group sites, the purpose of which was to examine whether or not these units did anything different from baseline in terms of psychological support provided to patients.

Individual interviews

Participants

At intervention group sites, the participants were site PIs and POPPI nurses (which included all study drivers), chosen because of their close involvement in the cluster RCT and the POPPI intervention implementation and delivery. A convenience sample of critical care unit staff was also chosen on the day of the site visit, with whom the POPPI intervention online training was discussed. These staff must have had completed the POPPI intervention online training and be willing to discuss how it affected their practice. At the usual care group sites, interviews were conducted with PIs and research nurse(s) involved in cluster RCT screening and recruitment.

Recruitment

As the key unit contact for the cluster RCT, the study drivers were contacted by the researcher, via e-mail, and invited, along with the other PIs and POPPI nurses, to participate in a final comprehensive site visit by the researcher. The Phase III visit had been discussed during the Phase II telephone interviews, so the study driver was willing and had agreed to co-ordinate this visit. They liaised with critical care unit staff and agreed a date for the researcher to visit to conduct the final phase of interviews.

Data collection

A standardised interview schedule was devised using information collated from Phase I and Phase II data for uniform use across all intervention group sites (see Appendix 4). The interviews took place before the participants and the researchers knew the result of the cluster RCT, to avoid this knowledge influencing views about the cluster RCT or the POPPI intervention.

Any change in practice from baseline at usual care sites had the potential to close the difference between the effect of the intervention and usual care, and produce a null result;90 for this reason, a purposive sample of usual care group sites were visited in Phase III. The sampling of usual care sites was informed by the Phase II telephone interview data. Sites were selected on the basis of suggestions of a change in practice of psychological support from baseline. It was also anticipated that there may be a negative effect of usual care site allocation on enthusiasm and attitudes towards the study. This had the potential to affect the trial delivery at control sites, and clinician willingness to screen and recruit patients into the cluster RCT. A modified version of the standardised interview schedule was used in four purposively sampled usual care group sites (see Appendix 5). These interviews explored clinician experiences and understandings of the cluster RCT in the context of their critical care unit.

At all sites, one-to-one interviews were conducted with unit PIs, POPPI nurses (where relevant) and research nurses. All interviews were digitally recorded, audio files were stored on a password-protected drive, and audio files were erased after transcription. Audio files were transcribed in full, removing any identifiable information, and saved in Microsoft Word format. The transcripts were stored on a password-protected computer and each participant had a unique identification number.

Nine themes were explored in detail at the intervention group sites using the semistructured interview guide (see Appendix 6) aimed at understanding:

• the fidelity, dose, and reach of the POPPI intervention, and using this to inform the overall implementation score
• which aspects of the unit culture influenced how the POPPI intervention and cluster RCT were implemented
• what procedures units used to ensure that patients were recruited into the cluster RCT.

Seven themes were explored in detail at the usual care group sites using the semistructured interview guide (see Appendix 7) aimed at understanding:

• which aspects of unit culture influenced how the cluster RCT was implemented
• what procedures units used to ensure that patients were recruited into the cluster RCT.

A convenience sample of unit staff (2 or 3 members of the multidisciplinary team) was selected on the day of the intervention unit visit at each unit, with whom the POPPI online training was discussed. These interviews were informal and field notes were taken, with questioning focused on:

1. identifying barriers to and facilitators of undertaking the POPPI online training
2. opinions of the POPPI online training content
3. whether or not the POPPI online training was something that staff felt that they had had the opportunity/was feasible to use in practice.

Data analysis

Interview data were thematically analysed91 using a seven-stage framework approach92 that allowed simultaneous analysis across themes and cases. The approach is a matrix-based method for analysing data that includes familiarisation with the data, indexing the data and creating summaries from the indexed data. An important feature of the approach is that it allows themes or concepts identified a priori to be specified as indexing categories from the outset (deductive analytical framework) and to be combined with other themes or concepts that emerge de novo by subjecting the data to inductive analysis (revising the data to see if anything has been missed).

A deductive approach was taken as existing ideas and themes were identified in the phase II interviews, so the content of the phase III interviews was broadly anticipated. The deductively developed data extraction framework (see Appendix 8) was applied to all intervention group site interview transcripts. A modified version was used for usual care group site interview transcripts (see Appendix 9). The data extraction frameworks were developed by the lead process evaluation researcher and reviewed by the wider research team. They were developed to systematically extract all data pertaining to the process evaluation aims.

To ensure confirmability and trustworthiness, the framework was piloted on a random sample of five interview transcripts. A second independent member of the research team extracted data from the same five transcripts. Once this stage was complete, the researchers convened to discuss the functionality of the framework in extracting all of the data. It was evident that some interview data remained unallocated to the framework, so the thematic framework was reviewed as it was applied to the data and revised a further two times until all data could be fitted in the thematic domains of the framework. Finally, the themes were mapped, and interpretation of those themes reviewed by the two researchers, to construct overall explanations of the data. This approach is less focused on producing new theory, but more about shaping and exploring existing ideas.

Routine cluster randomised clinical trial data

Routinely collected cluster RCT data were examined at the end of phase III, to evaluate the dose and reach of each intervention element. These data comprised:

• the POPPI online training uptake
• stress support session delivery
• use of the relaxation and recovery programme
• delivery of debriefing and support visit/calls with the POPPI nurses.

Implementation score

The inclusion of ‘implementation’ as a composite score allows for the grading of intervention group sites according to their degree of overall intervention delivery. This experimental approach was adopted to explore the nature of the relationship between the variation in the POPPI intervention delivery and the cluster RCT primary outcome. Implementation grading was developed using a combination of fidelity, dose and reach. Phase III qualitative interview data were distilled into scores, as were the phase III routine cluster RCT data, with the aim of grading sites into lowest, moderate or highest adherers.

Qualitative interview data

Interview data were considered for assessing the fidelity of intervention elements one (creating a therapeutic environment in critical care) and two (three stress support sessions for patients identified as acutely stressed). The fidelity to each of these elements was scored based on the unit fulfilling criteria to achieve one of the following scores:

• 3 = Full adherence: element delivered as intended; full effort made to follow recommendations/guidance.
• 2 = Mostly adhering: element predominantly delivered as intended; decent effort made to follow recommendations/guidance.
• 1 = Some adherence: element often not delivered as intended; mediocre effort made to follow recommendations/guidance.
• 0 = Low adherence: element rarely delivered as intended; poor/no effort made to follow recommendations/guidance.

To ensure inter-rater reliability, two researchers independently scored the same site, converting interview data into scores. They discussed any discordance between their scoring of the data and reached consensus on how to proceed with the scoring of a further two sites. Again, both researchers scored the interview data independently, and agreed on the allocated scores. Both researchers scored the remaining nine sites together. Sites that had the same score for any given element were cross-checked to ensure that the same standard had been applied to the scoring process throughout.

Routine cluster randomised clinical trial data

Routinely collected cluster RCT data were also examined at the end of phase III, to evaluate and score the dose and reach of the POPPI intervention, and comprised the following:

• Percentage of clinical critical care unit staff completing the POPPI online training.
• Time taken to reach 80% uptake (target) of the POPPI online training.
• Percentage of patients receiving two or more stress support sessions.
• Percentage of patients who scored ≥ 7 on the IPAT (i.e. were acutely stressed) who received no sessions.
• Percentage of patients using the tablet computer as part of the stress support sessions.
• Percentage of patients receiving the relaxation and recovery programme (DVD and/or booklet) to take home.

Integrating mixed-methods data

The interview data were combined with the routine cluster RCT data so that all three elements (four components) of the POPPI intervention had a score. The scores address fidelity, dose and reach of the POPPI intervention (Table 16). Component scores were combined giving units an overall composite score for implementation and ranking. Table 16 shows how the mixed-methods data were pooled to score (1) adherence to each intervention element and (2) overall intervention adherence. To ensure that each element of the intervention was weighted to account for its anticipated importance, the POPPI investigators were asked to independently weight the components using a total of 15 points each, representing probable overall importance.

The results from aim three are reported in Chapter 7 as part of the subgroup analyses.

Results

Visits and interviews

Phase I visits were conducted within the first month of the recruitment period at all 24 participating sites. The visits were conducted during the baseline period, in which all sites were delivering usual care and had not yet been randomised. Interviews were conducted at each site with one or both site PIs. In the instance that both PIs were present, they were interviewed together. Interviews lasted 45–60 minutes, and a total of 18 PIs participated in the phase I interviews. A period of field observation was spent on all 24 units during the site visit, ranging from 45 to 60 minutes. All field observation was conducted in the presence of one site PI. On one unit, the ward round could not be observed, but this was discussed in detail with the medical PI. Phase I confirmed that all 24 intervention and control units had no routine psychological assessment or formal support for patients. The bedside nurses provided emotional support for patients on an ad hoc basis. Follow-up clinics were in place at 15 of the 24 sites, but these were used in inconsistent ways across the sites and targeted a minority of patients. Criteria for invitation to attend the follow-up clinics varied, and included a requirement to have been mechanically ventilated for anything from 24 to 96 hours; variable lengths of intensive care unit (ICU) stay regardless of mechanical ventilation duration; only patients receiving level 3 care; and one unit where there were no set criteria (so the outreach team chose who to invite).

Phase II telephone interviews were conducted in month 9 at all 24 sites, by which time intervention units had been delivering the intervention for 3 months post-transition period (usual care groups continued to deliver usual care). Telephone interviews were conducted with one study driver at each of the sites and ranged in duration from 20 to 30 minutes. In total, 24 study drivers participated in the phase II interviews. The themes and subthemes from the phase II content analysis (see Appendix 10) were used to inform the development of the final phase III interview schedules.

Phase III site visits were conducted in month 11 of the intervention period, at all 12 intervention sites and at a purposively selected sample of four control sites. Individual interviews were conducted at each site, comprising at least one PI and at least two POPPI nurses at intervention sites, and the research nurse(s) and at least one PI at control sites. In total, 44 staff participated in the phase III intervention group interviews. Convenience samples of unit staff were also informally interviewed during the site visit, with whom the online training was discussed. Two or three members of the multidisciplinary team participated in these interviews at each site and a total of 28 informal interviews were conducted for this purpose across the intervention group sites. The professional groups involved in these interviews incorporated physiotherapists, occupational therapists, doctors and nurses. At the four control sites, 11 staff participated in the phase III interviews. A total of 83 interviews were conducted during phase III.

Aims one and two: dose, reach and fidelity (implementation); context; and recruitment

Element 1: creating a therapeutic environment in critical care

Dose

To facilitate the creation of a therapeutic environment in critical care, the POPPI online training course was rolled out at intervention group sites at the start of the transition period. The POPPI online training was to be completed by all clinical critical care staff, against a target of 80% of staff completing the course.

Expressed as a percentage of staff completing the POPPI online training out of all staff working in the units in the transition month, the median uptake among critical care staff at intervention group sites (n = 12) at the end of the transition period was 58% (IQR 49–69%) and all sites achieved the 80% minimum target within 4 months (Table 17).

Participants at all intervention group sites were asked at interview what they considered to be the main facilitators of and barriers to meeting the 80% target for online training. There was consensus among all sites that relieving staff nurses from the bedside to complete the training was the main expediter of achieving the target. This was possible at 11 intervention group sites where at least one POPPI nurse had clinical critical care experience and could take over patient care to allow staff nurses time off the unit to complete the training:

. . . I had to be more hands-on and actually go out and say ‘I’ll look after your patient while you go and do the training’.

At the twelfth unit, this was not used as a strategy, as the POPPI nurses were constrained by their clinical roles. The POPPI nurses also discussed how the use of mobile or tablet computers enabled staff nurses to complete the training during quieter periods of their shift while remaining at the bedside; this was particularly convenient during nightshifts.

A conflicting/high clinical workload was reported as the main barrier to undertaking the POPPI online training. Participants felt that the existing burden of other mandatory training for clinical staff impeded the time available to complete the non-mandatory POPPI online training:

I think it’s just because it was an extra thing to do. You come into work, you’ve got a busy patient, people are nagging you to go spend half an hour on a computer to do an online training, and they’ve already got all the mandatory online training things.

Reach

All intervention group sites continued to enumerate staff members throughout the intervention period to ensure that new staff (e.g. rotations of junior doctors) received the POPPI online training and that leavers were no longer included in the monthly figures. Figure 16 shows the median uptake of the POPPI online training across intervention group sites from the transition period until the end of the intervention period. Figure 17 shows the monthly uptake for each of the intervention group sites (n = 12) across the same period.

FIGURE 16 Median POPPI online training uptake across intervention group sites (n = 12) from the transition month until the end of intervention period. (Figure not reproduced here; see the original article.)

FIGURE 17 Monthly POPPI online training uptake at each intervention group site (n = 12) from the transition month until the end of intervention period. Reproduced with permission from JAMA 2019;321(7):665–75.71 Copyright © 2019 American Medical Association. All rights reserved. (Figure not reproduced here; see the original article.)

During the interviews, the POPPI nurses discussed promoting awareness of the online training by displaying posters and bunting throughout the unit, and using incentives such as badges, pens and cakes to encourage participation. Several units incorporated the online training course into new staff induction programmes as an effective way to ensure high coverage despite staff turnover. The POPPI nurses reported that generic group e-mails were often unsuccessful in encouraging online training completion, but that taking an individualised approach and e-mailing staff (for whom the training was uncompleted) proved effective in improving the reach of this component.

A small number of participants reported poor information technology (IT) access while at work, meaning that they had to be willing to complete the training in their own time, usually at home. This also obstructed access to POPPI-related e-mails. However, the POPPI nurses stated that the majority of staff were amenable to this strategy and it did not appear to have any bearing on uptake rates. For those who did have IT access in work, computers occasionally lacked audio, preventing users from interacting with one element of the training (patient experience and communication videos); however, transcripts were available.

Fidelity

To assess the fidelity to the creation of a therapeutic environment it was important to explore in participant interviews whether or not those who completed the online training applied learning from it in their clinical practice, and what factors affected this.

Education by the POPPI nurses and/or local PIs emerged as key in translating the online training into practice. These individuals acted as study champions and used various mediums to reinforce the online training messages and encourage their application in clinical practice. There was evidence of study champions at half of the intervention sites. These study champions conducted informal bedside teaching with staff nurses, distributed newsletters and disseminated key messages and reminders throughout the unit, using posters and noticeboards. Participants in these units reported widespread multidisciplinary engagement and enthusiasm among the team. Suggestion boxes were used at one unit as a way of encouraging staff to contribute ideas for improving the psychological care of patients, enforcing the learning from the POPPI online training. To ensure that the online training was applied in practice, another unit initiated workstreams, developed to ensure implementation of strategies to address delirium, sedation, sleep optimisation and noise reduction:

. . . this sort of intervention is not something you can do as a one-off. It’s a sustained process, isn’t it, of trying to keep reminding people about, not just the trial, but the whole belief that’s behind keeping quiet at night, turning alarms off, let people sleep and so on and so forth.

The remaining half of intervention sites had no evidence of a study champion. In the absence of study champions, participants reported less engagement from a small number of staff nurses, manifested through resistance to the online training and an unwillingness to use the principles to improve practice. There appeared to be a passive approach to education at these units and educational efforts were limited to posters and signage in the units. Participants from these units reported a perception of the online training as a ‘tick-box’ exercise, which needed to be completed but was not particularly influential on their current practice. Some of the POPPI nurses stated that a small number of senior staff nurses felt insulted at being made to undertake training that taught them basic patient care, and this contributed towards an attitude of hostility. Despite this apparent lack of leadership and engagement at these units, the online training uptake target was reached on time:

It was ages ago that I did it, but it was fine and everyone’s done it . . . I mean, I don’t know how much impact it has but, yeah, we’ve all done it . . . You do it and . . . then you move on.

In all 12 intervention units, participants reported that the mechanistic elements were easier to implement than those elements that challenged long-standing unit culture. During the period of field observation, it was noted that units had readily instated clocks, sleep aids and soft-close bins (where they had been absent). Patient orientation, optimisation of sleep, and noise reduction were the main elements targeted for improvement among all units. However, all units also reported a very small number of staff who were resistant to any practice change in the unit, and this was not confined to the POPPI intervention. For example, large ward-rounds held at the end of the patient bed (conflicting one of the recommendations from the POPPI online training on communication) and regular multiple disturbances to the patient during the night (conflicting the training on sleep optimisation) were reported as practices that were most challenging to improve:

. . . I think you can’t underestimate how much nurses like routine and how difficult it is to get cultural change.

I think that’s an inbred thing here, always done for years, is ‘turn the lights down in the afternoon for people to have a little nap . . . The patients need rest, they need a nap.’ You try and explain the evidence, but no.’

. . . they still wander around in little groups and there are some consultants who will do it in the night-time as well, do their ward rounds at 3 o’clock in the morning. I don’t think it’s changed their practice at all.

Participants at two units reported difficulty translating the online training into practice because of the physical confines of the unit. In particular, they felt that the size or layout of the unit constrained their ability to implement the recommended environmental changes, especially in relation to noise reduction and finding suitable locations to place clocks and whiteboards within patients’ lines of sight.

Element 2: three stress support sessions for patients identified as acutely stressed

Dose and reach

A total of 340 patients were recruited at intervention group sites during the intervention period. Of these, 27 patients provided consent to receive only a follow-up questionnaire, resulting in 313 patients eligible for assessment with the IPAT (and subsequent receipt of the stress support sessions, if applicable).

All 313 (100%) patients consenting for an IPAT assessment were assessed. Of these, 199 patients (63.6%) were assessed as being acutely stressed and at a high risk of psychological morbidity (i.e. an IPAT score of ≥ 7 points) and, therefore, eligible to receive the three stress support sessions. The median IPAT score was 8 points (IQR 4–13 points).

Of the 199 patients assessed as acutely stressed, 127 patients (63.8%) received all three stress support sessions, 33 (16.6%) received two, 21 (10.6%) received one and 18 (9.0%) received none (Figure 18). During interview, both POPPI and research nurses reported that the IPAT screening tool was simple and user friendly. As evidence of this, 100% of potential patients were screened with the IPAT to determine eligibility for the intervention. With regards to duration of the session, the median time of each session was as follows: session 1, 30 minutes (IQR 30–35 minutes); session 2, 30 minutes (IQR 30–40 minutes); and session 3, 30 minutes (IQR 25–32.5 minutes).

FIGURE 18 Number of stress support sessions received by patients (n = 199). Reproduced with permission from JAMA 2019;321(7):665–75.71 Copyright © 2019 American Medical Association. All rights reserved. (Figure not reproduced here; see the original article.)

During the interviews, the POPPI nurses were asked to reflect on what strategies they had used to ensure that patients received all three sessions. All POPPI nurses reported that it was easiest to deliver the stress support sessions if they were able to adopt a flexible approach to their work. This included an ability to work outside paid hours and swap shifts to accommodate session delivery, which most were willing to do:

. . . I came in on my own time just to do them, just so that I knew that they were done and that I could definitely come at the time I was saying, so there’s not been an issue really. And I get my time back; work has been good and said that if you come in on your own time you can have it back. So it’s not been an issue really.

The POPPI nurses reported that sometimes they had minimal notice of a patient being discharged from hospital, which led to patients missing some of the sessions. When possible, the POPPI nurses combined the content of multiple sessions in an attempt to deliver the intervention as fully as possible. Conflicting clinical workload was also reported as a barrier to the delivery of the stress support sessions. It was not always feasible to have a POPPI nurse available consistently, and when they were absent or on annual leave, the burden of stress support delivery fell to the remaining POPPI nurses who struggled to meet the demands of intervention delivery. This meant that they needed to work outside their normal work hours to fulfil intervention delivery with enrolled patients. Despite the demands on the POPPI nurses to accommodate session delivery, there was only one unit at which they discussed an unwillingness to do this (because of personal commitments and a perception that the trial procedures had placed unrealistic expectations on them).

At 11 units, participants reported minimal or no obstruction to the release of the POPPI nurses from their clinical duties to deliver the stress support sessions. They stated a belief that the intervention was viewed as worthwhile within the units, and staff assigned importance to intervention delivery being fulfilled. At the twelfth unit, nurse managers obstructed the release of the POPPI nurses from their clinical duties to deliver the stress support sessions. These POPPI nurses explained that their ability to deliver the sessions was partly dependent on the priorities of the senior nurses in charge on any given shift, and that they would deliver sessions outside clinical hours to remedy this. It was ventured that a combination of unit busyness and a lack of belief in the intervention as worthwhile were contributing factors to this obstruction.

Despite some challenges faced by the POPPI nurses, there were only four occasions when sessions were missed specifically owing to unavailability of a POPPI nurse (Table 18). This is evidence of the level of commitment and flexibility that the POPPI nurses applied to ensure that the intervention was delivered as fully as possible.

Unavoidably, there were several patient-centred barriers to this component of the intervention, at which patients declined some or all of the sessions. Participants were asked in interview to elaborate on reasons why patients declined to partake. The POPPI nurses reported that patients were often tired or unfit to participate in the sessions, and they wondered whether or not it was too early in a patient’s recovery trajectory to expect them to engage in such an intense intervention. If sessions were being delivered outside the critical care unit, the POPPI nurses sometimes found that the patient was unavailable at the arranged time. This was out of their control because patients may have had visitors or been receiving clinical interventions (e.g. scans, x-rays). Nonetheless, it was problematic to the POPPI nurses who were often working outside their normal hours to accommodate session delivery.

The personal action plan was developed for 121 out of 127 (95.3%) patients who received all three stress support sessions. These patients were re-assessed with the STAI-6 measure after the third stress support session (note that the STAI-6 was assessed in all patients, at both intervention group and control group sites, at the point of consent).

A total of 115 out of 127 (90.6%) patients who received all three stress support sessions were re-assessed with the STAI-6 prior to hospital discharge. Table 19 shows the comparison of the baseline with the post-stress support session 3 STAI-6 score for these 115 patients.

In session one, patients were given a tablet computer containing a relaxation and recovery app. Of the 181 patients who received stress support session 1, 171 (94.8%) were given the tablet computer containing the relaxation and recovery app. One additional patient declined to receive the stress support sessions, but did accept the tablet computer to use the relaxation and recovery app. All POPPI nurses reported at interview that the main reasons why patients did not receive the tablet computer were an imminent discharge from hospital or they declined it.

Of patients receiving both stress support session 1 and the tablet computer, 131 (76.6%) reported use of the tablet computer to their POPPI nurse during the course of the sessions. The cluster RCT case report form captured reported reasons for why patients did not use the tablet computer; these were that patients preferred not to use it, were too ill, had a hearing impairment, lacked technical capability or found it difficult to use, were too tired, preferred TV or were discharged from hospital. During the interviews, the POPPI nurses were asked to expand on the main factors that they felt acted as barriers to and facilitators of the relaxation and recovery programme. They reported that sight problems, muscle weakness, poor concentration and problems with dexterity were the main barriers. They also described what appeared to be a generational barrier, with elderly patients being less willing to use the tablet (compared with relatively younger patients).

Family involvement was reported to be the main facilitator of the use of the relaxation and recovery programme. Families were able to encourage use of the tablet computer and the programme. In addition, they could assist in overcoming barriers associated with weakness, sight and dexterity:

. . . her whole family has got involved in it. They’re all listening to the DVDs and relaxation, and you know, she’s part of that. I heard her talking to her family and she was repeating what I had said to her. So it was good for the fact that I felt that she’d obviously taken it in.

Fidelity

To determine whether or not the stress support sessions were delivered as intended, it was planned that sessions would be audio-recorded and analysed by a member of the research team. However, the POPPI nurses were met with refusal from patients from the outset of the intervention period; subsequently, it appeared that they stopped approaching patients for consent to audio-record sessions. This meant that no sessions had their content analysed, and the fidelity of the stress support sessions is essentially an ‘unknown quantity’. What follows is a narrative based solely on what was discussed during interviews with the POPPI nurses at all 12 sites. There appeared to be an inclination to discuss the challenges, despite the researcher encouraging discussion of positive aspects of this component. Findings, therefore, cannot be extrapolated across the unit for the entire intervention period but reflect specific incidents and challenges that occurred.

Units were responsible for appointing their own POPPI nurses, and participants reported that an appropriate choice of POPPI nurse was key to successful intervention delivery. Their clinical roles varied, but there was consensus that senior/more experienced staff nurses were most suited to this role. Half of all POPPI nurses had senior clinical roles comprising nurse consultant, outreach practitioner and sister/deputy sister, with only one unit having three staff nurses in the POPPI nurse role. Some units selected non-clinical nurses for this role, namely research nurses, because it was felt that they could be more flexible in the time required to deliver the intervention. Research and outreach nurses in the POPPI nurse role were less acquainted with patients, having not been involved in their clinical care. At two units, research and outreach nurses reported finding it difficult to build rapport and trust with patients and ventured whether or not this would have been easier had the patient been known to them in a clinical nurse–patient capacity prior to intervention delivery.

All POPPI nurses reported pre-session preparation time as an important factor to facilitate smooth session delivery, including familiarisation with patient medical notes and the content of previous sessions. Most POPPI nurses discussed feeling that unrealistically short hours had been allocated by the trial for the delivery of this element when preparation time and post-session duties were factored in:

I take 10 minutes, sit with my file, go back over what needs to be achieved in session 1, session 2 or session 3 . . . So I have exactly what I need in here to deliver a session, and I’ll sit with what I need in front of me. So I’ve got it there. I can glance down and make sure that I’ve picked up on every point that I need to and know it’s always there from my previous session so that I can write up my notes with my next session in mind. I haven’t come away from a session and thought ‘I didn’t say this’, or, ‘oh no, I did say that’.

Each stress support session lasted for a median of 30 minutes (Figure 19). Some POPPI nurses reported difficulty in adhering to the stress support session guide with some patients difficult to keep on track, as patients often wanted to talk about their problems that were not related to the critical care unit. Some found it challenging to turn the conversation back to issues related to critical care, particularly when they felt that patients were opening up to them:

. . . you’ve got your, I think, very well-presented checklist of what you need to do in each session, [X] is very passionate that we tick all those boxes, and you’ve got this and this, and it’s really hard to try and make it mix . . . I can only describe it as sometimes, whilst you’re chatting, you’re almost trying to . . . fight with is the wrong wording, but almost fight with that patient to get them on track to the checklist . . .

FIGURE 19 Duration of each stress support session. (a) Session 1; (b) session 2; (c) session 3; and (d) total. (Figure not reproduced here; see the original article.)

The POPPI nurses stated that the personal action plan component of stress support session 3 was difficult to co-design with a certain subgroup of patients, namely those individuals who had particularly complex or life-changing injuries. They perceived this patient subgroup as difficult to engage with making long-term plans outside their hospital stay, and that these patients could, understandably, set only short-term goals while in the very early stage of their recovery trajectory.

For the 181 patients receiving stress support session 1, 72 (39.8%) received this session in the critical care unit. This proportion decreased with delivery of sessions 2 and 3, with the majority delivered in hospital wards, as patients came closer to hospital discharge (Table 20). The POPPI nurses at one-third of units also reported a lack of privacy in the unit as a barrier to patients discussing sensitive or emotive topics during the sessions.

Debriefing and support for the POPPI nurses

Following the 3-day training course on delivering stress support sessions, nurses were to receive debriefing and support, consisting of one site visit by their allocated POPPI trainer, and three individual telephone calls with the same trainer. The visit was timed to be held after all POPPI nurses at a site had carried out sessions with at least one patient each. The individual calls were due to start after each nurse’s first patient, and at 2-month intervals thereafter.

Table 21 shows the delivery of debriefing and support visit and calls coded as green, blue and light green. Three sites were coded with three ‘greens’, while one site had three ‘light greens’. The other eight had varying combinations. Overall, of 48 cells in the table, most were ‘green’ (23 cells), followed by blue (15 cells) then light green (10 cells). Looking at the columns, visits were mostly coded ‘green’. Receipt of calls declined across the period, from call 1 to call 3. See Appendix 11 for an overview of the trainer’s perspective of the sessions.

During the interviews, all but one of the POPPI nurses unfavourably reported the debriefing and support. Keeping patients on track with the session guide during the sessions sometimes posed a challenge, and the POPPI nurses raised this concern during the debriefing and support visits/calls. Despite seeking advice on how to deal with this (and to deliver sessions with as much fidelity as possible), the POPPI nurses recurrently reported being rigidly told to steer patients back towards the guide. Many did not feel that they had the skills to do this capably and would have benefited from more detailed support in this area:

. . . she’ll say ‘so you didn’t write down the action plan then?’ And I’ll go ‘no, I didn’t’, and I . . . I can sort of see, from her eyes, she’s thinking ‘well no, you should have done that and next time you should do that.’

In addition, the POPPI nurses reported feeling an element of ‘judgement’ from the debriefing and support team, in that they were being checked up on and disapproved of when they reported an inability to deliver sessions per protocol:

I said to [X] ‘I’ve had this patient who I felt like I was just patronising them and they didn’t really take it in’, and she said ‘that’s your stressful thought’, and I was like ‘oh my God, [X] is analysing me; she’s delivering me a stress support session and I’m talking to her about one I’ve done for somebody else!’

Overall, the POPPI nurses described feeling that if they had been able to discuss stress support session delivery less apprehensively with the debriefing and support team, they would have been more willing to disclose problems that arose in sessions, potentially addressing the barriers to successful delivery that presented in subsequent patients:

There was never any guidance anywhere, any shape or form if things didn’t go as per normal session.

. . . there were never any guidance of what we do if we get the session interrupted or the patient stops the session. How do you then move on? Do you stick a bit on or do you just say ‘right, we’ve missed that bit’, because it has a massive impact upon the outcomes of what we’re doing.

Element 3: relaxation and recovery programme for patients identified as acutely stressed

Reach

During stress support session two, patients were given a self-help booklet and a DVD for use after the sessions and then at home. In session 3, nurses were to co-design a ‘personal action plan’ with the patient and record this in the booklet.

For the 160 patients who received stress support session 2, most [n = 153 (95.6%)] received both the DVD and the booklet to use after the sessions and to take home with them. A small number [n = 6 (3.8%)] of patients did not receive the relaxation and recovery DVD but took the self-help booklet, and one patient received neither the DVD nor the booklet. A personal action plan was co-designed between the POPPI nurse and the patient for 121 (95.3%) of the 127 patients who received all three sessions.

Staff attitudes and unit culture

Study champions

Active study champions were key for positively affecting multidisciplinary team engagement and enthusiasm. At units with study champions, no division in levels of interest among the team and an overall willingness of staff to engage with the study were reported. As evidence of this, the POPPI nurses reported little or no resistance from their colleagues to completing the online training or to be released from clinical duties to deliver stress support sessions. Participants at these units viewed the intervention as worthwhile and of benefit to the psychological well-being of patients. On the contrary, in one unit, the nurse PI/POPPI nurse had a very poor attitude towards the study and disagreed with some of the core principles of the intervention. There was very poor engagement at this unit from both consultants and senior nurses/sisters, and it is possible that this negativity was contributed to by the attitude of the nurse PI. It may also have directly affected the delivery of the intervention, because the nurse PI reported omitting what they deemed the ‘airy-fairy’ elements of the stress support sessions.

Nursing staff

During the interviews, participants commonly reported viewing the POPPI intervention as a nurse-led intervention, and something that nurses felt that they could contribute to in a meaningful way. Nurses were, therefore, predominantly enthused across all intervention sites.

Medical staff

In units where the medical PIs were engaged and took a pro-active role in recruitment and promoting study awareness, this was cascaded down to junior doctors, who the POPPI nurses reported to be enthusiastic and receptive to POPPI-related training and teaching.

In units where the medical PI was disengaged or had minimal involvement with the study, the nurse PI carried the workload. During the interviews, the POPPI nurses at these units reported negative attitudes from both senior and more junior doctors, who were resistant to undertaking the online training and hostile towards implementing the POPPI training recommendations in practice. It may be postulated that this attitude from medical PIs in the units was directly conveyed to their consultant colleagues as well as more junior doctors. The POPPI nurses attributed the medical disengagement to the fact that nurses took ownership of the trial, and that medics viewed it as a nurse-led intervention.

It was reported at most units that, in general, younger, less experienced staff from all professional groups are more receptive to any type of practice change in the units, whereas more senior staff appear to be more resistant to training and education.

Control sites

Participants at all units reported that the trial had been well received and that, prior to control allocation, staff were enthusiastic and keen to be involved in the study with high interest in the role of the POPPI nurse. Across all four units, participants described a culture of heightened awareness of psychological support for patients, and of clinicians who had specific interests in psychological care and delirium. It was also unanimously reported that on being allocated to the control arm, there was widespread disappointment in units. During the interviews, the effect of this sense of disappointment was explored with participants. They all reported that, despite initial disappointment, there was no effect on enthusiasm in the units and the study ran smoothly, conducted by research staff ‘in the background’. Interestingly, during the interviews many participants discussed a sense of relief at being allocated to the control arm, as they had anticipated a high workload associated with delivering the intervention:

Disappointed and relieved at the same time . . . So I was relieved that we didn’t need to do that, but equally, at the same time, disappointed, because obviously you want to give a treatment or an intervention to patients too, because I can only imagine – I would hope that it would make peoples’ experience and lives better. So yeah, a bit of both really, but it’s what it is, so that’s fine. Somebody’s got to be the control group.

Research nurses reported the use of the STAI-6 (a brief questionnaire which assesses anxiety) with patients who were not receiving any subsequent psychological support as difficult at times:

. . . you bring up these feelings for them, which can be quite upsetting for them, because it’s probably the first thing they’ve been asked about, that, really. I mean, on the unit. It’s just hard to sort of walk away after you’ve brought out all these feelings for them. You give a bit of reassurance in terms of contacting their GP if, when they’re discharged, they still have a lot of these feelings, and just give them just general advice and reassurance. Tell them it’s a normal thing.

There did not appear to be anything that happened at the sample of four control sites that suggested any change in psychological support from baseline.

Recruitment factors

Intervention sites

Research nurses were primarily responsible for screening patients for recruitment into the cluster RCT, but most worked weekdays only and there was poor or no evening or weekend cover for screening. During the interviews, research nurses reported swapping shifts when possible to ensure that someone was on duty to avoid eligible patients being missed. Two units used WhatsApp Messenger (WhatsApp Inc., Mountain View CA, USA) groups to communicate among the team regarding potential eligible patients in an attempt to maximise recruitment. All research nurses reported that, by involving family members right from the beginning of the recruitment process, patients seemed more willing to be involved. Families acted as an additional channel of communication with the patient, and research nurses believed that families encouraged participation because they could see the effect that the critical care stay was having on the patient’s psychological well-being.

All research nurses were in agreement that giving patients sufficient time to read and digest the study information, and discuss it with their families, improved the recruitment process. They allowed patients to ask questions before enrolment and were willing to visit the patient multiple times until they were fully informed prior to consent. Research nurses discussed recognising potential patients early, who may become eligible in the near future, and making themselves familiar to the patient. Some POPPI nurses were directly involved in the patient’s clinical care, whereas others who were not would visit the patient for an informal chat to become acquainted with the patient. PIs would sometimes visit the patient to make themselves known to them, and informally discuss the study. If such approaches had been made, patients appeared to be more receptive when the time came to approach them for consent:

I’m this woman who has come to say ‘come tell me all your secrets, those dark nightmares that you’re having.’ . . . what I’ve found is that by recognising the patient early, before the 48 hours, and bringing yourself into the picture, makes you more acceptable . . . so by the time I actually go in and wanting to tell them more about POPPI, they already know me, and that seems to work well for me, very receptive after that.

At two units, participants reported recognising that some patients associated participation in the study with a stigma of being mentally ill, and this perception appeared to dissuade their participation:

That seemed to be the main thing, the fear of the stigma of being crazy in their records and their GP knowing about it.

One POPPI nurse described how this happened to her with a patient, but that recognising this meant that she adapted the way in which she approached subsequent patients, ensuring that this did not happen again:

I used the word counselling, which I shouldn’t have, because as soon as I said that she was like . . . and then I’ve never done that since . . . And she was like ‘no, I don’t want counselling. I don’t need it.’ I couldn’t backtrack then because I had said that word.

The use of lay language was also important in helping patients and families understand the study design. The POPPI nurses discussed avoiding the use of words such as ‘counselling’ or ‘mental health’ as they felt these carried negative connotations for patients and were more likely to increase refusal rates.

Despite measures put in place to promote recruitment, identifying patients in the critical care unit who had capacity to discuss the study before discharge was a significant challenge of the study. In the critical care units, patients were often quickly discharged to a ward as soon as they became medically fit. This short time frame to approach eligible patients frequently presented as a challenge to research nurses, as many reported during interview that patients were not capable of understanding and retaining sufficient information to give informed consent at this stage:

. . . they’re still quite muddled, and you can talk to them one day and say, ‘oh I’ll come back and speak to you later about it.’ Introduce the topic, come back later, ‘hello, do you remember we had a chat?’ And they go ‘no!’. . . . I think sometimes when they first come out of sedation or whatever, that there’s so much that they have to take on board and I’m not entirely sure that they retain the information we give them.

Control sites

Participants reported using the same strategies as intervention sites to promote recruitment. These consisted of ensuring good coverage of staff able to consent patients, promoting communication between the research team and involving families in the process. One unit took a further step in attempting to minimise missing eligible patients by training two staff nurses in Good Clinical Practice in addition to the research team, which ensured 24/7 coverage for recruitment. In keeping with the findings at intervention sites, research nurses also reported some difficulty in finding patients who had capacity to consent or digest study information while in the critical care unit:

. . . they’re still a bit too muddled to formally consent, and actually the window of opportunity to get these people, we’ve found a bit hard.

Aim three: relation between intervention delivery and primary outcome

Implementation grading for the subgroup analysis

Based on the data in the unit-level summaries, a detailed set of criteria was developed and applied to the interview data to allow the robust scoring of elements 1 (creating a therapeutic environment in critical care) and 2 (IPAT assessments and stress support sessions) from qualitative data to a quantitative score. Tables 22 and 23 show the criteria for scoring fidelity to elements 1 and 2.

Table 24 shows the POPPI intervention component scores for each of the intervention group sites (anonymised) along with the weighted composite scores. A detailed breakdown of these scores per intervention group site can be found in Report Supplementary Material 7. Reweighting did not change the ranking of intervention group sites but it did split those that had tied. At this stage, sites were split into tertiles, as specified in the a priori statistical analysis plan, according to overall intervention adherence:

• 3 = highest adherence
• 2 = moderate adherence
• 1 = lowest adherence.

The results of the process evaluation subgroup analysis of implementation grade and cluster RCT primary outcome are reported in Chapter 7.

Summary of findings

High levels of uptake and adherence to each of the three elements of the POPPI intervention were seen during the intervention period at intervention sites. The units’ staff and cultural factors influenced their ability to fully implement the intervention, with variation between sites observed across each element.

By the end of the transition period, 971 out of 1669 critical care staff had completed the online training, equating to a median percentage of staff completing the POPPI online training of 58% (IQR 49–69%), with all ICUs achieving 80% (i.e. the prespecified minimum target) by intervention period month 3. Local initiatives to translate the online training into practice included optimisation of sleep (e.g. through sleep packs, night-time lighting and clustering of care), reduction of noise (e.g. through soft-close bins and minimisation of alarm and telephone noise), improved patient orientation (e.g. through clocks, staff–patient interaction and whiteboards) and increased family involvement. Some intervention sites found it challenging to change long-standing practices and/or were restricted by the physical environmental limitations of the unit.

During the intervention period at intervention group sites, 340 patients were recruited, of whom 313 consented to receive an IPAT assessment. All 313 were assessed; 199 of these patients were identified as acutely stressed and were eligible to receive stress support sessions. A total of 127 patients (63.8%) received all three, 33 (16.6%) received two, 21 (10.6%) received one and 18 (9.0%) received none. Of the patients who received session 1, 171 were given a tablet computer containing the relax and recover app to use between sessions and 131 (76.6%) reported use of it to their POPPI nurse. Most patients who received at least two sessions were given both the DVD and the booklet to take home. There was variation in delivery of stress support sessions across intervention sites: facilitators included the ability to work flexibly and pre-session preparation by the POPPI nurses, whereas barriers included unanticipated discharge of patients and conflicting clinical workload of the POPPI nurses.

Engagement of unit staff and a positive unit culture were key to ensuring effective implementation of both the online training and the stress support sessions. Positive study champions were key to this engagement of both nursing and medical staff. As a nurse-led intervention, nurses felt that they could contribute in a meaningful way and were predominantly enthused across all intervention sites. Engagement of medical staff was very dependent on the engagement of the medical PI. Younger, less experienced staff from all professional groups were more receptive to any type of practice change in the units, whereas more senior staff appeared to be more resistant to training and education.

With regards to control sites, there was initial disappointment at not being randomised to be intervention sites, but there did not appear to be any suggestion of any change in psychological support from baseline.

Recruitment processes were similar for intervention and control sites. Research nurses were primarily responsible for screening patients for recruitment into the cluster RCT, but most worked weekdays only and there was only some evening or weekend cover for screening. All research nurses reported that by involving family members right from the beginning of the recruitment process, patients seemed more willing to be involved. It was reported that, occasionally, participants recognised that some patients associated participation in the study with a stigma of being mentally ill, and this perception appeared to dissuade their participation (in the intervention sites during intervention period only). The use of lay language and avoidance of words such as ‘counselling’ or ‘mental health’ was also important. As patients were often quickly discharged to a ward as soon as they became medically fit, nurses reported that this short time frame to approach eligible patients presented a challenge to research nurses, as patients were often not capable of understanding and retaining sufficient information.

Considerations to maximise intervention implementation

As variation in implementation of the complex intervention was clearly observed, five factors that need to be considered when implementing this intervention in a unit have been identified. These are as follows.

Resources

To maximise the quality and fully deliver this intervention in an NHS setting, adequate and dedicated resources are required. With regards to the online training, to train all staff would require computer access in the unit, and the allocation of time in paid work hours for undertaking the training. This is likely to increase staff willingness to participate in such an initiative. In turn, mandating the online training for all staff would help to ensure compliance. For delivery of the stress support sessions, the POPPI nurses would also require protected time. Ideally, this would be outlined formally, as part of their clinical role, with dedicated time allocated. Owing to the flow of patients, this would need to be on a flexible basis to accommodate the patient-centred barriers that may be present and should not be subject to over-rule by nurse managers who may not prioritise psychological care in the unit. Although the POPPI nurses demonstrated a willingness to work outside their clinical hours and on days off to deliver the sessions for the research study, this is not sustainable in routine clinical practice and they need to be given protected time to fulfil the role.

Unit culture

It is a challenge to change long-standing practices in units. The POPPI nurses and clinicians discussed the resistance to change that is encountered from staff, and how all units have at least a small number of staff who are difficult to engage in any new practice. Measurable changes, such as noise levels and sleep optimisation, were more readily adopted in units, whereas other cultural practices (such as large ward rounds conducted at the end of patients’ beds) were viewed as the way things had always been done and were more difficult to change.

Environmental limitations of the unit may be very important in implementation. In very open-plan small units, staff found it difficult to even address the more simple elements such as reducing noise levels and orientating patients with whiteboards or clocks. Moving forward to future implementation in the NHS setting, it is clear that some units would be able to adapt to the intervention more fully than others, but this should not be equated with staff resistance to compliance with recommendations.

The selection of the POPPI nurses

The POPPI nurses who worked clinically in the units were in the unique position of being able to free ward staff from the unit who needed to undertake their training. The POPPI nurses were also often more familiar with recruited patients, so found it easier to build rapport with both patients and their families. However, these nurses struggled to balance their clinical duties with the demands of intervention delivery and frequently found themselves delivering stress support sessions in their own time. The POPPI nurses who worked in non-clinical roles (i.e. research nurses) were more flexible in their work commitments, and so were able to prioritise tasks to ensure that they could deliver stress support sessions at suitable times. However, they were not able to free staff from the bedside to undertake training because they did not work clinically in the unit and were not previously familiar with the patients with whom they delivered sessions to. The POPPI nurses who were research nurses also found it more difficult to connect with patients and families. Ideally, this role would be appointed to a dedicated core team of nurses in the unit who took sole responsibility for it, to overcome conflicting clinical workload barriers. Discussions with the POPPI nurses revealed that many agreed that this role would be best suited to nurses who had an existing interest in psychological care and were involved in the unit’s follow-up clinics.

Active education

A dedicated practice educator or a POPPI nurse should take responsibility for ongoing education in units, acting as intervention champions, which would help engage staff and maintain enthusiasm. This is particularly important when introducing a new intervention into routine clinical practice, and education could form part of induction programmes for new staff. As was found in units that integrated the online training into inductions, this is an effective way to ensure coverage of training programmes for units with high staff turnover. Champions are also key for engaging the multidisciplinary team.

POPPI nurse training and support

The POPPI nurses would benefit from further, more in-depth training to equip them to deal with the wide range of patient experiences that they are likely to face. They commonly discussed feeling out of their depth when patients were difficult to keep on track with the session guide and reported that there was a lack of guidance on dealing with these circumstances. More comprehensive training would equip them with the skill set to deal with patients with complex needs, and the confidence to deliver stress support sessions adapted to each individual patient’s needs. More comprehensive debriefing and support should be offered to the POPPI nurses to allow them to discuss their experiences without judgement, and to receive guidance on overcoming barriers with future patients. This may be better provided locally to allow for a more face-to-face approach.