Implementation outcomes and iterative implementation plans: qualitative data

This chapter presents the final outcome of the analysis of the qualitative data collected across the six participating NHS trusts. Tables 15–22 summarise the factors identified in each trust against the four core constructs of NPT. We describe successive iterations of implementation plans following each wave of data collection, then draw out common findings and important differences relevant to understanding the delivery of REACT. Where appropriate, we draw on quantitative data.

Narrative summaries of the context, key findings against each NPT component, the lead researcher’s reflective summary and feedback from final trust analysis days for each trust are presented in Appendices 10–15 (which include Tables 37–42 describing participants in each trust). Figures illustrating the common barriers and ways to enhance implementation of REACT revealed by these data are in Appendix 16 (see Figures 21 and 22).

Trust summaries and iterative implementation plans

Trust responses to each iteration of the implementation plan are summarised against NPT’s four core constructs:

1. coherence – the sense-making work that people do individually and collectively when faced with operationalising a set of practices
2. cognitive participation – the relational work that people do to build and sustain a community of practice around a new technology or intervention
3. collective action – the operational that work people do to enact a set of practices
4. reflexive monitoring – the appraisal work that people do to assess and understand how the new intervention affects them and others.

Implementation plan version 1

Challenges for this first iteration were to deliver REACT in all six trusts, train staff by providing an online manual followed by face-to-face training, and appoint an IMPART lead as PI in each trust, as described in Chapter 4.

Wave 1 outcomes

Table 15 shows uptake of REACT by staff and relatives in wave 1 trusts. Table 16 shows qualitative outcomes in response to the four NPT core constructs.

Staff in both trusts in wave 1 could clearly see the potential value of an online toolkit for relatives and for NHS staff:

[Groups] can be very responsive and warm and everything but they’re also yeah you have to get here for 6 o’clock on alternate Tuesdays . . . [A website is] much more kind of accessible and people can read stuff in their own speed.

One potential benefit is you can very easily offer I suppose everyone with an internet connection really the same information . . . consistently across the board . . . And hopefully if it’s used then you can get used to working with a family that have a certain knowledge base which I think would be good.

Some of our newer staff will really benefit from reading that information and knowing what information is useful to share. Particularly when you’re young, you lack some of the life experience like you know I did.

Yeah I think it fits in in terms of reaching the family interventions target and as I said before I think it fits in with the BFT [behavioural family therapy] as well. So it kind of hits a couple of the targets that we’ve got to do I think.

However, as Table 16 showed, staff in Woods were under huge pressure with high caseloads and low morale. With REACT seen as a research project and without financial incentives, the team never really signed up. Implementing anything in this team would have been a challenge. Agreement to join the IMPART study was driven by the trust’s R&D agenda:

Trust leadership are very keen on research, although I don’t think that they adequately communicate that enthusiasm to people on the ground . . . I think had you given [managers] the choice they would probably have said well we . . . support research as an important thing but we don’t really feel able to participate at the moment.

They’re not necessarily the things that I would clinically say are the most important, but they’ve got money attached to them . . . Sadly they are often the priorities.

There was no evidence of REACT being embedded in any care pathways or service structures, or of any team managers directing staff to offer REACT. Staff themselves did not find REACT easy to integrate into their daily practice, or the website easy to use:

People just forget it . . . When you’re doing an assessment, when you’re meeting a family member, you have a thousand and one things on your mind that you’d like to assess and write down and stuff and it’s difficult to remember the spiel.

You load up the dashboard, you first get this screen which is kind of sort of meaningless . . . It’s hard to know what does what, it’s kind of trial and error.

Moor Trust covers a rural area where staff had much smaller caseloads (estimated around 15) but distances to visit service users were great. EIP staff were embedded in general community teams, making formal and peer training challenging. Key individuals were responsible for driving REACT, but beyond these individuals there was low engagement. Trust IT systems ran old web browsers that did not support the encryption algorithm used by REACT (which was configured to require a reasonable strength of encryption and reject known insecure algorithms), and the firewalls blocked videos embedded in the website and hosted on YouTube (YouTube, LLC, San Bruno, CA, USA). Staff did not have access to mobile devices to show relatives REACT in their homes, leading to requests for a paper version.

Concerns were also expressed about the possible impact of REACT on relationships with relatives as a result of more information:

If I was a relative and I went on and I see, ‘Oh we’ve got the right to ask for this and this and this’ – the trust doesn’t provide everything it’s advertising on REACT . . . For example it talks about art therapies . . . As it stands, our service users can’t access that internally. So that’s an issue . . . A lot of the things that are linked to EI we’re still in the very early stages of thinking about how, and that’s not a Moor Trust exclusive issue.

As in Woods, REACT was not seen as priority because it was not linked to financial targets, and identification of REACT as being part of the research agenda led staff to look to the research team to engage staff and relatives to use REACT:

You have come today and it has prompted me, reminded me for example about the REACT toolkit.

I think you really need to just grab them in the team meeting one day and get them logged on, because I really don’t believe they’ll prioritise logging on.

In both trusts, lack of feedback or engagement from relatives very quickly demotivated staff. In Woods, early abandonment was triggered by the REACT supporter leaving and the role not being reallocated:

I think it just feels a bit kind of stagnant.

Yeah I’ve given it to a couple of patients, I’ve had a look at it myself, I think it looks really good. But the patients that I’ve given it to they haven’t accessed it . . . I’ve asked them you know a couple of times and it’s, ‘Oh I’ve not got round to it yet’.

Implementation plan version 2

Implementation plan version 2 aimed to address some of the issues above. Factors identified in wave 1 fed into workshops in wave 2 trusts where IPv2 was designed and delivered. It attempted to increase staff engagement by making REACT more visible and user friendly. Some elements of IPv2 were further developed in IPv3: these are shown in italic text in Table 17.

REACT NHS booklets

These booklets, shown in Figure 16, were produced for staff to show to relatives to support their introduction to REACT. Designed to be an attractive description of REACT and its potential benefits for both staff and relatives, the booklets came with a business card with the REACT toolkit Uniform Resource Locator (URL) so that relatives would have the address to hand, and an information card to prompt staff to record the information (name and e-mail address) needed to later send an invitation from the REACT website.

FIGURE 16 Booklets and merchandise for IPv2. Reproduced from Greenhalgh et al.74 © Trisha Greenhalgh, Joseph Wherton, Chrysanthi Papoutsi, Jennifer Lynch, Gemma Hughes, Christine A'Court, Susan Hinder, Nick Fahy, Rob Procter, Sara Shaw. Originally published in the Journal of Medical Internet Research (www.jmir.org), 1 November 2017. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included. (Figure not reproduced here; see the original article.)

The booklet included an invitation to use REACT with or without taking part in the collection of research data (data showed that this was a barrier for relatives – see Chapter 7). It was important that relatives could read a description of REACT without going online. The packs were welcomed, and fitted with how clinicians generally offered services and built relationships with relatives.

In IPv3, minor modifications were made in response to feedback, including using the term ‘friends and family’ on the cover to be more consistent with the terms services used.

Merchandise

Mugs, pens and sticky notes with the REACT URL (see Figure 16) were also suggested by staff at the workshops, to raise awareness within teams and ‘give something back’. However, feedback on the merchandise was mixed. Some reported it serving its purpose: ‘I first became aware of REACT, I think I saw it on a mug and asked somebody about it’, but others were less positive. One person commented ‘I think it’s a waste of funding’. One team offered the merchandise as prizes in a competition for the highest number of relatives invited (Ocean); others handed them out to all staff. Teams were encouraged to use them as they saw fit.

E-mail reminders

E-mails to nudge relatives to visit REACT were already part of the toolkit but were redesigned to be more user friendly, and reminders to staff were introduced. Modifications included significantly reducing the length of the e-mails, making the hyperlink to the toolkit more prominent and increasing visibility of the logo at the head of the e-mail. New subject headers explicitly stated the content or action required.

In IPv3, the source and wording of the e-mails were changed to reduce the likelihood that e-mails went into spam or junk folders.

An easy-to-remember URL

The original URL required users to type in the right trust initials (which for one trust changed during the study). The new URL made it easier for staff and relatives to find the right REACT website and the inclusion of ‘nhs’ within the URL conveyed that REACT was being offered through an NHS service, to differentiate it from the REACT trial website (www.reacttoolkit.co.uk), which was being used in parallel.

Planning document ‘Maximising the success of REACT for your EIS’

This was used to structure a collaborative process with key individuals in the trust to help them to think about what was needed to support REACT in their trust. It could help teams to identify who was best placed to offer REACT, how it fitted with what they already did well, and the existing places where REACT could be promoted to staff and relatives. The final version of the document is available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020).

In IPv2, this was designed as an internal self-reflection process for staff. In IPv3, it evolved to facilitate a conversation with the identified agents of change, and included a plan for how the conversation would continue to be revisited in the future. Key questions developed as part of the workshops included:

• Why are we offering REACT? What key targets or aims will it help us meet?
• How will staff team members be introduced to REACT?
• Where does REACT fit into the existing care pathway for relatives?
• Which people in the team/service will carry out each of the key REACT roles?

Dashboard changes

The website front page was upgraded to show clearly how many relatives’ invitations had been sent, and how many had accepted. Quotations from relatives who had found REACT helpful were added and the link to create a relative invitation was given prominence. Names of key functions were changed.

In IPv3, an icon from the main website was introduced so the dashboard was more easily discoverable. An audit function was made available to REACT supporters, REACT champions and IMPART leads so they could easily see which relatives had been invited, and whether or not they had activated an account. Staff were encouraged to follow-up with relatives who had not created an account.

Wave 2 outcomes

Table 18 shows uptake of REACT by staff and relatives in wave 2 trusts. Table 19 shows qualitative outcomes in response to the four NPT core constructs.

As in wave 1, most staff who were interviewed understood what REACT consisted of and could easily identify its potential value to relatives and staff:

Much more user friendly, much more sophisticated, much more aesthetically pleasing, and much more modern. There was a carers’ pack that the trust sent out and I was very unimpressed with that pack, just very dry, very sort of legal, not helpful I didn’t think.

There’s BFT [behavioural family therapy] and that’s offered to everybody as well, but I don’t think there’s anything been sort of something you can log on and look at and use at your own time, you know, in your own pace.

In Ocean, there was some concern about the benefits of DHIs to older relatives, and the need to delay the introduction of REACT until the time was right:

A lot of people that come into our service are kind of coming in at their wits’ end and initially all they want is their loved one to get better. So I think usually other than some basic psychoeducation verbally that’s all that they want at that point because it’s just a bit too overwhelming. And usually after a couple of months that’s when you can start kind of drip-feeding it in.

Many staff also felt that DHIs were not inherently as good as face-to-face support, particularly when they were unable to access all of the information that staff were party to:

It does meet a big need but . . . it doesn’t encompass everything. There needs to be that one-on-one, that face-to-face kind of ability for someone to talk to someone about someone you know, especially like professional-wise you know.

Concerns about staff workload, low morale and high rates of staff turnover were also evident in wave 2 trusts. Despite this, there was evidence of engagement: REACT booklets were put in carer welcome packs, and REACT was promoted at friends and family groups.

Interestingly, in both trusts, the IPv2 resources were used to change the way that relatives were signed up to REACT. Rather than all care co-ordinators signing up relatives themselves, care co-ordinators used the booklets to introduce REACT to relatives face to face, and used the cards to get their details, which they then passed to REACT supporters and IMPART leads. This helped care co-ordinators overcome the technical challenges but meant that they were not very familiar with the website content and did not see activity on the dashboard, and so received no feedback about relatives’ use.

Despite generally positive feedback about the IPv2 changes, REACT was still given low priority, either because it was seen as aimed at carers, not service users, or because it was still seen as research:

I have given it to a couple of people when I’ve done first assessments and yeah it probably isn’t seen as a priority with me . . . You go in and start work and developing care plans and things with the clients [and] maybe you don’t always then focus on the carer.

I believe in research, I like the idea of trials and trying to improve, so if I can remember what I’m asked to do I’ll try and get people on board, but we do forget . . . So reminders are good I think.

REACT was also seen as belonging to a particular professional group who sat outside the multidisciplinary team of care co-ordinators:

It shouldn’t be psychology should it, REACT, it’s just an information sharing but because it is held in psychology people are thinking oh it’s another psychology strategy.

Staff in Seashore suggested having a ‘REACT champion’ in each team to ensure that REACT remained a regular agenda item and a priority for all staff.

Finally, the demotivating impact of slow uptake by relatives and lack of activity on the forum was evident in both trusts, although there was also evidence that positive feedback inspired staff to use REACT:

I’ve had such nice feedback from people really, relatives really sort of saying I’m so glad you told me about this, ‘cos I’ve been Googling and trying to find stuff out and it’s all just coming back and I don’t understand it, whereas this is kind of it’s easier to access and it’s user friendly I think.

Implementation plan version 3

Implementation plan version 3 was developed during wave 2 and aimed to address some of the key issues identified in wave 1, 2 and 3 trusts. Factors identified in waves 1 and 2 fed into workshops in wave 3 trusts where the implementation plan was designed and delivered.

Implementation plan version 3 primarily addressed concerns over staff not inviting relatives; staff training not being accessed, particularly by new staff members; the need for feedback about sent invitations; and the longevity of someone driving REACT in the trust after the IMPART study ended. All components of IPv2, including those shown in italic text in Table 17, were included in IPv3, which is described in Table 20.

‘Request access’ function

Lakes Trust staff felt that a ‘request access’ function would empower relatives who had heard about REACT and allow them to choose when they wanted to be invited. Marsh Trust staff felt that the function was redundant as they already had a working system to generate relatives’ invitations.

Staff induction packs

Staff induction packs were developed as ‘quick start’ guides for the main REACT roles, including IMPART lead, REACT supporter and REACT champion, and were used to train new staff quickly. These were available within the updated online manual described in Updated ‘how-to manual’ and as printed documents. Packs included a table of who does what and posters for staff offices and communal areas and relatives’ waiting areas [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)].

The REACT champion

This new role supporting the IMPART lead was developed to support individual team members to offer REACT, keep REACT on the clinical agenda and ensure that it retained a presence within teams after the completion of the IMPART study. This helped across larger trusts with multiple EIP teams and wider areas to cover. The REACT champion would be the ‘go-to’ person for information on how to use REACT and would work closely with the REACT supporters. The role would ideally be taken by someone already leading on carer support.

Updated ‘how-to manual’

The new manual included mini ‘how-to’ videos for each task and screenshots so that actions could be easily followed. REACT supporter and clinician roles were supported by a quiz to check understanding; completing this resulted in a certificate to use in professional development portfolios.

The newsfeed

The newsfeed was an attempt to increase staff visits to the website and provide feedback about relatives’ activity on the website and other carer-related events. All staff with a REACT login received regular e-mail updates of the most-talked-about topics in their trust forum and any news that REACT supporters wished to share. However, this proved problematic because of the lack of activity on the forums. The forum was the most visited part of the toolkit (see Chapter 5), but this seemed to be largely owing to relatives checking back in the hope that there was activity, rather than responding to a post (see Chapter 7). The newsfeed became in danger of having the opposite effect of decreasing staff engagement by highlighting the lack of uptake, which staff found demoralising.

Printable PDF summaries of REACT toolkit modules

Staff welcomed the printouts of information modules because they helped to overcome the challenge of remembering their log-in details or not having the technology to show relatives the website in home visits. The PDFs gave relatives a clearer idea of toolkit content and increased the number of people responding to the invitation [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)].

Demo version of REACT

This was created for staff in Lakes as requested. The REACT champions used it when they met to become more familiar with the REACT website and practise tasks required to send invitations without ‘breaking’ the live Lakes REACT website or having access to confidential information on the forums to which they were not privy.

Wave 3 outcomes

Table 21 shows uptake of REACT by staff and relatives in wave 3 trusts. Table 22 shows qualitative outcomes in response to the four NPT core constructs.

Owing to the study design, wave 3 trusts had the most time to explore and use REACT in a way that suited their existing service. In Marsh, staff had undertaken a lot of work in adapting REACT to fit their service. Specifically, early concerns that the clinical team was unable to support the REACT supporter role meant that this had been adopted by a non-clinical member of the R&D team who also moderated the forum and responded to direct messages.

In contrast, there was little evidence of any activity related to REACT in Lakes before the start of wave 3. This is consistent with the activity graphs shown in Chapter 5 (see Figures 8–13). This seemed to be largely owing to the initial IMPART lead (a senior psychiatrist with strong research links) leaving the post shortly after the start of the study. Although this role was delegated to a colleague (also a psychiatrist), no one in the trust had voluntarily or actively bought into REACT becoming part of the clinical service.

Although the same potential values in offering REACT to relatives were identified in both wave 3 trusts as had been seen in waves 1 and 2, concerns were also raised that impeded the implementation of REACT. The main concern involved the REACT forum: Marsh in particular had previously experimented with online forums with negative consequences, including staff being individually identified and ‘trolled’ online:

My main concerns were I had experience with forums before, and if they are not well maintained, if they are not well administered and moderated, you can have problems . . . It’s not a forum about football or . . . This is a forum about something serious.

In Lakes Trust, a lack of trust policies for managing risk online was repeatedly raised by a range of staff. This resulted in staff in the REACT supporter role refusing to moderate the forum for fear of being held responsible for risks they did not know how to manage:

It’s absolutely no chance I’m going to be putting my registration at risk . . . Somebody just says I need help right now, doesn’t get it within 20 minutes, a stereotype saying [?] off the bridge and then well they put a cry for help knowing [that I] was on duty and he didn’t see it.

Other concerns included a lack of fit between REACT and the ethos of the EIP teams, particularly the general use of DHIs, and use of diagnostic terms. In REACT, common diagnostic terms were defined to help relatives understand what they mean, but EIP teams were keen to avoid these terms as stigmatising and frightening:

I think the first description was one of schizophrenia and I wasn’t impressed with that ‘cos I think that is scary for having that as your first introduction to psychosis . . . We wouldn’t ever give somebody a diagnosis of schizophrenia within the first year probably anyway.

Digital health interventions are not commonly used, and require a shift in culture that may be felt to threaten the centrality of the staff role:

I suppose it is a slight break with tradition isn’t it, to use an internet-based source, because I suppose it depends how professionals are; our most valuable tool is ourselves and I don’t know whether there’s something that potentially is a bit daunting about handing that over to something else, i.e. technology.

. . . Who’s taking ownership of the information, who’s overseeing the risk and things like that, so I was just sort of reminded about the reticence maybe or the anxiety about health care and health-care professionals sort of losing some of that oversee over what’s happening.

The same pressures of workload and prioritisation of financially linked priorities were identified as in the previous waves, resulting in a lack of support for the REACT supporter roles:

If we’ve got our managers saying well . . . there’s £20M to this physical health target . . . it’s quite disheartening for the nurses because all you want to do is serve your patient and in essence you’ve got to come off clinical duty so you can prioritise this target . . . There’s money attached to absolutely everything.

Also consistent with previous trusts, REACT was strongly identified as a research project. Although in all trusts this actually stimulated clinical activity (see Figures 8–13) because the research interviews and workshops reminded staff about REACT, it was also seen as inherently time limited and as the responsibility of the research team to implement.

In Marsh Trust, this led to the REACT supporter role being taken on by a non-clinical member of the R&D department just to make the project viable. The clinical team did not feel that they had the resources to support REACT but the R&D department was keen to take part in the research and so this collaborative solution was forged. In many ways, this worked well during the lifetime of the IMPART study. The new REACT champion recruited many relatives and gave their contact details to the REACT supporter to sign up, and this generated the highest number of relatives’ accounts of any trust. However a lack of engagement from the rest of the clinical team is likely to lead to REACT being abandoned at the end of the IMPART study.

Staff are used to research projects that are time limited and do not lead to lasting service change. They know that there is no R&D funding after the end of the study. Owing to a lack of infrastructure to sustain interventions developed and tested using research funding, our research team was unable to offer a longer-term solution to this issue, or even promise to make the REACT website available after the end of the study:

We do mention it every week in our team meeting, but it’s under our research headline, and we’re actually involved in about three other research projects, so there is a little bit of fatigue shall I say with research projects, so maybe we need to relocate it under our carers.

In Lakes, there was the added complication that REACT was seen as ‘owned’ by psychiatry. The trust already had a comprehensive family therapy service, which would have been the ideal service to support REACT but it was not involved in the intervention. This confused staff:

Yeah I do think there is something a bit odd about a family resource coming from psychiatry not from the family therapists; I do think that is unusual.

Summary

All staff in all six trusts could see clear value in offering REACT to relatives, which was consistent with the findings from the costs and savings survey reported in Chapter 5. However, despite collaborative attempts to develop iterations of the implementation plan that addressed barriers to implementing REACT, other issues remained.

Some of these were linked to limitations in the content and design of the REACT toolkit. Limited time and resources meant that we could tweak some elements of the dashboard or wording of e-mails, but fundamental changes were not possible during the IMPART study. However, we have clear ideas of the required changes to content and design, such as making REACT available as an app that does not require a login at each visit. With additional funding, these can easily be addressed.

Other barriers we are less able to address are described below.

Fears among staff about the growing role of DHIs, their potential to replace staff and their responsibility for risk are understandable and not unreasonable.

The view of research as ‘nice to have’ (or otherwise) but not fundamental to health-care delivery, particularly for staff under such pressure, was not a surprise. Similarly, the lack of infrastructure to continue interventions that have been developed and implemented as research projects was a challenge we were aware of from the outset. However, in both instances, we may have underestimated their direct impact on staff engagement. Change in the NHS takes time, and some trusts were only beginning to really explore the challenges they faced in offering REACT when the end of the study period loomed large and there was no continuation plan:

The thing is with this job . . . there’s change all the time and there’s new systems and new forms to get used to. You know, I think it just takes time for anything new to get embedded.

These wider contextual issues are further elaborated in the explanatory synthesis for the final version of our implementation plan (IPv4) in Chapter 8.