In this chapter, we revisit the main aims and specific objectives of the study and show how each of these has been met. We outline the strengths and limitations of the study that need to be taken into account when using the study findings. We highlight implications for clinical service delivery, implementation theory and further research. Finally, we reflect on our PPI strategy, which is given in Appendix 1.
Study objectives
The overall aim of the IMPART study was to identify critical factors affecting the implementation of an online supported self-management intervention for relatives of people with recent-onset psychosis or bipolar disorder into routine clinical care, and to use this information to inform an implementation plan to facilitate its widespread use and inform wider implementation of DHIs. Over a funding period of 30 months, we were able to achieve all six objectives listed in Chapter 2.
Measure uptake and use of REACT by NHS early intervention for psychosis teams and relatives
Over the study period of 18 months and across all six trusts, 281 staff accounts were created, 355 relatives’ invitations were sent, 310 relatives were invited and 159 relatives created an account. The most common characteristics of the relatives were that they were white British, a mother and aged > 50 years, and caring for their son/daughter with a diagnosis of psychosis who was living at home.
The mean time spent using the website was 40.6 minutes (SD 54.54 minutes) and the mean number of logins was 3.78 (SD 4.43), although there was considerable individual variation. We do not know how much use of REACT is required for effective change, but these levels of engagement compare relatively favourably with other online interventions, which often show lower than intended engagement118 and in some cases only a minority of participants visited the website more than once.119
The most common pattern of staff behaviour was 1–2 hours being trained to use REACT (including presentation by the research team), < 2 hours familiarising themselves with the content and < 5 hours referring relatives over the entire study period. REACT appeared as an agenda item for a range of meetings, but was only a regular item at clinical team meetings for approximately half of the staff, and was commonly discussed for < 5 minutes. REACT supporters generally spent longer training and using REACT than other staff did, but received little or no supervision.
Identify critical factors affecting implementation of REACT
Chapter 8 presented a full explanatory synthesis of the critical factors affecting the implementation of REACT. In summary, staff in EIP teams were under great pressure, with high caseloads, several competing priorities and, in some trusts, high levels of staff absence. Staff found it difficult to prioritise time spent supporting carers.
Staff and relatives agreed that REACT could offer an appropriate way to increase access to information and support for carers, but only if it was delivered as part of a comprehensive care package that also included face-to-face support. Staff who used REACT felt that it had facilitated communication with relatives, saw it as a valuable information resource for staff and relatives, and an appropriate way to meet national and local clinical targets, but they did not generally feel that REACT had saved them time in supporting relatives, and may have even increased the amount of contact they had. Staff also reported frustrating practical and technical difficulties.
Relatives were generally very positive about the content of REACT. They valued the comprehensiveness of the modules, and particularly valued hearing about the experiences of other relatives and service users through the videos. However, many reported technical issues and were disappointed by the low level of activity on the forums, making them reluctant to post. This, in turn, made staff less motivated to refer more relatives, creating a vicious cycle working against sustained use.
REACT did not fit with staff members’ current ways of working, which were primarily paper based and community located. This incompatibility was exacerbated by a lack of up-to-date mobile technology. Staff felt that they needed more support and training to use DHIs, and had specific fears about being trolled online and managing risk. Fundamentally, they saw the human relationship as the main agent of change in mental health services, and felt that DHIs somewhat threatened this.
Finally, REACT was offered in the context of the IMPART research study and consequently staff saw the research team, not clinicians, as being responsible for driving it forward. They also recognised that REACT might not be available after the research period and lacked longer-term funding, making them question the value of changing their current working patterns to accommodate REACT.
There was some evidence that the iterative versions of the implementation plans developed during the study (see Chapter 6) led to increasingly more relatives being invited to use REACT. However, the relationship was far from straightforward, with wide variation between trusts.
Some of these factors are consistent with the findings of previous research in this area: specifically, staff concerns about the impact of DHIs on staff–patient (in this case relative) relationships, and the need for appropriate IT equipment, collaborative technology design and training and support have all been identified in other health contexts.120 The negative impact on staff engagement of poor system design, IT failures and fears that the use of technology will dehumanise care have also been highlighted.121 Staff fears about managing risk, being trolled online and the negative impact of the research context are unlikely to be particular to REACT, and need further exploration in other contexts.
Identify resources required (and cost implications) for successful implementation of REACT in early intervention for psychosis teams
Here, we outline the main resources required by our final implementation plan (IPv4). We have not allocated specific numerical costs, as these will vary depending on the organisational structure in any one trust (staff costs) and on the number of trusts adopting REACT (cost of delivery per trust).
Implementation plan version 4 (see Chapter 8) was designed to address each of the barriers to implementation and enhance identified facilitators, and hence suggested a fundamental change in the way in which REACT is offered.
Rather than each NHS trust hosting a separate version of REACT, restricted to relatives in their trust only, the intervention could be more easily delivered if it were available nationally from one centrally located and funded source, supported by dedicated trained REACT supporters, and open to all relatives in trusts that adopt the programme. This would overcome the challenges that staff found in dedicating time to the REACT supporter role and in accessing supervision. It would create a much larger population of participating relatives, generating a critical mass to stimulate an active forum. Finally, it would create an effective way to engage relatives as peer workers in the delivery of NHS services as they could be trained as REACT supporters.
Set-up costs
The primary cost for this set-up phase is staff time and a licence fee to adopt REACT. Clinical staff and those with responsibility for organisational strategy, transformational change and IT need to be involved with relatives in a joint review of the pros and cons of using REACT in their service. All identified concerns should be addressed before adoption. Policies (including risk management) should be adapted to accommodate the online nature of REACT. Key roles need to be allocated, and pathways designed, clearly stating who will offer REACT, when, how and with what support. A mechanism should be established to review progress and adapt REACT and the implementation plan at regular intervals.
Ongoing costs
Ongoing costs also primarily involve staff time. IPv4 recommended that a REACT champion should be identified in each team. Their role would be to create and maintain a visible REACT presence for all staff, ensuring that it is a regular agenda item in appropriate meetings. The branded merchandise developed in this study (sticky notes, travel mugs, posters, etc.) could support this. The champions would co-ordinate a rolling programme of staff training (formal and informal) and facilitate communication between the clinical team and REACT hosts.
Staff identified to refer relatives to REACT need time to be trained and to get to know the website. They need REACT booklets that describe the intervention and its benefits, which should also be placed in carer packs. Staff need up-to-date mobile technology that allows them to show the DHIs to relatives in their own homes. No additional software is required.
We estimated the total cost for consumables (merchandise, booklets, posters, etc.) to be approximately £400 per team, based on 20 staff supporting 200 relatives. The cost of licensing REACT per annum would depend on the number of trusts that used it, with more trusts reducing the costs. IT costs would depend on the technology chosen, but is not specific to REACT. Staff costs would vary between trusts.
Savings
REACT is not anticipated to significantly reduce the amount of staff time dedicated to supporting carers. Our data suggest that this time is already less than needed to meet national clinical targets and deliver NICE-recommended care.51,52 In fact, REACT may increase the time spent communicating with relatives. In the longer term, this should improve carer and service user outcomes (and save money), but would require further testing.
Investigate the impact of REACT delivered by early intervention for psychosis teams on relatives’ self-reported outcomes
Relatives had high GHQ-28 scores at baseline, with approximately 60% scoring above the threshold for clinically significant distress. These levels of distress are consistent with those previously reported for relatives in EIP services.53,122 Between baseline and the 12- and 24-week follow-ups, there was a general pattern of declining distress, social dysfunction, depression, anxiety and insomnia, and increasing carer well-being and eHealth literacy. However, these changes were small and none was statistically significant.
Develop a user-friendly REACT implementation plan and related resources to facilitate widespread use and dissemination
Implementation plan version 4 is described in full in Chapter 8. In addition to the necessary trust resources outlined against the objective Identify resources required (and cost implications) for successful implementation of REACT in early intervention for psychosis teams (staff time and training, merchandise, licence and IT), other key elements include updating the content and functionality of REACT; identifying a sustainable provider and long-term funding model at the national level; and building a mechanism to ensure that the ongoing audit of the reach, uptake and impact of REACT that is not onerous or perceived as research.
Use the findings from this study to further develop theories of implementation of digital interventions in real-world practice
Normalisation process theory73 was used to understand the work undertaken by staff in participating trusts to implement REACT. The NASSS framework74 was used to synthesise these data with the key factors affecting relatives’ engagement and the wider contextual factors. Rather than developing these theories further, we tested the utility of both in understanding the implementation of REACT.
Normalisation process theory proved very useful in informing data collection and analysis, particularly in relation to the work undertaken by the staff. It also worked well in informing the iterative development of IPv2 and IPv3. This was the majority of our data, and so this theory would have been sufficient to develop an effective implementation plan. However, we were keen to develop this plan further to ensure that it also addressed the complexity of the wider context and the engagement of relatives. We therefore used NASSS to synthesise all of the data from the IMPART study. This led to the development of a final implementation plan (IPv4) that differed significantly from our original implementation plan (IPv1).
The NASSS framework builds on many previous theories, and synthesising our data from staff and relatives in the key domains was relatively straightforward. Working across two different frameworks could have led to theoretical confusion, but in practice it worked very well. The theories proved highly complementary: NPT gave us a detailed focus on staff behaviour, and NASSS allowed us to then integrate the key themes identified by NPT with the data from relatives and the broader contextual factors, to present our explanatory synthesis. In combination, they helped us to meet our study objectives.
Study strengths
Our study was strongly embedded in implementation theory. At a practical level, the use of theory helped focus our data collection and structure our analyses. We were able to test the utility of the NPT and NASSS frameworks in a novel context, and create generalisable knowledge to inform the delivery of DHIs other than REACT, and within contexts other than EIP teams.
Data collection took place across multiple and varied NHS trusts. This increased the external validity and generalisability of the findings, ensuring that our implementation plan, if flexibly used, should facilitate implementation across most NHS trusts in England.
Our research was very timely. The findings are of direct relevance to the government agenda to increase the spread of DHIs12 and to improve mental health services, the delivery of NICE guideline-recommended care106 and specifically support for carers.109
We examined the perspectives of the end user (relatives) as well as of staff doing the direct work of implementation. This was important because it clearly demonstrated that the way in which REACT is implemented by staff had a direct impact on engagement by relatives, and that relatives’ engagement then fed back to affect the work undertaken by staff.
We designed the study to be collaborative and informed by perspectives of all relevant stakeholders. Relatives formed part of our study team and contributed to its design, data collection and analysis. SRGs were set up at each NHS trust, and attempts were made to include a range of staff roles and relatives and service users in these groups (although this was not always successful in every trust). These groups became the vehicle for co-design of implementation plan iterations.
Study limitations
The main limitations of the study relate to the study design, use of theory and research process.
Study design
The research team had a dual role of delivering REACT while also collecting data to understand the process of implementation. This was confusing for trust clinical staff because it strongly embedded REACT within a research agenda, leading them to assume that REACT would be time limited and driven by the research team. This was a key factor in non-adoption. It also created a tension for the research team. On the one hand, our role was to observe and record, and to intervene only via the strategies outlined in our implementation plans. However, as originators, we also wanted REACT to be viewed positively by staff and, consequently, it was tempting at times to try to facilitate the implementation process. This highlights the need for independent implementation and independent evaluation of DHIs to overcome researcher bias.
Our study design included analysis of data from interviews, observations and document analysis. In the event, the number of data we were able to collect through observation was very limited. We worked with IMPART leads to try to identify meetings in which REACT was likely to be a key agenda item, and asked the meeting chairpersons if we could attend. There were very few meetings in which REACT appeared as a specific agenda item, and, where it was discussed as part of routine clinical practice, the staff did not think it appropriate that we attend the meeting owing to the confidential nature of material being discussed. Therefore, our analysis relied more heavily than we would have liked on interview data and document analysis, with less direct observation being possible.
In the ‘you said . . . we did’ cycles, staff and relatives made suggestions for how the obstacles to using or offering REACT could be overcome. The opportunity to influence the development of REACT was well received and the limits of what could be changed because of time and resources were understood. However, some changes that could not be made, such as offering REACT as a mobile app rather than a website, were repeatedly requested by two trusts (Lakes and Marsh), and this caused frustration for staff as these changes would have overcome the challenge of needing to remember a password.
For relatives, the confusion in our study design came from embedding our research within the clinical delivery of REACT. The invitation to complete questionnaires for the IMPART study initially came as part of the process of setting up a REACT account, leading some relatives to question if this was part (or even the whole) of the intervention. These processes were separated once we became aware of the issue.
Study theory
Normalisation process theory proved a useful theory to guide the collection and analysis of staff data, but was limited (at the time) in helping us to understand relatives’ engagement with REACT or in clearly specifying the wider contextual factors relevant to the implementation of REACT. We added additional codes to our coding framework to accommodate data specifically relevant to understanding the context (within EIP teams, and wider context) and the research process, which was identified as a particularly significant contextual factor. We analysed the relatives’ data using an open thematic approach.
Since the design of the IMPART study, NPT has evolved to more explicitly consider the importance of context. May et al.123 in particular highlight the interaction between the plasticity of intervention components and the elasticity of contexts in determining success of implementation. Despite this development, we chose to use the NASSS framework to synthesise our findings.
Study process
We were very ambitious in the amount of work we set out to deliver, given the time scale and resources. Our lack of experience in designing and delivering IT projects meant that we underestimated the time and resources needed to deliver this, although this is not uncommon even in large-scale national digital projects.124 Consequently, some staff were frustrated by IT failures, which, with a part-time IT developer, we could not address immediately. To meet our study objectives, we drew on the additional resources of a Master of Science student (GA-A) who led the systematic review of literature in phase 1, and a Doctorate in Clinical Psychology student (Johanna Barraclough) who led the analysis of interviews with relatives.
We underestimated the speed with which behaviour change can be achieved within the NHS, particularly when the change required is seen as part of a research agenda rather than a clinical agenda. We were unable to interview relatives as early as planned because this depended on REACT first being adopted by staff who could then invite relatives to use REACT. Consequently, these data had less impact on the development of IPv1–3 than planned. This also led to fewer relatives completing the outcome measures within the study period and, therefore, the pre–post analysis of outcome measures was likely to be underpowered to identify any significant changes.
The time frame of the study was limited, including only an 18-month window of observation following the initial deployment of the REACT intervention. Consequently, we were able to observe only the initial stages of adoption/non-adoption and early abandonment. Data collected over a longer time scale would be required to understand scale-up, spread and sustainability. This seems to be a gap in implementation science, with fewer studies funded to understand these stages of implementation.125 (Note that a request for a funded extension to the IMPART study was rejected.)
Finally, the majority of participants in this study were white British, even in trusts that served populations with high ethnic diversity and, therefore, the generalisability of the findings is restricted to this population. As participants were identified by staff (rather than the research team), this bias may reflect a more general challenge across the services in engaging relatives from minority ethnic groups.
Despite these limitations, our findings have important implications for clinical service delivery, implementation theory and further research, which are outlined in the following sections.
Implications for clinical service delivery
Our findings have clear implications for the development of clinical services in relation to supporting relatives and using digital technology.
Despite extensive government and charity recommendations to involve relatives more actively in supporting people with psychosis or bipolar disorder, and to offer education and support to meet their own needs,51,52,105,109 these remain challenging for staff. Mental health services remain chronically underfunded.126 Staff interviewed in this study felt unable to meet all of the targets set, and so prioritised within these. They focused primarily on targets that involved working with service users, and those with clear financial incentives attached, at the expense of support for relatives.
One reason for this is likely to be that individual staff caseloads are measured by number of service users. Work to support relatives requires additional time and resources, and historically has not been measured. Reported caseloads in EIP services were much higher than the original target of 15 per full-time-equivalent care co-ordinator.127 National audits have marked progress in trust delivery of AWT standards. Nationally available averages for 2016 suggest that 16.1% of people with first-episode psychosis in contact with their families took up family intervention, and 37.8% of carers took up carer-focused education and support.102 Figures for individual trusts and data on how many people were offered and declined family intervention are not currently available. However, there are as yet no financial consequences for failing to hit carer targets. Work that staff undertake with relatives needs to be visible and rewarded, which would require increased funding for mental health services.
If digital health interventions such as REACT were to become part of mental health services, the data suggests that staff and relatives would prefer their funding to augment rather than replace funding for face-to-face support. Staff and relatives were very clear that they did not want DHIs to replace face-to-face interventions, which the evidence supports as the most effective service model.128
Clinical staff who are asked to implement DHIs need reliable access to up-to-date mobile hardware and secure software. They need to work with integrated systems that require a single login, and training and support to develop their IT skills and confidence. Risk policies need to be adapted for services delivered online, and individual levels of responsibility need to be made clear. DHIs need to be strategically implemented on the basis of service demand, and with longer-term commitment to cycles of review and adaptation, rather than short-term cycles of failure and abandonment. Service demand should be determined through organisational infrastructures that support the meaningful involvement of service users and carers, as well as staff. These elements were not reliably available in any of the participating trusts in this study.
Implications for implementation theory
There are well over 100 implementation theories and frameworks,129 and at least 28 focus specifically on health technologies.74 With notable recent exceptions,129 there is very little guidance on how to select the theory most likely to be of use to a particular study or project. Rather than develop more theories, existing ones should be tested, and encouraged to evolve over time and contexts, just like the technology-enhanced services they try to understand. Our experience was that NPT and NASSS were useful for different purposes. NPT helped with a detailed analysis of the work undertaken by staff, whereas NASSS was more useful in then situating this work within the wider context, in terms of the health-care setting, the wider political context and the particular point in time. NPT recognises the importance of context, whereas NASSS provided a framework around which to understand factors that are likely to be important, and highlighted the futility of some aspects of the implementation plan in the absence of any control over factors in the wider context.
In particular, adaptations to the functionality of REACT were made to address issues raised by staff around content, usability and feedback; however, without addressing the ways in which DHIs are developed, tested and commissioned within the NHS, staff were unlikely to commit to using REACT as they perceived it as transitory, research driven (rather than clinically driven) and, therefore, of low priority.
The language used to describe concepts within the theories was a challenge. Although members of our team who were more familiar with NPT (EM) found it easy to understand and to use to help make sense of our data, those who were using it for the first time found it more difficult to understand and differentiate the core components. One example was the term ‘cognitive participation’. Those with a background in psychology intuitively interpreted this as being about the beliefs and attitudes held by the staff rather than the relational work that it refers to.
Further work is needed to understand the relative contribution of different theories to help implementation scientists to consider which theory will work best to guide their work in different contexts, and how to work with multiple theories for different levels of explanation.
Implications for future research
Implementation of REACT was so inextricably linked to the behaviour of staff delivering it and the context in which it was situated that we agree with the proposal of Mohr et al.14 that it is more helpful to consider a DHI such as REACT as a technology-enabled service (TES) rather than a digital product. This has implications for how REACT is developed and tested over time. In this study, the implementation plans have been presented as distinct from the toolkit, but they are actually an integral part of the intervention, and future evaluation should test the effectiveness of the entire TES in the context within which it is being delivered.
Mohr et al.14 also suggest that a TES cannot be evaluated using traditional efficacy trials. TESs have to be allowed to evolve and adapt in an agile way, otherwise they quickly become redundant. For this reason, trial designs that take several years to deliver, and that prioritise internal validity, may not be fit for this purpose. However, where public money is being used, it is essential that we have robust evidence of effectiveness, and there is an urgent need for new methodologies that allow us to do this.
The parallel RCT evaluating REACT supported by relatives and offered nationally,55 and this implementation study in which REACT was supported by clinicians and offered locally through EIP teams56 both tell us important things. However, integrating the findings will not tell us exactly what works for whom in what context. Future research to evaluate DHIs should consider hybrid designs that allow all aspects of the TES to develop as part of the study, and that test outcomes in situ. Possible designs to consider include realist evaluations130 and the ACTS model.86 The ACTS model uses an iterative process that includes two basic functions (design and evaluate) across three phases (create, trial and sustain). In the create phase, a minimally viable TES is created, along with an implementation blueprint. In the trial phase, both effectiveness and implementation are evaluated and changes in the TES are made using optimisation and continuous quality improvement strategies. The final phase of sustainment involves the withdrawal of research or donor support, leaving a functioning, continuously improving TES in place.
A key challenge to this design is to balance design flexibility with scientific rigour. There are many aspects to this, but of particular note are the need for transparency to prevent bias (highlighted in this study), especially given the large sums of money at stake, and randomisation to control for the impact of confounding variables (highlighted by the REACT trial131 in which both arms showed substantial improvement over time).
Finally, DHIs are often promoted as interventions that increase accessibility and, for this reason, can reduce unequal access to health care. However, early research shows that differences in access to internet and mobile technology predicted by sociodemographic factors (primarily age and ethnicity) can create a ‘digital divide’ between those with and without access.132,133 Our findings highlight the additional impact of assumptions that both staff and service users/relatives make about who these interventions are most suitable for, which then determine who is offered them and who uses them. This supports the need to explore many levels of division, including accessibility, skills, engagement and outcomes, in order to ensure that DHIs do not increase rather than reduce existing health inequalities.134