Implementation, intervention and resource outcomes: quantitative data

This chapter summarises the quantitative data collected to describe the levels of implementation of REACT within each of the six trusts, the characteristics of the relatives who signed up to REACT, the impact of REACT on relatives’ outcomes measured using online questionnaires, and the resources needed to implement REACT. Chapters 6 and 7 present primarily qualitative data about the process of implementation, which attempt to explain the outcomes presented in this chapter.

Implementation outcomes are presented for each wave. Trusts in each wave received successively more developed implementation plans, which were informed by the qualitative findings of our process evaluation (see Chapter 6). Presenting the data in this way allows us assess whether or not implementation outcomes improved with each iterative version of the implementation plan. Relatives’ characteristics and intervention outcomes are not presented by wave as we did not hypothesise any relationship between implementation plans and relatives’ characteristics or the impact of the intervention. There were also insufficient data to allow any meaningful comparisons to be made between waves.

Implementation outcomes

Staff activity and relatives’ uptake of REACT in each NHS trust

Figures 8–13 and Table 6 present the number of staff accounts created, invitations sent to relatives and accounts created by relatives in each of the six participating NHS trusts. Figure 8 also shows the timing of key research-related activities in each trust.

FIGURE 8 Timeline of clinicians’ and relatives’ activity in wave 1 trusts: Woods Trust. (Figure not reproduced here; see the original article.)

Wave 1 trusts

As seen in Figures 8 and 9, uptake, defined by the numbers of staff accounts created and numbers of relatives invited to use REACT, was relatively low, and similar in both trusts. Although more clinicians set up accounts in Woods than in Moor, subsequent activity was lower in Woods (fewer invitations were sent and fewer relatives signed up to REACT). There were no significant changes over the study period. Research team activity was minimal in wave 1 trusts, which received only IPv1.

FIGURE 9 Timeline of clinicians’ and relatives’ activity in wave 1 trusts: Moor Trust. (Figure not reproduced here; see the original article.)

Wave 2 trusts

Wave 2 trusts were given an enhanced implementation plan (IPv2; see Chapter 6). This appears to be associated with additional uptake and use in Seashore, with more relatives invited to register by staff members and more accounts created by relatives, but there was substantially less activity in Ocean throughout the study period, as shown in Figures 10 and 11.

FIGURE 10 Timeline of clinicians’ and relatives’ activity in wave 2 trusts: Ocean Trust. (Figure not reproduced here; see the original article.)

FIGURE 11 Timeline of clinicians’ and relatives’ activity in wave 2 trusts: Seashore Trust. (Figure not reproduced here; see the original article.)

Wave 3 trusts

The two trusts in wave 3 received the most intense implementation plan, the content of which is described in Chapter 6. As in wave 2, the trusts differed in levels of activity, with Marsh Trust showing more activity than Lakes Trust, as seen in Figures 12 and 13. IPv3 appears to be associated with an increase in the number of invitations sent and relatives’ accounts created, but not in the number of clinician accounts created. Marsh Trust was the only trust with more relatives’ than clinicians’ accounts. In contrast, the most clinician accounts were created in Lakes Trust, although the number of relatives’ accounts remained comparatively low.

FIGURE 12 Timeline of clinicians’ and relatives’ activity in wave 3 trusts: Lakes Trust. (Figure not reproduced here; see the original article.)

FIGURE 13 Timeline of clinicians’ and relatives’ activity in wave 3 trusts: Marsh Trust. (Figure not reproduced here; see the original article.)

Table 6 provides summary statistics for accounts created by staff, invitations sent in total (including repeat invitations), number of relatives invited and number of relatives’ accounts created. Table 6 also shows summary data for levels of support offered to relatives by each trust prior to using REACT, as reported in the National Audit of Early Intervention Service Providers conducted by the Royal College of Psychiatrists in 2016.102

The highest number of clinicians signed up at Lakes Trust (n = 64; wave 3) and the lowest number signed up at Ocean Trust (n = 32; wave 2). Seashore Trust (wave 2) sent the most invitations in total (n = 112) and to individual relatives (n = 87), whereas Lakes Trust sent the fewest (despite having the largest number of clinician accounts). A minority of clinicians across all trusts sent invitations, with the fewest being in Ocean (n = 4) and the most in Seashore (n = 18). Marsh Trust had the most relatives’ accounts (n = 53; wave 3), and Woods Trust had the lowest (n = 7; wave 1). High uptake in Seashore may be linked to the lack of other forms of family intervention offered, based on national audit data. According to the audit, Lakes Trust offered family intervention to the highest proportion of relatives (62%), whereas Moor (wave 1) and Marsh offered family intervention to the lowest proportion (3%), much lower than the national sample average of 31%. As for carer-focused education and support, Ocean offered this to the highest proportion of relatives (87%) and Moor to the lowest (3%), again substantially below the sample average of the audit.

Data collection from relatives

Figure 14 illustrates the recruitment and attrition of relative participants. Clinicians across the six trusts sent a total of 355 invitations to relatives; 159 relatives’ accounts were set up across the six participating trusts. Sixty-two participants consented to complete outcome measures, of whom 56 (baseline), 21 (12 weeks) and 20 (24 weeks) completed at least some measures. Sixteen participants completed (at least partially) the measures at all three time points.

FIGURE 14 Relatives’ demographic and intervention outcome data: recruitment and retention. (Figure not reproduced here; see the original article.)

Participants’ characteristics

Table 7 summarises the sociodemographic characteristics of relatives. Comparisons were made between clinical relatives and research relatives, but there were no significant differences in any of the variables. The majority of participants in both groups did not work full time, were aged > 50 years, were female, were from a white ethnic background, used broadband or Wi-Fi to access the internet and were educated to Advanced level (A level) or above.

Table 8 provides information about the caring roles of research participants from the ‘About your caring role’ questionnaire completed at baseline (see Appendix 5 for 12- and 24-week follow-up; this information was not collected from clinical participants). The majority of participants had been in caring roles for > 1 year and had an average of 45 hours of predominantly face-to-face contact with the person they were supporting each week. Most participants reported that caring for their relatives had not affected their employment status or performance, because they either had not been in paid employment prior to their caring role or were still performing the same work. A small number of participants at each assessment point reported either stopping work or reducing their hours owing to their caring role. Participants reported caring for between one and four relatives, with the majority caring for one relative only. Demographic and clinical characteristics of the relatives cared for are also presented in Table 8 (see also Appendix 6). The most common relationships were mother (participant) and child (relative cared for). Individuals who were cared for were primarily aged < 25 years and living at home. The most commonly reported diagnosis was psychosis. However, there were relatively high rates of missing data or instances when participants did not know the diagnosis of their relative.

Web use of relatives

Table 9 summarises the 24-week web use of research and clinical participants, and the results of independent t-tests comparing their web use (see also Appendix 7). The mean number of logins for the group as a whole was 3.78 (SD 4.43), but there was a lot of variation, with a range of 0–31 logins (median 2 logins, IQR 1–8 logins). The mean total time spent on the website was 40.6 minutes (SD 54.54 minutes), with a range of 0–298 minutes (median 20.1 minutes, IQR 4.9–57.5 minutes). In general, the research participants used the website more than the clinical participants, and the difference was significant in the total number of logins, number of visits to modules 1–6 and module 11, number of visits to the forum, total time spent on the website and total number of module visits (composite measure of visits to the 12 different modules).

Figure 15 displays the average number of visits during the 24-week follow-up period for clinical and research participants to the different sections of the website. Module 1 (‘What is psychosis?’) and the forum were the most frequently visited sections for both clinical and research participants, and module 9 (‘Understanding mental health services’) was the least visited by both groups.

FIGURE 15 Website use of clinical and research participants. (Figure not reproduced here; see the original article.)

Intervention outcomes

The impact of REACT on relatives self-reported measures

The descriptive statistics of self-reported measures at the three time points are summarised in Table 10. Results regarding the GHQ-28 screening scores (items scored as 0, 0, 1, 1) indicated that 34 participants (60%) scored above the clinical threshold of 4 out of 5 at baseline, compared with 11 (55%) at 12 weeks and 12 (60%) at 24 weeks. The average total GHQ-28 scores (items scored 0–3) were also above the suggested clinical threshold of 23 out of 24 at each follow-up point, confirming high levels of distress in the sample. Mean values of the GHQ-28 total scores, and subscales of social dysfunction, severe depression and anxiety and insomnia, the CWS well-being scale and eHealth literacy all improved with small effect sizes. Paired t-tests with 95% CIs compared baseline scores with those at the 12- and 24-week follow-ups; none of the changes from baseline was statistically significant (see Appendix 8).

Resources: costs and savings of implementing REACT

All trusts were given free access to REACT and IPv1 at the start of wave 1. Trusts in waves 2 and 3 received additional resources associated with IPv2 and IPv3 at no cost from the research team. These were estimated to cost approximately £400 per trust, and would be an additional treatment cost if REACT was implemented in routine clinical practice. The plans are described in full in Chapter 6. A summary of estimated item costs is provided in Appendix 9.

Survey responses

Forty-eight staff members completed the costs and savings survey across five trusts (Table 11). Lake Trust had the highest response rate; no staff completed the survey in Woods. The majority of respondents were clinicians who had referred relatives to REACT, followed by REACT supporters. Given the variation in number and role of respondents, it is not valid to make comparisons between trusts or across different waves. However, a general picture across all trusts can be ascertained.

Costs

Individual staff time required to implement REACT

Table 12 presents the time spent implementing REACT by staff in each trust. Three types of activity were measured: time spent training and learning about REACT; staff activity within their teams, such as setting up accounts and inviting relatives; and research activity.

Training and learning

Most staff reported having received 1–2 hours of training and that most of this was delivered by the research team. Approximately two-thirds of respondents had spent < 2 hours becoming familiar with the content and features of REACT.

Staff activity in their teams

Approximately 40% of respondents neither showed other team members how to use REACT nor were shown by others in their team. The majority of respondents had never set up an account for other team members. Approximately one-third of the staff had spent < 1 hour inviting relatives, and very few spent > 5 hours. Eighty per cent of REACT supporters had received either no supervision or just under 1 hour of supervision in total during the study period.

Research activity

This is presented separately from clinical activity but is included here, as taking part in the research clearly triggered clinical activity for some staff (see Chapter 6). Most staff members spent < 1 hour on research activities, including taking part in interviews and implementation plan workshops. Approximately 13% of the respondents reported having spent > 5 hours on various research activities.

Organisational activity to implement REACT

The survey also explored time spent discussing REACT in organisational meetings (Table 13). Approximately half of the respondents had seen REACT as a regular agenda item at meetings. These were mostly monthly clinical team meetings, attended by 5–20 people. However, in most instances, REACT was discussed for < 5 minutes and only two respondents reported that REACT was discussed at a regular meeting for > 15 minutes. Other meetings in which REACT had appeared as a standing agenda item included business meetings, EIP steering group meetings, referral meetings and R&D meetings. REACT was an ad hoc agenda item in a clinical improvement group in Seashore, a strategy meeting in Lakes and a mental health care group clinical governance meeting in Moor.

Additional costs

Respondents were asked whether or not any additional resources had been required to deliver REACT. In one trust, it was reported that a computer had been bought; in another, additional software had been bought (although accessing REACT requires only an internet browser).

Savings

The final section of the survey explored whether or not staff members could identify areas where REACT contributed to savings of resources or other benefits to staff or relatives (Table 14). The majority of respondents did not feel that REACT reduced the time they spent supporting relatives. Those who did save time estimated this to be only 1–2 hours across the whole study period. However, > 80% of respondents reported that REACT helped their services to better support relatives by equipping staff members with a resource to share with relatives and helping them to reach both national policy and local service targets to support relatives. Respondents in Ocean also felt that the toolkit improved their confidence and offered a backup for their important conversations with relatives. They highlighted that REACT not saving time was a positive thing as it may indicate staff spending more time communicating with and supporting relatives as a result of having REACT and improved confidence. In Lakes, staff felt that REACT had improved communication with relatives owing to relatives having more understanding of psychosis/bipolar disorder. In Moor, staff reported that REACT was particularly helpful to relatives who do not wish to attend family interventions but were seeking a better understanding of mental health problems. Finally, in Seashore, a member of staff mentioned that REACT fostered thinking of alternative ways to support carers using digital/online resources.

Summary

Overall, little time was dedicated to REACT in any of the trusts. There was individual variation, but a general pattern seemed to be that staff spent a couple of hours being trained to use REACT, a couple more to familiarise themselves with the content, and then no more than a few hours over the entire study period referring relatives. Very little time was spent in any trust discussing REACT in meetings, and REACT supporters did not receive much supervision in their role. Despite this relatively low level of investment of time, staff still did not generally feel that REACT had saved them time in supporting relatives. It may be (as suggested by one participant) that this was because REACT itself facilitated increased contact with some relatives, as it facilitated communication. However, other benefits were identified, including REACT being seen as a valuable information resource for staff and relatives, and an appropriate way to meet national and local clinical targets.

Relatives who were invited to use REACT were predominantly white British mothers aged > 50 years reporting high levels of clinical distress. On average, they used the website for < 1 hour in total, although there was considerable variation between individuals. All outcome measures showed change in the desired direction over time, but changes were small and none was statistically significant.

There was some evidence to suggest that successively more developed versions of the implementation plan led to greater numbers of relatives being invited to use REACT. However, the relationship was far from straightforward, with wide variation between trusts in the same wave. How the implementation plans were used in each trust and other relevant factors affecting the implementation of REACT are explored in Chapters 6 and 7.