Work package 3: DOORWAYS

Introduction

Advances in psychopharmacology, changing attitudes towards mental illness and growing concerns about the quality of life in asylums have led to the gradual transition from the long-term hospitalisation of those with mental health problems to an increase in community treatment. However, hospitalisation for acute episodes is still an indispensable feature of mental health care and consumes the greatest proportion of mental health service costs. Despite this heavy investment, service users and front-line staff continue to complain about the quality of inpatient care.10 More than 50 years ago, when institutionalism was being re-evaluated, Wing and Brown11 carried out their well-known ‘Three Hospitals’ study, which confirmed that the hospitals that provided richer social environments and social opportunities had markedly less disturbance in verbal and social behaviours.11 Between 1960 and 1964, the social environment improved in all three hospitals, while the clinical condition of patients improved alongside it. Those patients with the least social interaction, the least access to the outside world and with the fewest interesting activities to take part in and who spent the majority of their time doing nothing were the most unwell, but as their environments became more stimulating, they too showed some improvements. These associations between environmental stimulation and acuity remain.

In addition, as bed numbers continue to fall, wards are generally now reserved for the most acutely ill with consequent increases in the proportion of inpatients who are compulsorily detained. Both these factors are likely to increase levels of behavioural disturbance.7–9 This can lead to a very fraught atmosphere on the wards, with nurses spending the majority of their time dealing with crises as they emerge rather than engaging in therapeutic activities or interactions with service users. The Department of Health153 has acknowledged that in spite of advocating therapeutic environments, this fire-fighting activity, along with an abundance of administrative work and lack of support and supervision, make this very challenging indeed. Taken together, this often brings about ward environments that appear to be more custodial than therapeutic.

Apart from the stressful ward atmosphere, service users also report that there is often very little to do with their time while they are inpatients, which results in intense boredom.10,19 Service users recognised that the lack of therapeutic activities (or the cancellation of those on offer) is often a result of staff shortages and/or too many demands being placed on staff. Walsh and Boyle34 found that service users do, or try to, actively seek out entertainment and company as a means of providing purpose and routine for their days. Nolan et al.154 found that the camaraderie and social support from fellow service users, as well as their ability to provide normalisation and positive examples of recovery, were very highly valued and, therefore, fostering such opportunities is important. Some service users have indeed found the support that they need from other service users who are more available than nurses.155 These interactions can most easily be encouraged through the provision of organised ward activities.

Echoing the Department of Health findings, nurses report the primary reason for not spending time on therapeutic activities or direct patient contact on the need to focus on resolving crises for a small number of patients, as well as the increase in the administrative duties expected of them.40,41 There have been a number of attempts to improve this, such as changing the allocation of nursing time from formal observation only to specific structured, individualised activities (often on a one-to-one basis), which improved patient-reported quality of care while the number of staff sicknesses and staffing costs decreased.114 Hansen and Slevin127 changed the structure of an acute care ward by introducing regularly held care planning meetings, group therapies and community meetings. Post implementation, service users on the ward reported significantly greater involvement in their own care, greater support between staff and service users, and greater opportunities to practise practical skills in preparation for discharge. A solution suggested by Mind10 was to attempt to improve staff retention which would hopefully lead to a decreased reliance on agency staff who may not have the skill or familiarity with the ward to run such activities.

A Sainsbury Centre for Mental Health16 report provided a more upbeat picture, finding that 73% of ward managers across England reported that both practical therapeutic activities (e.g. cooking skills) and talking therapies occurred regularly on their wards, and 64% reported leisure activities (e.g. film nights) occurring regularly. The most commonly offered therapy was art therapy, on nearly half of all wards, compared with 35% for psychosocial interventions. Specifically, therapies with a strong evidence base, such as CBT, occurred on only 20% of wards. The recommendations include increasing the availability of training in therapeutic activities that have an evidence base. What the reports do not discuss is whether these activities were simply in a ward timetable or whether they actually occurred in reality. The lack of more evidence-based activities taking place is primarily as a result of not having sufficient numbers of nurses to run the wards and also fulfil other ward duties, the high number of temporary staff who may be unfamiliar with the activities and a shortage of OTs and psychologists.

Although not equivalent with formal therapies, service users also value informal ordinary conversations with nursing staff34 and research indicates that, for service users, the most important aspect of quality inpatient care is that staff members take time and care with them.115 The CQC156 specifically said that when staff were questioned about activities, 96% reported activities on weekdays and 61% on weekends and evening, but when service users were questioned this dropped to 65% and 46%, respectively. Furthermore, Walsh and Boyle34 and the Royal College of Psychiatrists157 report that there is a dearth of activities in the evening and weekends, when service users would like them. This is echoed by NICE guidelines,158 which state that meaningful activities should take place outside Monday to Friday and should not be restricted to 09.00 to 17.00. NICE guidelines further recommend psychological talking therapies for psychosis and other disorders for those who are hospitalised. A voluntary organisation called Star Wards offered tips and advice on a variety of things wards can do to improve their services, among which are both formal and informal activities.159

Bowers’160 review of the changing nature of the types of service users and their needs specifically recommends that it would be within nursing staff’s remit to provide evidence-based psychotherapy in inpatient settings. Cowls and Hale161 found that inpatients also valued input from staff other than nurses and appreciated the activity-based groups as well as more psychoeducational groups. Indeed, the Royal College of Psychiatrists157 go as far as to say that all staff should share the responsibility of engaging service users. Many training/accreditation programmes have been set up to address these very issues (e.g. Cowls and Hale161) in a variety of different ways. The Department of Health153 and Clarke162 outlined some of the ways in which many such programmes could have been improved. They suggested more service user and carer involvement in the design, implementation and subsequent evaluation of acceptability of inpatient services. They also suggested involving acute inpatient staff themselves at all stages, a broad range of health-care professionals, not just nurses, and adequate training and supervision for ward-based staff in addition to ward-based learning. The aims of our project were to implement such a programme to improve inpatient services and evaluate its impact rigorously.

The aims of the study are to:

• investigate in detail the difference that increasing therapeutic activities makes on service user and staff perceptions of the environmental milieu
• explore the sustainability of positive effects (particularly on staff morale and the level of increased activities)
• examine the impact of the programme on the perceived amount of care received and its costs.

Methods

The effects of the intervention were measured by service user and staff ratings made on three, five or seven occasions. The study was organised as a stepped-wedge design (a type of cluster randomised trial in which it is the timing of the intervention that is randomised). As all clusters eventually receive the intervention, this design is often used in situations when the researchers do not wish to deny the intervention to any particular cluster (Figure 5). Please see Statistical methods for more details and Appendix 5 for a fuller discussion of this design. All service users entered the data set once only and were unaware of the condition (pre or post intervention) they were allocated (blind assessment) and all assessments were blind rated. Staff entered the data set if they had been on the ward during the assessment period. Non-permanent staff must have completed seven shifts in the past month. At the time of the assessment, staff working on control wards were unaware of when they would receive the training. Clearly, on those wards that had already been allocated to the intervention condition the staff knew what group they were in; however, all staff assessments were self-report. Service users were allowed to enter the data set only once, even if they had multiple or long admissions. Staff completed assessments each time they were available and had consented and so could have multiple entries in the data set.

FIGURE 5 Intervention schedule for DOORWAYS. 0 represents a ward for pre intervention, but for which we did collect data. 1 represents when a ward had received the intervention and data were collected. Time is measured in periods of 6 months. (Figure not reproduced here; see the original article.)

Procedures

Ethics approval for the study was granted by Bexley and Greenwich Research Ethics Committee (reference: 07/H0809/49). Researchers approached all eligible patients following consultation with ward staff. Researchers gave each participant an approved information sheet explaining the study, answered any questions and gained signed informed consent. Participants were free to withdraw at any time. Handovers and ward meetings were used to publicise the study with the nursing staff. Recruitment lasted for a period of 4 weeks at each of the time points. Data were stored in a locked office and entered into the database, and every entry was double checked by a different researcher to ensure accuracy.

The setting

This study was carried out in two mental health trusts and covers several distinct geographic areas.

In the first trust, we studied wards serving three borough catchment areas. The first borough serves an inner city population that has a deprivation index of 29.73 (ranking 41 out of 326 English boroughs163). Five wards participated in this study: three men’s wards and two women’s wards. Each had 18 beds. The catchment area also provided an intensive care ward comprising14 beds but this was not included in the study.

The second borough was located in a more suburban area, with a deprivation index of 22.76 (ranking 107 out of 326 boroughs163). Three wards participated in our study: one men’s ward and two women’s wards. Two wards had 22 beds. One of the women’s wards was unique in that, although considered part of the acute service, it was a ‘ward in the community’. This ward had only eight beds and accepted service users only if they were stable and not on a section. The ward was a detached house with a large garden located in a suburban street and the ward ethos was to provide as many therapeutic activities as possible.

The third borough had an average index of deprivation score of 31.24 (ranking 29 out of 326 boroughs163). Four 18-bedded wards provide acute inpatient care. One ward provided beds for women and one ward for men. One of wards was a specialist service, an early intervention unit accepting service users of both genders with psychosis from across the entire trust.

The final four wards were in a separate trust. This area was also mixed socioeconomically. Two of the wards served an urban borough with a deprivation score of 31.94 (ranking 28 out of 326 boroughs163). Both wards were mixed gender and had 18 beds. They also provided a PICU (not included in the study). The final two wards served two boroughs, both of which were suburban. The first had a deprivation score of 16.71 (ranking 174 out of 326 boroughs163) while the second had a score of 14.95 (ranking 203 out of 326 boroughs163). These were also mixed gender and provided 18 beds.

Participants

Service users using inpatient services on the participating ward at the time of data collection were eligible to take part in the study if they met the following inclusion criteria: they had been on the ward for a minimum of 7 days, they were able to communicate in English and they were well enough to consent. The exclusion criterion was that the participant had previously participated in the trial. There were no diagnostic exclusion criteria.

Nurses of any grade were eligible to be included in the study. Temporary agency staff were required to have completed seven shifts in the previous month in order to be eligible so that we could be sure that they were sufficiently familiar with the ward environment to complete the measures.

Main outcomes

For all measures obtained by questionnaires it was a requirement that > 80% of the questions were completed. Responses with < 80% completion were counted as missing.

Service user measure: VOICE84

Service user researchers constructed a measure from qualitative data, which was validated by expert panels of service users and tested for its psychometric properties. VOICE is easy to understand and complete and, therefore, is suitable for use by service users while in hospital. The 19-item measure has good validity and internal and test–retest reliability. The main outcome is the total score and we will carry out secondary analysis on the subscale scores.

Staff measure: VOTE85

VOTE is a 20-item measure that captures staff perceptions of the daily pressures of working in acute inpatient mental health wards. Each item is answered by choosing from a six-point Likert scale. The outcomes investigated were the total scores and, for secondary analyses, the factor scores as set out in Chapter 2. This score represents each staff participants’ view and low scores are positive. Test–retest reliability and the internal consistency of the measure and subscales were good.

Secondary outcomes

Service user measures

Service Satisfaction Scale – Residential Services Evaluation78

This is a derivative measure adapted from the SSS-30,79 designed to evaluate residential services for people with serious mental illness. The original SSS-30 has been used in a variety of settings and demonstrates sound psychometric properties.164

Rates of violence

For each service user who consented to participate in the study, we extracted data from routinely collected logs of violent incidents from the preceding 7 days. In this context, ‘violent’ can refer to either physical or verbal acts of aggression and includes both when the patient is the perpetrator of the violence and when they are the victim. Each incident was classified according to a 5-point scale; for this study we only considered incidents that were in one of the most serious three categories. It was possible for a service user to have been involved in more than one incident during the 7-day period and so the outcome measure of interest was the rate of these violent incidents. We collected these data within one trust only, so numbers of individuals will differ between the samples.

Global Assessment of Functioning147

The GAF is a 100-point rating scale of a patient’s overall level functioning that is based on observation, interview and medical records.

Positive and Negative Syndrome Scale165

PANSS is a 30-item instrument that evaluates the presence, absence and severity of positive (e.g. hallucinations and delusions), negative (e.g. blunted affect and passive social withdrawal) symptoms and general psychopathology (e.g. anxiety, depression). Researchers were trained in how to conduct the interviews and were subsequently approved by a senior member of the research team (TC).

Nurses Observational Scale for Inpatient Evaluation166

Nurses Observational Scale for Inpatient Evaluation (NOSIE) is a 12-item scale designed for nurses, largely focused on the assessment of socially unacceptable behaviour. Nurses were asked to rate participants on items such as ‘Sits, unless directed into activity’.

Client Services Receipt Inventory-Inpatient142

This instrument collects activity data on psychiatric inpatient wards. It is used to collect both ward activities and individual time that service users spend with health professionals. It assesses how much engagement service users report as well as having the ability to calculate the cost of such engagement using unit cost data from Curtis.136

Staff measures

Maslach Burnout Inventory-Human Services Survey49

The MBI can be totalled to produce a score which measures work related ‘burnout’ over 22 items, answerable by a six-point Likert scale. It has good psychometric properties and is widely used.

Index of Work Satisfaction81

The IWS is a 44-item scale that can be totalled to produce a score that measures health professionals’ levels of job satisfaction. It is answerable using a seven-point Likert scale.

Background information

Service user measures

Demographic and clinical information: each participants’ age, gender, marital status, ethnicity, education, employment status, legal status and number of previous admissions were recorded.

Staff measures

Demographic information on each participants’ age, gender, ethnicity, employment band, length of employment and education, and previous employment in wards/trusts was gathered.

Ward data

Ward Atmosphere Scale:167 The Ward Atmosphere Scale (WAS) is a well-established and validated measure of the ward environment consisting of 100 yes or no items. It was completed once a month throughout the entire study by two senior members of staff.

For each ward, we recorded the number of patients who had been on the ward in the last 30 days, gender of all patients, number of nurses on the ward, proportion of shifts undertaken by temporary staff, average length of stay, number of incidents on the ward, and ward acuity as indexed by average GAF, NOSIE and PANSS. We collected information on the number of activities provided on the wards and the number of service users participating in these from ward managers at each time point. We compared average frequencies before and after the intervention. In addition, we accounted for ward effects using a fixed-effects framework (see Statistical methods).

Intervention

Following consultation of NICE guidelines, trust clinical leads, ward managers and nursing staff directly involved with each ward, eight interventions were chosen, based on evidence of feasibility and outcome in wards. Ward managers suggested that three of these interventions should be compulsory. These were SCIT,129 CBT-based communications training for nurses (cofacilitated by a service user educator) and computerised cognitive–remediation therapy (CRT) (to involve OTs). In addition, pharmacists were recruited to run a medication education group. The wards then had a choice to receive training in two of the following five interventions: Hearing Voices Group, Emotional Coping Skills Group, Problem Solving Skills, Relaxation/Sleep Hygiene and Coping with Stigma group. Each ward made their choices based on what activities were already occurring on the ward or the client mix. Please see www.perceive.iop.kcl.ac.uk/doorways.html168 for a summary of each. A modified SCIT intervention was carried out on forensic wards as a value added study (see Appendix 4).

Statistical methods

Sample size

The total number of measurements originally envisaged to be available was seven wards and 15 patients per ward sampled over three time points after baseline, that is, a total of 315. The first six wards were to be randomised in pairs (30 patients each time). Assuming an intraclass correlation (ICC) of 0.05 with clusters of size 30, this sample size would have given approximately 80% power to detect a standardised effect size of 0.5 (moderate), using significance tests with alpha 0.05 (double sided). The patients were to be different at each time point so there was no additional clustering at the patient level to be taken into account. During the course of the project the number of wards was increased to eight and finally to 16. This design gave increased power (90%) with the assumptions above. Because of the stepped-wedge design the actual number of wards and service users in the intervention and control groups varied according to time point, so the above calculations were approximate.

Randomisation

The time at which wards were allocated to receive the intervention was randomised by one of the statisticians. Randomisation was carried out separately within boroughs and on three occasions as wards became available to receive the intervention. Random permutations of numbers were generated to designate the order of which wards were allocated to the intervention. The order that wards within a borough appeared in the randomly generated list determined the time point at which they were allocated to the intervention. This was performed using the ralloc procedure in Stata. After the initial baseline assessment period, two wards were randomised to receive the intervention with a further two wards randomised every 6 months until all wards had received the training. The design is shown in Figure 5.

Statistical analysis

The effect of the intervention was determined through effects on (1) service users, (2) staff members, including more general ward effects as well as individual level data, and (3) the cost of perceived care contacts and cost-effectiveness. In addition, data were collected to measure the demographic make-up of service users and staff, the atmosphere on the wards and any adverse events. We also recorded data relating to how the intervention was carried out, how many staff were trained and how many therapeutic sessions were run. All analysis was carried out using Stata versions 11 and 12.

Service users

The primary outcome measure of the effect of the intervention on service users was the VOICE score. This was investigated first using a t-test and second using a regression model. The model used was a Normal linear regression model. To match the design of the study, we always adjusted for the effect of ward and time. We used fixed-effects for both, using an indicator variable to account for the ward and a time variable which counted 6-month periods starting from 1 January 2009. We investigated linear and non-linear effects of time. As well as this main result, we also investigated if any covariates were confounding the intervention effect. We did this by regressing each covariate with the VOICE score and, separately, with the intervention variable. If there was evidence that the covariate was associated with both the intervention and the outcome, as judged by having a p-value of < 0.1 then it was included in the final model. This analysis was run on the total VOICE score as well as its two subscales: security and care. The same strategy was used to investigate the SSS-RES scores of service users. In addition, 95% CIs are reported for all estimates.

We also investigated interaction effects on the intervention outcome for voluntary patients and involuntarily admitted patients and two other variables (gender, ethnicity) identified a priori to be associated with VOICE.

The effect of the intervention on the rate of violent incidents on the wards was analysed using a Poisson regression model. This estimates a rate ratio, comparing the rate of violent incidents on post-intervention wards with a baseline of the rate of incidents on control wards. Thus, a ratio of < 1 indicates that the intervention reduced the rate of violence. Again, the effect of ward and time was always taken account of by using fixed-effect methods and both unadjusted and adjusted (for confounding covariates) estimates are reported.

We collected information on the number of activities provided on the wards and the number of service users participating in these from ward managers at each time point. We compared average frequencies before and after the intervention. In addition, we accounted for ward effects using a fixed-effects framework. To corroborate the staff data on the change in number and participants in the activities we also undertook an analysis of the activities data that were collected using CITRINE. Specifically, we compared the number of different activities and the total number of sessions that service users reported having participated in before and after the intervention. We used a bootstrapped fixed-effects model to obtain estimates of the intervention effect adjusted for ward characteristics and to take into account the non-normal distribution of the data. In addition, we tested for potential confounders including calendar effects as in the other analyses.

We ran two focus groups on wards for which the intervention was being implemented to elicit service users’ and staff views about the intervention.

Staff

The primary outcome measure for staff was the VOTE score. This was analysed using a regression model with a random effect to account for the effect of repeated measures being taken on staff. Ward and time were, again, accounted for as fixed effects by using an indicator variable to account for the ward and a time variable that counted 6-month periods starting from 1 January 2009. Non-linear effects of time were investigated. The same model was used on the VOTE subscales: workload intensity, team dynamics and interaction anxiety. This strategy was also used to analyse the MBI and IWS scores of staff.

The following variables derived from previous studies were considered as potential confounders: gender, age, ethnicity, employment band, first language and length of employment. Confounding factors were assessed and included in the model if they were associated with both the intervention and the outcome with a significance of p < 0.10. We performed exploratory analyses by including an interaction between the subgroup and the staff training arm and tested the nested models using likelihood ratio tests on all outcomes to explore potential differential effects between men and women, and between white and non-white staff.

Cost and cost-effectiveness

We estimated the cost of the intervention based on the cost of employing a clinical psychologist to lead the training sessions and the opportunity cost of nurses and OTs attending these training sessions [i.e. the cost of the time they could have spent performing other activities (see Appendix 6 for costs)]. We discounted the cost of the intervention over the average time that the wards were followed up after the intervention in this study. We also compared the cost of perceived staff contacts before and after the intervention. Our regression analysis followed a similar format as the analysis of service user data, that is, we estimated a linear fixed-effects model adjusting for calendar and ward effects as well as other potential confounders. To allow for skewness and kurtosis in the cost data we calculated bootstrapped standard errors (SEs). Furthermore, we estimated the cost-effectiveness of the intervention by linking cost data with the two outcome measures, VOICE and VOTE, in turn. We plotted their joint distribution on a cost-effectiveness plane using confidence ellipses to delineate the area that bootstrapped replications were most likely to lie in. To further characterise the uncertainty around the point estimates, we constructed cost-effectiveness acceptability curves using a net benefit approach. These indicated the probability that care in the intervention group was more cost-effective than the control group for different monetary values placed on a unit improvement in the two primary outcome measures.

Results

Baseline data

A total of 1108 service users took part in the study. This is 70% of the population of service users who were eligible to participate at the time of the assessments. A total of 560 staff members consented to take part and we had valid data from 539 (444 pre-training intervention and 280 post training). Some contributed more than one set of data.

Service user participants

Just over half of service user participants were unmarried white men. Nearly three-quarters were not in employment and had experienced multiple previous admissions to inpatient care. Table 16 shows a full description of the sample.

Staff participants

Table 17 describes characteristics at first point of entry to the study. For continuous variables (age and length of employment) these are mean averages of the variable at first point of entry to the study.

The design of study allowed staff to be part of the study on more than one occasion. The majority only took part once, but Table 18 shows the breakdown of how many staff contributed more than once.

On average, 51.6 nurse training sessions took place on each of the wards. This includes some nurses who attended multiple sessions.

Wards

Sixteen wards were included in this trial. Five wards were men’s wards, five were for women and six were mixed. All baseline data reported in this section were collected during the first data collection period for each ward. We used the WAS to record the views of one or two senior staff of their wards and took an average of their responses as our measure of the ward atmosphere (Table 19).

Outcomes and estimation

Did service users perceive a change in the ward following the introduction of activities?

High scores on VOICE are indicative of a poorer view of the ward. The mean VOICE score was 56.5 in the pre-intervention wards and 54.2 for the post-intervention wards, suggesting an improvement. However, an investigation of service user perceptions of the therapeutic environment showed no improvement with introduction of the intervention. A regression model adjusting for only ward and time estimated the standardised intervention benefit as 0.19 (mean VOICE score pre-intervention = 56.5, SD = 19.1, n = 644; mean VOICE score post-intervention = 54.2, SD = 17.2, n = 414). The only confounder identified was legal status (an a priori moderator) and the adjusted model (Table 20) provides weak evidence for benefit (standardised effect –0.18, 95% CI 0.38 improvement to 0.01 deterioration; p = 0.062). We found two other effects (independent of treatment or ward); a deterioration in VOICE score over time by 0.06 SD per month (95% CI 0.01 to 0.12; p = 0.021) and, over the whole trial, voluntary patients were more positive about the ward environment than involuntary patients by 0.27 SD (95% CI –0.40 to –0.15; p < 0.0001).

We also investigated the effect of legal status on admission (voluntary vs. involuntary) and other potential moderators. We found a significant interaction only with legal status (p = 0.006), with good evidence that the intervention improves VOICE scores of people admitted involuntarily [standardised improvement of –0.35 (95% CI –0.12 to –0.57; p = 0.002)]. Among people in hospital voluntarily, we found no evidence of an intervention effect (standardised effect = –0.01, 95% CI –0.23 to 0.22; p = 0.955).

We explored the components of the perceptions of the therapeutic environment by performing analyses on the two subscales of the VOICE (security and care subscales). The mean score on the security subscale was 25.0 for pre-intervention wards and 23.6 for intervention wards. Results from the adjusted analysis showed that the average intervention effect was an improvement on the security subscale of 0.27 SD, although this was not statistically significant (95% CI –0.53 to 0.00; p = 0.05). A similar effect with time was found in this model as for the primary analysis of the VOICE total scale.

The mean score for the second VOICE subscale (care) was 27.4 for pre-intervention wards and 26.6 on intervention wards. The analysis of this outcome provided no evidence of an improvement in the intervention arm (average reduction in intervention arm 0.18 SDs, 95% CI –0.37 to 0.01; p = 0.07). This model included a linear effect with calendar time, although no statistically significant trend was found.

Post-intervention focus groups

The purpose of the focus groups was to find out how service users perceived the intervention. Most participants were not aware that there was a programme of activities, although some could identify particular therapies. Participants told us that often the planned groups did not take place. For those groups that they could identify, there were mixed views. Some found the groups useful, especially in filling up the day. Conversely, some said the groups were repetitive or even ‘childish’. Service users also reported that the quality and style of the activities was dependent on specific nurses. They felt some staff were more knowledgeable or better able to engage with them than others.

What effects did activities have on staff views of the wards?

A total of 677 VOTE scores were from 428 individuals on pre-training wards (mean 68.2, SD = 12.0) and 398 scores were from 271 individuals on post-training wards (mean 70.3, SD = 12.7). Results from the regression model showed no evidence of a change in VOTE scores [standardised effect size = 0.04, 95% CI –0.09 to 0.18; p = 0.54 (see Table 21)].

Our exploratory analysis examined effects for staff subgroups and suggested a difference between men and women, but not for ethnicity. The model including an interaction between trial arm and gender estimated a negative effect for men (0.19 SD, 95% CI 0.01 to 0.36; p = 0.02) and a non-significant benefit for women (0.05 SD 95% CI –0.20 to 0.11; p = 0.55).

The VOTE score consists of three subscales, measuring workload intensity, team dynamics and interaction anxiety. Using the same model as above, workload intensity mean score prior to training was 31.0 (SD = 6.4) and 32.2 (SD = 7.7) post training. Lower scores indicate poorer perception of workload intensity. A regression-adjusted time and ward and the confounding factor of length of employment provided no evidence (p = 0.78) that the training altered nurses’ views on their workload intensity. The estimated intervention effect was 0.02 on a standardised scale (95% CI –0.13 to 0.18), representing a slight worsening of VOTE scores on post-intervention wards (see Table 21).

Team dynamics mean score at pre-intervention was 17.7 (SD = 5.3) and 18.2 (SD = 5.5) post training. Higher scores again indicate deterioration in the perception of team dynamics. Using a regression model adjusted for the effect of age we found no evidence (p = 0.34) that the training led to an improvement in nurses’ perceptions of team dynamics. The estimate of effect was a deterioration of 0.08 SDs (95% CI –0.08 to 0.23) (see Table 21).

Interaction anxiety mean score prior to training was 18.6 (SD = 4.4), and 18.5 (SD = 4.5) post training. Higher scores indicate deterioration. The regression model adjusted for nurses’ age found no evidence (p = 0.99) that training changed staff perceptions of interaction anxiety. The estimated effect was a worsening of 0.001 SDs (95% CI –0.15 to 0.15; see Table 21).

Post-intervention focus groups

Nurses reported that, overall, they enjoyed running the activities and appreciated the opportunity to feel that they were doing something therapeutic rather than paperwork and crisis management. They did report that this was hard to consistently carry out, in particular when high numbers of the staff were temporary staff or there were not enough staff on duty. They did not feel that they had the necessary organisational support to do the best job. They also said that they felt that the activities were somewhat repetitious, that is, service users sometimes got bored of the activities on offer, especially when they were in hospital for long periods of time. They said that they often felt bored running the same groups, but appreciated that being trained in more numerous activities would not have been possible. They also said that service users had repeatedly reported that they would prefer to personalise the groups (e.g. bring real-life problems to the problem-solving groups rather than hypothetical ones). Overall, they had a positive view of the activities but practicalities often got in the way.

Costs and cost-effectiveness

What is the effect of the intervention on the cost of perceived contacts?

We estimated that the total cost of the intervention across the 16 wards was approximately £156,000 which amounted to a cost of £10 per patient-week given an average number of 18 patients per ward and an average post-intervention follow-up of 55 weeks (Table 22). About 80% of this cost was attributable to the opportunity cost of nurses attending the training sessions. The average cost of perceived staff contacts over the course of a week was £259 before the intervention and £242 after the intervention. Our regression analysis suggested that the intervention increased perceived costs by £12 (95% CI –76 to 98; p-value = 0.774) but this difference was not statistically significant (Table 23).

Figure 6 shows the uncertainty surrounding the effects of the intervention on cost and VOICE estimates in combination. We estimated that the intervention most likely improved VOICE scores but increased costs of service contacts at the same time (i.e. 50% of our bootstrap replicates were in the north-west quadrant). It was unclear what the value of increasing the score by one unit was but, even at a willingness to pay of £100 per improvement in VOICE, the probability that the intervention was cost-effective was low (Figure 7). In the cost-effectiveness analysis using VOTE, results differed somewhat from the previously reported separate estimations because to combine these two end points, the analysis had to be undertaken at the unit level. In the cost-effectiveness acceptability plane using VOTE, the probability that the intervention led to deterioration of staff perceptions and reductions in costs was highest with 41% followed by the probability that the intervention would lead to deterioration in VOTE and an increase in cost with 29% (Figure 8). Regardless of the willingness to pay for improvements in VOTE, our analysis suggested that, with a probability of being cost-effective of 40%, it was unclear whether or not the intervention was cost-effective if staff perception was the only relevant outcome measure (Figure 9).

FIGURE 6 Cost-effectiveness plane using VOICE with 95% confidence ellipse (WP3). (Figure not reproduced here; see the original article.)

FIGURE 7 Cost-effectiveness acceptability curve using VOICE (WP3). (Figure not reproduced here; see the original article.)

FIGURE 8 Cost-effectiveness plane using VOTE with 95% confidence ellipse (WP3). (Figure not reproduced here; see the original article.)

FIGURE 9 Cost-effectiveness acceptability curve using VOTE (WP3). (Figure not reproduced here; see the original article.)

Ancillary analyses

Service user measures

A total of 1032 patients completed the SSS-RES [625 on pre-intervention wards (mean 91.3, SD = 27.1) and 407 on post-intervention wards (mean 86.4, SD = 24.2)]. A linear regression model suggested an intervention benefit of 4.15 points (95% CI –9.22 to 0.92; p = 0.109). As with the VOICE measure, there is good evidence (p = 0.005) for an interaction effect with legal status. For those who are compelled to accept treatment, the intervention benefit was estimated as –8.44 (95% CI –14.36 to –2.52; p = 0.005), but no evidence of a treatment effect in voluntary patients (0.61; 95% CI –5.39 to 6.60; p = 0.842).

Violence

The overall level of violence recorded was low. A total of 67 violent incidents were recorded during the trial; 40 of these occurred on control wards and 27 on post-intervention wards. Accounting for the number of patients on these wards gives a naive rate ratio of 0.68.

Using a Poisson regression model adjusted for the effects of ward and time we found that the intervention reduced the rate of violent incidents by an estimated 14%. However, there is no evidence (p = 0.73) that this result is not due to chance. The 95% CI for this estimate is very wide, ranging from a 63% reduction to a 99% increase. We found there was only a slight time trend, with an estimated 2% increase in the rate of incidents every 6 months. Again, there was a wide 95% CI for this estimate, ranging from a 26% reduction to a 40% increase. We did not find any covariates that met our criteria for confounding (Tables 24 and 25).

Staff measures

Index of Work Satisfaction

The mean IWS score was 160.4 for pre-intervention wards and 165.3 for intervention wards. The analysis, adjusted for time and ward, found no evidence of a change following the intervention (average standardised intervention effect = 0.02, 95% CI –0.11 to 0.16) (see Table 26).

Maslach Burnout Inventory

The mean MBI was 39.5 and 40.7 for pre-intervention and intervention wards respectively. The analysis adjusted by staff age showed no evidence of a significant difference in burnout following the intervention (standardised average intervention effect = –0.09, 95% CI –0.24 to 0.06) (see Table 26).

We explored the specific components of the MBI scale to detect differential change with time. The MBI consists of three subscales: emotional exhaustion, depersonalisation and personal accomplishment. Personal accomplishment is interpreted in the opposite direction to the first two (i.e. higher scores denote improvement).

For the factors in the MBI we compared the factors with the norms for mental health staff in the MBI Handbook.49 We found that emotional exhaustion (EE) was higher both before and after intervention (pre intervention 21.5 and post intervention 22.3, compared with the MBI handbook, 16.89). Depersonalisation was about the same, but personal accomplishment (PA) was higher (i.e. better) (pre intervention 35.5 and post intervention 35.2, compared with the MBI Handbook, 30.87). However, no factors were affected by the training (see the analysis results in Table 26).

Did we increase the number of ward activities?

According to data obtained from ward records, the mean number of activities before the intervention was 6.3, whereas after the intervention it was 7.8. The regression analysis yielded a mean treatment effect estimate of 1.5 (95% CI –0.4 to 3.4; p = 0.12). Before the intervention, the average number of participants in activities was 28 and it was 39 after the intervention. The average treatment effect in the regression model was 6.3 (95% CI –4.1 to 16.6; p = 0.23). The average number of activities that service users reported to have attended before the intervention was higher than after the intervention (1.91 vs. 1.89) but, after adjusting for ward and calendar effects, there was a trend towards a positive effect on the number of activities attended (coefficient 0.58, 95% CI –0.01 to 1.21; p = 0.06). Women were more likely to report having taken part in activities but adjusting for the effect of gender in the regression led only to a marginal change in the treatment effect estimate (coefficient 0.59, 95% CI –0.02 to 1.22; p = 0.59). Although the average number of activities attended over 1 week was lower after the intervention than before the intervention (3.18 vs. 3.38), our regression model suggested that the number of sessions attended increased by 0.68 (95% CI –0.67 to 2.13; p = 0.71) after adjusting for ward and calendar effects. This difference was not statistically significant (Tables 27–29).

Adverse events

No adverse events were reported as a result of the intervention.

Discussion

Summary

The DOORWAYS study sought to improve the perceptions of the ward environment among those living and working on acute inpatient mental health wards by training staff to provide activities and increase the number and quality of therapeutic activities taking place on these wards. For service users, the measured perceptions of the environment moved in the direction expected and, for those who were admitted compulsorily, this was significant. But nursing staff had more complex views on the inpatient environment. We also found that we were able to make an impact on the day-to-day life of the wards in a number of ways. However, there was only limited indication that the intervention was cost-effective at the service user level (depending on how much a unit improvement in the VOICE is valued) and even less likelihood that the intervention was cost-effective at the ward level with regard to the staff measures, especially our main outcome, VOTE.

Impact on wards

The associations between social interaction and taking part in therapeutic activities and their impact on service users’ improvement is not a new one.11 Not only do activities break up the monotony of time spent on wards,10 they also provide a forum in which service users can interact with each other, which we know is important.34 Therefore, we attempted to increase the activities on the wards and had moderate success in this. As the intervention consisted of training nurses to deliver activities, we had no control over the actual number of sessions that were actually run, apart from the ones the psychologist was involved in. In spite of this, we were successful in increasing the average number of activities on the 16 wards in this study. All of these wards already had a set weekly activity schedule so it is likely that some of the existing activities were dropped and replaced by the ones in this study. This may be one reason why our improvement of an average of two extra activities a week was not statistically significant. Sainsbury’s Centre for Mental Health16 specifically recommended that more evidence-based therapeutic activities are run on wards, while NICE158 recommends psychological talking therapies be offered to those with psychosis. Although very much valued,159 the majority of groups already being run on the ward were activities such as cookery or bingo, and so we believe that the introduction of evidence-based therapeutic activities was a positive one. The cost of the intervention was small; spending per patient did not increase and, in fact, may have decreased over time as budgets were cut. However, the number of contacts and activities did increase, which service users considered to be valuable. Furthermore, it is probably true, as others have found,40,41 that administrative duties and crisis management often means that therapeutic activities are relegated to the bottom of the ward’s to-do list. Service users in the focus groups recognised this:

Sometimes the nursing staff, they’re quite occupied with what they’re doing around the unit. They don’t really have much time, they very rarely have much time to themselves to tell you the truth, they probably have a break once in the shift.

It is perhaps not surprising that our study was not able to singlehandedly change this. In light of that, increasing the mean number of activities from six to eight can be considered a success.

One of the results of the increased pressure on inpatient wards is the tinderbox atmosphere that is associated with violence.8,9 During the period involved in the study, we recorded a total of 67 incidents of violence. Forty of these were on non-intervention wards while 27 were on intervention wards. We estimated this to be a 14% reduction in the rate of violence on intervention wards. Although we cannot claim statistical significance for this result, changes in relatively rare events are difficult to find and the study was not powered for this secondary outcome. Viewed in this light, and knowing about the background service pressures, this was a positive outcome and is indicative of an improved atmosphere on the wards following our intervention.

Impact on service users

The primary aim of this study was to improve the experiences of service users, which has been highlighted in many recent reports as vital in evaluating the performance of a service (e.g. Francis18 and CQC156). Our study, which began in 2009, was ahead of this recent focus. Although both perceptions and satisfaction of those using services moved in the right direction, we did not reach the generally accepted threshold of statistical significance for all those on the ward and so we cannot rule out the role of chance in producing these differences. But we were able to show improvements for those who are admitted under a legal section. We chose to investigate longer-term effects as these seem to have more relevance for the NHS, whereas other studies of changes in inpatient care have tended to investigate shorter-term effects, such as a few weeks. As wards aim to discharge service users as soon as they are able to do so, service users do not typically spend 6 months on the ward and so it was not possible to use repeated measures to assess their views of how activities had improved the environment, but this would have increased the power to detect small but consistent effects. We also only analysed the intention-to-treat effects of providing an intervention (training staff) on the outcome (change in perceptions of the ward). This is shown as the link between the intervention and the effects in green in the model (Figure 10). However, the effects may not be direct and the model shows other potential linkages that we might investigate through our database in the future. For instance, the effects of the intervention might be mediated by the effect on the number of interventions that are provided. Moderators include the turbulence on wards, such as the changes we noticed in ward leadership.

FIGURE 10 Model of potential relationships between intervention and outcome (WP3). (Figure not reproduced here; see the original article.)

Furthermore, this study was set against a backdrop of budget cuts and service reorganisations, which clearly had an impact on the wards, producing variability and a secular time trend. Our study also did not have sufficient power as it was not possible for us to collect data from all wards at all time points because we extended the study. Running such a complex trial on a large number of wards is a significant logistical challenge and will always present serious practical and analytical difficulties to researchers.

Our baseline analyses169 found that both activity and one-to-one sessions were associated with service users’ perceptions of the quality of the care they were receiving. These findings demonstrate that even those on an acute ward can engage in and appreciate the activities provided and the meaningful interactions they can have with staff, which is reflected in their perceptions of the ward. An often repeated justification for not running activities in the first place is that people are too ill to take part. Our study proved that this is not the case and that symptom severity was not associated with taking part in fewer activities.169 Furthermore, our post-intervention focus group supported this finding:

I found the recognition one with the faces where you start off with the blank face and the face changes quite interesting because it shows that you can’t judge someone by a first look especially in a place like this because there’s lots of emotions and feelings going on in the place so it’s actually to get to know the person behind the, what they, I found that quite interesting.

Nursing staff can also take heart in this finding that their efforts are recognised, appreciated and valued.

Feedback from our focus groups showed that many service users were not aware of the activities taking place. This could be for a number of reasons: a lack of clearly displayed ward timetables, no introduction to the ward on admission, no mention of it in the community meetings, and no one inviting them to take part. Furthermore, both service users and staff reported that activities were sometimes cancelled at the last minute, but our data still suggests, on average, an increase in activities over time despite these cancellations. There was also some feedback that the activities, or the way in which they were run, could have been modified. Both groups reported that the activities became boring over time with repetition:

Yeah they repeat the same things again. That’s the only thing that I find difficult. They say ‘oh DOORWAYS meeting’ and I think oh what’s the point in going to a DOORWAYS meeting when it’s the same thing as last week. So there needs to be a bit more of a change going on.

An alternating schedule of activities planned over several weeks might have overcome this problem. Service users also reported that some activities were childish or like being in school again, so perhaps the complexity/level of the activities needs to be raised.

Impact on nursing staff

The results from the nursing staff were more complex. In spite of DOORWAYS requiring nurses to engage and interact with service users more frequently, there were no effects on interaction anxiety, as measured by the VOTE subscale. Nurses on the wards in the post-intervention focus groups reported that their confidence was growing following training. In the post-DOORWAYS focus group, a nurse described how they felt after our project finished:

Because we were a bit unsure of ourselves . . . But I think once we got into it we didn’t have a problem at all, we got more and more confident and it’s not a problem anymore to, and I think it’s helped us to run other groups because we’re more confident in that so now we can run other groups as well without any problem.

We did find that workload intensity, job satisfaction and overall VOTE scores worsened slightly but not significantly and therefore opposite to our hypothesis of an improvement. Much of this effect we consider to be a reflection of the changes in the local NHS services during the study. Three out of the four catchment areas in our study reduced the number of wards and budgets were also reduced. In one case, staff had to reapply for their own jobs and some wards did not have ward managers but were run by junior staff or overseen on a part-time basis only. In light of this turbulent background, it is not surprising that being asked to deliver more activities had an effect on their view of the ward. This consequence was also reflected in an increase in the MBI emotional exhaustion scale. However, it is interesting that there was no effect on the personal accomplishment and depersonalisation scales of the MBI, or on the interaction anxiety measure of VOTE. Therefore, the effects, although important, are not solely explained by the increases in activities. Pragmatic health services research needs to consider such issues in terms of interpreting significant (or non-significant) effects. A similar issue is reported in Chapter 5 when we discuss that even in the absence of any intervention, the view of the services deteriorated over time. However, these staff findings may also be a feature of a clustered trial; with relatively few numbers of clusters and a large variability in the situation of each cluster it is always possible for results to be shifted by extreme conditions on a small number of clusters. In other words, a few events or circumstances negative enough on two wards would be enough to push the results into significance. One of our future aims is to interrogate the data to see if there was a consistent perception on all wards.

Costs and cost-effectiveness

The intervention required investment in training and, when apportioned over the wards and service users, this amounted to £10 per patient-week. The cost of meaningful service contacts and activities increased (non-significantly) by £12 per patient-week following the intervention (including the intervention cost). This does not reflect an actual increase in expenditure but rather a relative increase in resources from a fixed amount. For example, if the actual cost per patient-week of an inpatient stay is £2100 (7 days at £300 per day), then £12 per week extra represents a shift of 0.5% towards per direct patient care. The cost-effectiveness acceptability curve shows that an improvement in the VOICE would have to be valued very highly in order for the intervention to be cost-effective. It was unclear whether or not the intervention could be regarded as cost-effective using VOTE.

Strengths and limitations of this study

A major strength of this study was the use of the stakeholder perspective in the main outcomes from the service users and staff members. No study to date, to our knowledge, has had such a focus. Although entirely well intentioned and very much welcomed, many of the strategies and programmes aimed at improving inpatient services are developed by senior management or bodies such as the Royal College of Psychiatrists.161 We involved the service users and nurses using the wards on a day-to-day basis and ensured this programme delivered what mattered to them.

There were a number of reasons behind our decision to use a stepped-wedge design for this investigation. First, as the intervention was at the ward level, a cluster-based design was clearly required in this context. Second, a stepped-wedge design allows nursing staff on all wards to receive the training. This is in distinction to a regular cluster randomised trial in which only half of the clusters would be allocated to the intervention arm. Third, it was not possible for us to train all wards at the same time owing to limits in the number of researchers and psychologists employed. By staggering the timing of the intervention, we could reach more wards with the same number of researchers.

The economic evaluation was conducted from the perspective of the service user with their perception of meaningful contacts/activities being the driver of costs. Service user data relied on self-report and this may have some limitations in terms of recall accuracy. However, we have assumed that if the contact was meaningful then it would be recalled.

Research implications

Future research needs to use the tools of implementation science to uncover the factors that might predict successful implementation. Although we were able to show that activities had increased and we have anecdotal evidence of where it proved to be relatively successful, we also know that there was much variance unaccounted for. We will be investigating some of these exploratory hypotheses on our current data set to best suggest the remit for future studies. A better understanding is needed of the strategies for knowledge integration, especially in high throughput inpatient settings. We also plan to explore differences between wards with special reference to their culture and those differences produced by boroughs and trusts. We will also be carrying out further analyses investigating the impact of the intervention on clinical indicators such as PANSS scores, NOSIE and GAF. The results from the economic evaluation indicate the shift that is likely towards more direct care provision and this approach should be developed in further studies.

Clinical implications

One of the most common complaints about inpatient services is the extreme boredom and lack of therapeutic activities occurring on the wards (e.g. Walsh and Boyle34 and CQC170). We successfully increased the number of activities and received positive feedback from those participating in the study, demonstrating that, with effort and dedication, it is possible to increase the number of activities. This was carried out with little increase in the costs and showed a transfer of costs to more patient valued care. Although there was variation in terms of implementation success, it should be possible to adopt our design to increase the skill and confidence of nursing staff, which has often been found to be perceived as poor.29 This alone would be a positive outcome, not well captured in our current study but revealed in our qualitative analysis from the focus groups. Our baseline analyses also showed that symptom severity did not have an impact on the uptake of activities and participation in activities was related to more positive perceptions of the wards, demonstrating that they are a much valued component of inpatient services regardless of illness severity. This belies the belief that the acutely ill cannot take part in meaningful activities and also supports the view that more therapeutic activities could be of value and are appreciated.

Conclusions

This programme of research had some positive and some surprising results. Service users’ perceptions of their care were moving in the right direction and, for those who were admitted compulsorily, there was a significant effect. Nurses appear to be working in such a pressured system that their perceptions were mixed in spite of the positive views of our project discussed in the focus group. The number of activities taking place and the number of violent incidents also moved in the right direction. A much larger study of this kind is needed in order to detect statistically significant effects, as power was low even after we increased the number of wards.