The PERCEIVE programme makes an important contribution to the evaluation of inpatient mental health services. Mental health services in the UK are in crisis with budgetary cuts, pressure on beds, staff demoralisation and service user dissatisfaction (e.g. Mind10 and Clegg185). In light of the recent Francis report,18 improving the experience of service users is of upmost importance across the entirety of the NHS. The overarching aim of the project was to evaluate efforts to improve the service user experience using measures designed by service user researchers and nurses.
Therefore, the PERCEIVE programme had several components. First in WP1 LIAISE, we developed measures of stakeholder perceptions of services for use across the programme through an iterative process involving service users and nurse researchers in their design and including feedback from service users and nurses who live and work on the wards throughout the development. We also produced for the first time, in WP2 CITRINE, a measure of patient-level costs that also involved all stakeholders. In this way, we were able to ensure that what is important to service users and nurses are the questions being asked, not what an expert researcher or clinician judges as the important questions to ask. These measures are now available for future evaluations of health services.
Second, in response to the numerous reports about how service users are dissatisfied with the lack therapeutic activities and interactions on inpatient wards, the intense boredom and the lack of anything to do on inpatient wards, our WP3 DOORWAYS study aimed to introduce more therapeutic activities and interactions onto wards. With trust management and ward staff involvement, we developed a comprehensive training programme that was unique to each ward’s specific needs in order to give the nurses the skills and confidence to run evidence-based group activities. We then evaluated the feasibility of this training programme and its real-life impact on wards against a changing background of trust upheaval and reorganisation and further budgetary cuts.
Finally, in WP4 BETTER PATHWAYS, we evaluated a novel approach to deal with the pressure on wards to reduce length of stay and have enough beds for those who need them. In a pioneering move, one catchment area in our trust operated a novel system of admission and discharge, modelled on a system used in accident and emergencies around the world. The inpatient service has senior clinicians on duty 7 days a week and service users are thoroughly evaluated on arrival and discharge planning begins immediately so that the individual is treated in the way most appropriate to his or her needs. Prior to our study, the only assessment had been a pre–post evaluation, which is fraught with problems of interpretation. PERCEIVE designed the evaluation of this triage model with a rigorous comparative design. PERCEIVE evaluated stakeholders’ perceptions as well as the economic benefits or drawbacks and also whether or not it was working, as it was designed when it started originally a decade earlier.
Summary of results
Excerpts from Wykes et al.186 This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
This programme of work involved four distinct but interrelated studies.
LIAISE, the first WP, involved the development of two measures of the perception of living on, and working in, inpatient wards from the perspective of both service users (VOICE) and nurses (VOTE). These measures were developed by a service user researcher and nurse researcher as we believe (and there is evidence that) individuals in focus groups and reference groups are more comfortable when the researcher is from the same community and that they are more candid in their views. These measures were then psychometrically validated as part of the final two WPs.
The second WP, CITRINE, developed a self-report measure of staff contacts and the uptake of therapeutic activities on inpatient wards, which was subsequently used to calculate the costs of therapeutic activities and service user and staff interactions in WPs 3 and 4. Clinical staff and service users were consulted in the process of the questionnaire’s development. It was piloted to determine validity, reliability and acceptance from service users. The final questionnaire records, through self-report, service users’ activities and meaningful interactions with ward staff. Self-report is important as this project sought to carry out assessments from the service user perspective and particularly what they viewed as meaningful. The concept of what constitutes a meaningful contact was much debated in the early stages of the study as it was found that service users reported fewer staff interactions as meaningful compared with the actual number (not assessed as being meaningful or not) that was observed.
DOORWAYS, the third WP, sought to increase the number of therapeutic activities service users could potentially take part in. Ward staff often report that crisis resolution, the paperwork and documentation they must complete and a lack of confidence, or a loss of skills over time, often prevent them from engaging in a therapeutic way with service users. This WP attempted to change this. First, we did this by attempting to provide the majority of ward staff with training in how to communicate effectively. This training was cofacilitated by a service user in order to present communication skills from a service user perspective. Second, clinical psychologists ran training sessions and provided in situ training in a number of evidence-based therapeutic activities. That our intervention proved less effective than we anticipated (although there was a significant improvement in people compulsorily admitted) may have much to do with wider changes in the services that were outside the control of the research. Much of the success of our approach was predicated on stable ward management and ethos specific to each ward. We found that the support of the ward managers was absolutely vital to the implementation of DOORWAYS. In several instances, wards did not have managers, were in the process of shutting completely or had job reorganisations that had an impact on whether or not our project was able to make a meaningful difference, despite the best efforts of the nursing staff.
In the final WP, BETTER, we sought to evaluate the cost-effectiveness and stakeholder perceptions of a novel method of admission and discharge. A triage service aims to facilitate the quick transition to the appropriate type of care service that users will benefit from; therefore, improving stakeholder perceptions and reducing hospital stays and costs. Our study did not provide evidence to support the premise that this was a more efficient system. However, there were a number of contextual reasons why the service did not fulfil its earlier promise. Across the entire trust, all catchment area mental health services were facing budgetary cuts and service reorganisation. During this turbulent time, the contrasting routine system may have focused on reducing admission length and did reduce the number of available beds, hence, the contrasting system was also processing admissions and discharges in a more ‘triage like’ way.
Patient and public involvement in the PERCEIVE study
The PERCEIVE study aimed to put service user involvement at the heart of increasing the evidence base on improving care and the experience of people using adult acute mental health services in the NHS.187 The importance of the experience of NHS services is now a policy priority in the wake of the Francis report18 into the failings of Mid Staffordshire Hospitals NHS Trust. At the same time, the PERCEIVE programme sought to assess whether increasing activities and psychological interventions on wards or changes in the service delivery improved both clinical outcomes and experience. Research suggests that these are priorities for service users.20
There is an overall emphasis on service user involvement as part of the NIHR application process with clear descriptions of the intended involvement throughout the programme. However, there is little information available on whether or not the research actually managed to achieve this involvement. As service user and other stakeholder involvement was particularly emphasised in this programme grant, not just in the governance of the programme but also in the research objectives and the conduct of the research itself, the following sections describe how we achieved this across the programme grant as a whole and as far as possible assesses the impact of this involvement.
Service user involvement in the governance structure
Service users were members of the Steering Committee and the Data Monitoring and Ethics Committee. However, their attendance was poor and although each meeting did have service user involvement, this could have been improved. We will, in the future, encourage attendance through a more proactive approach and organise meetings at times more suitable to service users (e.g. evenings or weekends).
Design of the PERCEIVE programme
The programme had a coapplicant who is a service user researcher (DR) and has experience of psychiatric inpatient care including of detention and compulsory treatment. However, it should not be thought that what is being brought to the table is some idiosyncratic ‘personal’ experience. Having used inpatient services for four decades and also having a research career, DR has both conceptualised her own experience in terms of the research literature and conceptualised the research literature in terms of her experience and, crucially, others who share it. Her understanding is scientifically and socially shaped.
DR led WP1 (LIAISE), which consisted of developing outcome measures from the perspectives of service users and nurses. To do this, LIAISE utilised a model which DR has used in developing patient-generated PROMs since 200717 and she also contributed to the design of the other three WPs in terms of the research priorities of service users and the kinds of interventions that they would find acceptable and helpful. She helped in the advancement of the conceptualisation of DOORWAYS (WP3) as not just an increase in activity and psychological therapy but a whole-ward enhancing intervention that might be expected to lead to a better patient experience. She also supplied a dose of common sense in terms of the difficulties that could be expected and resistance to organisational change in the NHS, particularly in mental health services.
Service user involvement in measure development
Work package 1 developed a measure from the service user perspective. It was a user-led piece of work and the measure is called VOICE. It also set out to develop a measure from the nurse perspective (VOTE). These measures were not measures of ‘satisfaction’ as they are fraught with difficulties,104 but aimed to tap directly into experiences and perceptions of acute care. For the service user measure, we employed a service user researcher. The first phase followed a participatory approach in that both participants and researchers had experience of service use. It began with four focus groups made up of people who had been inpatients in the local trust in the previous 2 years, with one group made up exclusively of people who had been detained. However, in the event, all groups included people who had been detained. The four focus groups each met twice and only one person across all four groups did not return for the second meeting. This indicates that they were very engaged with the process and it might be argued that this was because it was user led. The details of the rest of the measure generation can be found in Chapter 2. One further point worth noting is that the measure was subject to full psychometric testing and was found to be robust. This is interesting as the population on whom it was tested was an inpatient population and standard textbooks suggest that the ‘subjects’ for psychometric testing should be ‘cognitively intact’. Many of our participants had diagnoses of psychosis which might be thought to compromise them in this task and so, again, it may be the fact of being user generated that makes the measure amenable even to people who have poor mental health because it is about things that really matter to them at that moment.
In order to further assess the impact of service user involvement on research, our programme was included as part of a study carried out by the NIHR Mental Health Research Network. The service user researcher from the LIAISE WP was interviewed by an independent evaluator for studies adopted by the Mental Health Research Network (MHRN). She had this to say:
There’s a huge amount of benefit to employing service user researchers. It’s very valuable to be able to say to service users this is who I am. When you’re running a group or an interview, it breaks down those traditional barriers. Imagine if you had a psychiatrist running a focus group asking people about issues on the ward – how open are they going to be about that? There is something about a shared experience that makes people feel more comfortable and more open and honest in what they say. I remember when I was talking to one participant she said ‘I feel like I can tell you everything, you understand me and I can really talk to you’. So I think it makes a massive difference.
Once VOICE was complete, it was used in WP3 (DOORWAYS), the trial of activities and psychological interventions, and WP4 (BETTER PATHWAYS), the triage ward study. The service user researcher collected data and there were other service users collecting data as well. The recruitment rates were no different between service user researchers and the rest of the recruitment team – all were uniformly high.
How does VOICE perform?
In the battery of measures for service users to complete, there was also a conventionally derived satisfaction one. In WP4 (BETTER PATHWAYS), VOICE was sensitive to differences in activities that participants were engaged in but the conventionally derived one was not. In WP3 (DOORWAYS), VOICE did not find that perceptions improved overall post interventions, which was to be expected as the number of activities taking place did not increase to a significant level. However, it was able to detect differences in those who were compulsorily detained.
Extending service user involvement to other stakeholders
INVOLVE, the programme set up by NIHR to promote patient and public involvement in research, states that mental health professionals do not count as the ‘public’ for involvement purposes. However, we did extend our model of measure development to nursing staff. We proceeded in exactly the same way as for VOICE to produce a measure named VOTE. In this case, we recruited nurse researchers to work with focus groups and expert panels of nursing staff and then to collect data on the wards. The participants in the groups and panels were current staff. We also employed a nurse researcher as part of the team of researchers collecting data for WPs 3 and 4.
CITRINE measure development
For the development of the measure of patient-level costs of inpatient care, CITRINE, we also included both service users and nurses. Service user researchers provided advice on the whole project and also collected data for reliability and validation exercises. In the measure development, staff members were interviewed about a variety of issues related to activities of nursing staff in relation to inpatients. Focus groups in the form of ward-based community meetings were held with service users in the initial measure development and to discuss the presentation of the self-report measure. Following advice from service users, the final measure was adapted in presentation to ensure that it would be suitable for as many people as possible by providing some support for completion (see Chapter 3 for details).
Service user involvement in data collection
In addition to the close involvement of stakeholders in all measure developments, service user researchers were also part of the research teams responsible for data collection in all other areas of the programme grant. These staff members were recruited on the same basis as all members of the team (for their professionalism and expertise as researchers) but also required the added expertise of experience as a service user.
It is difficult to assess their impact on the team, but a number of issues did arise for those service users who had received inpatient care. In particular, some of the trauma associated with their own admission reasserted itself for them. This was dealt with through close supervision that also included a senior service user researcher who also had experience of admission as well as a senior clinical psychologist.
Although service user involvement was seen by the research team as a benefit, there was resistance by some ward staff who were concerned that service users were able to access case notes (with participant consent). However, the professionalism of our staff and close supervision of the team was able to reduce the distress to our staff and overcome the ward staff resistance.
Service user impact on analyses
Service users were involved in the development of all the analysis plans and were especially involved in the analyses in WP1, in which they specifically discussed with statisticians the best approach to the factor analyses and the assessment of the measure psychometrics. Service users were particularly important in helping to understand the statistical output and often were able to fall back on the qualitative data we had collected as well as their own experience to ensure that the team understood the meaning of tables and also to lead the secondary or exploratory analyses by providing potential hypotheses to test. A final set of focus groups were carried out, facilitated by a service user researcher, with the aim of understanding if the inputs to WP3 had been helpful and if there were any other issues that needed consideration in the analysis or follow-up of the programme grant.
Service user impact on dissemination
In keeping with our ethos, all papers had at least one service user author. Service user researchers also presented data from the programme at local, national and international research meetings. During the programme, we kept stakeholders involved in our programme by updating them on the progress through newsletters for which service users contributed and acted as editors. A further round of dissemination is planned when the current programme is completed.
Interim feedback has also been delivered and is planned to be delivered to service managers in order to ensure that the programme provides useful local data for service planning. The first of these meetings was with the Medical Director, Clinical Service Directors and Clinician Lead for the inpatient services to feedback the data from WP4. This feedback also allowed these stakeholders to suggest new analyses for future data exploration which may be of use locally. We have also planned similar exercises with service managers and ward managers as well as local service users.
Service user impact on protocol developments (add-on studies)
During the process of carrying out the programme grant protocol and as we analysed its results, other issues emerged. These became the focus of add-on studies.
Design in mind
We noted during the research that the design of wards often had an effect not only on the overall ambience and atmosphere of the ward but also the activities that it could support. Following an extensive review, it became clear that there was no hard evidence of these effects that could be used as part of our programme measures. Therefore, we undertook a further development in a user co-led study to develop a measure of mental health ward design using a participatory method and involving service users and staff. Following this development, a novel method was developed and tested in a study involving service user photographs of ward design. We asked service users to take photographs of their best and least liked features of the ward design and then we uploaded these photographs and invited those who had taken them to speak about them. This method allows more service user involvement as it does not limit the scope of the designs or their potential impact and also allows service users to be involved who may not be able to articulate clearly their preferences and dislikes in a more abstract way. The impact of service users was, therefore, not only to provide intellectual input via the review process but also to develop methodology.
Perinatal VOICE
Similar impacts occurred in an extension of the VOICE measure to perinatal services that arose out of discussions with colleagues undertaking their own programme grant assessing perinatal mental health services. After discussions with their own service user researchers and advisors, it was decided that our project would assess if women using perinatal mental health services had different concerns and needs than those service users in our programme grant. As a result, a measure specifically to be used with women in the perinatal period was developed as a modification of VOICE. This is now being psychometrically tested online with the help of a perinatal non-governmental agency and the Service User Advisory Group of the perinatal programme grant.
Strengths and limitations
The primary strength of the PERCEIVE programme grant was that its focus was on the experience of those using and working in inpatient services. The vast majority of service evaluations and improvement schemes are based on what expert academic researchers or clinical managers put forward as worthy of investigation. There is a distinct lack of stakeholder involvement in deciding what factors and questions are important in service evaluation or how to measure them. An aim of our project was to deliver service user- and nurse-generated, psychometrically tested, tools to assess services. Our use of other established measures alongside them, which picked up on the same patterns as our measures, provided further validity.
Our project was run in services that were changing as we progressed, which meant that we were attempting to change and measure systems that were already in flux. As mentioned in Chapter 5, WP4 was proposed as a comparison of two distinct triage services. However, one of the boroughs had such a lag in converting their service into a triage service this change did not occur. Furthermore, our project took place during a time when, nationwide, services were being squeezed. Funds were cut and, in three of five services in the study, wards were being closed, which contributed to the pressure on beds as well staff morale (especially in cases where staff had to reapply for their posts). In addition, wards lost ward managers and were managed by part-time managers or less experienced staff acting as temporary managers. Adverse events, for example the unexpected death of valued member of staff, had an impact on both ward colleagues and the service users. All of these background variables contributed to the turbulence on wards and hence the result that perceptions of the services fluctuated for both groups of stakeholders.
Research is evaluated on the generalisability of findings. We were particularly concerned with our DOORWAYS study, in which we tried to increase the number of psychologically based activities on wards. Our initial trust base is part of a longstanding and nationally recognised academic NHS partnership and has recently become part of an academic science centre. The users of these services are research active. We also tested the effects in a different trust that has fewer academic ties. The culture and ethos of the second trust, as well as the demographics in the area it serves, are very different from the original trust. The fact that we did not find differing results between these trusts leads us to conclude that our results are generalisable to comparable trusts in large cities nationwide, regardless of their research activity.
The stepped-wedge design used in WP3 led to some challenges when analysing the data. The original proposal was designed for the initial eight wards, but our Data Monitoring and Ethics Committee suggested that we increase the numbers of wards (clusters) in order to improve generalisability and allow for more accurate modelling of the between-ward variation. However, although we did increase power (in terms of the numbers of patients), once the project reached the analysis phase it was clear that our modification of the design was not as effective as we had hoped. This was because of two features of the data. The first was that the assumption of random variation between wards, as required for some of the statistical models, did not seem to be very realistic (there was at least one apparent outlier, for example). Second, external changes over time were substantial and affected the wards in different ways, increasing their variability so that even the new total of 16 wards was likely to be too low for accurate modelling of the random effects. These problems affected our ability to detect only effects in service users compulsorily admitted above background ‘noise’. However, the design did have benefits over a simple cross-sectional comparison. Neither the wards nor the researchers knew which wards would be randomised next (with the exception, of course, of the last wards). Pre-intervention measures of the variables that were to be our key outcomes were measured by self-report prior to each randomisation so neither staff nor service users in the control wards knew whether or not they would receive the active treatment (staff training). However, as a result of the practicalities of the DOORWAYS trial, researchers could not be blind to which wards had already received the intervention, as they spent time on all the wards and would see the activities occurring.
Some practicalities of research could not be avoided. Three different psychologists were employed to carry out the training of ward staff, which may have led to a variation in the quality of training. It must be acknowledged that as the research progressed, implementation methods and engagement improved. Researchers also changed over time but any interview rating was ensured to be reliable against a key rater each time staff changed. The core management team remained the same throughout and aimed to provide consistent training to all staff. Despite these changes, there were no consistent fluctuations over time in any of the outcome measures that we could account for by our changes in staff or learning.
Research implications
Our creation and validation of two stakeholder measures provides our own team as well as other research teams with useful tools in evaluating inpatient services. Colleagues have already approached us about using these assessments in the north of England and the USA, and VOICE has even been translated into Portuguese. We fully encourage the use of these measures.
The CITRINE WP has identified a number of studies that have investigated how inpatient staff spend their time and, specifically, the extent to which service users are in receipt of that time. The overall finding is that staff–service user contact time is very limited. There were no economic evaluations of inpatient-based therapeutic activities identified. This is a clear research gap, especially given that so many resources are tied up in inpatient care. As well as the need for evaluations of new activities, there is also scope (using modelling methods for example) to evaluate existing interventions. We have developed a new tool for measuring staff contacts with service users and activities that the service users undertake. Use of this tool will provide useful economic information needed for evaluating inpatient-based interventions.
DOORWAYS found that implementing such a research project in an unstable and unpredictable environment is not without its challenges. The tools of implementation science need to be used to uncover the factors that might help future work in this area succeed. Our future work includes further exploring the differences between wards, boroughs and trusts. We will also be looking in greater detail at how the absence of management and significant events that destabilise wards interfere with the implementation of the intervention. We have not explored some of the data in great depth so will be carrying out further analyses investigating the impact of the intervention on clinical indicators such as PANSS scores, NOSIE and GAF. We will also be investigating if VOTE scores for those nurses who were trained as part of our programme are different from those nurses who, although on the wards at the time, were not trained.
The BETTER PATHWAYS WP investigates a triage system in one single site in comparison with a routine care system. Since we have completed this project, further sites in our trusts and across the nation have moved to a triage system. No two sites will ever run their triage services in exactly the same way and a comparison between many of these sites would be able to tease out those practices that are working well. What is holding up discharge from locality wards, such as social reasons like housing, would also be worthy of further research.
Clinical and organisational implications for mental health inpatient units
This programme demonstrates the valuable contribution that stakeholder research makes. Our consistently high recruitment rates demonstrate that people are eager to help create evidence for good practice. In light of the Francis report,188 trusts need to increasingly focus on stakeholder experience as a measure of service delivery.
Our study found that the higher the numbers of meaningful contacts that service users had with nurses, the better were their perceptions of services. So, despite the negative media presentation of services, including inpatient mental health services, those receiving such services appreciate quality communication with nurses. We also discovered that regardless of how acutely ill a person may be they are still able and, in fact desire, to take part in therapeutic activities. This is in opposition to the general concerns that staff had about whether or not service users in such acute stages of mental ill health were able to engage in therapeutic activities. These findings suggest that NHS leaders and service directors need to consider support structures for staff to create an ethos on wards where therapeutic activities and interactions are of equal importance to the completion of paperwork and the management of crises.
Increasing activities did not lead to a worsening of staff perceptions of services. This is clearly important as the extra burdens may have affected the quality of interactions with inpatients. This was also a time during which services across both trusts were being cut and our results are perhaps not surprising. As noted above, this finding demonstrates the importance of service director and manager support to ensure that wards are staffed, organised and run adequately not only for the benefit of the services users but also for the nurses themselves. This could also contribute to greater staff retention, which is an issue across the NHS. Nurses often fed back to our team that they would like to be able to run more activities and engage more with service users as those are the things that motivate one to choose a career in psychiatric nursing. The economic analyses revealed that there was a modest cost increase following the DOORWAYS intervention. This is not an expenditure increase but rather a redistribution of already allocated resources towards more direct care. There was limited suggestion of cost-effectiveness of DOORWAYs, but only if a high value is placed on a unit improvement in the VOICE. In terms of staff perceptions, the intervention did not appear cost-effective.
Our study did not find any clear advantage of a triage system over that of routine care. We found that the introduction of a triage system did not result in a reduction in the length of stay, nor were perceptions of the wards very different between the systems. However, it did not lead to an increase in re-admissions, as feared, and costs were not significantly different from non-triage services. On its own, the triage ward functions as it should (discharging people rapidly) and, as expected, the nurses did not report more burnout. What was unexpected was that staff on the locality wards are finding it very challenging. In the locality wards, nurses reported the most burnout and this needs further exploration. The triage system does not work in isolation and the effects on locality wards especially on what can improve staff moral would be a possible next step in its evaluation.
Conclusions
In the current climate of the NHS, it is becoming increasingly clear that the experience, particularly that of service users, is a vital component of the evaluation of the quality of services. Our study will contribute to quality evaluation by providing a methodology and a set of outcome measures that will strengthen these research efforts. However, all future studies need to understand that variation in inpatient care between wards, within wards and across time are likely to undermine the surfacing of clear results. Despite the best efforts of service directors and nurses working on wards, there is clearly room for further improvement to inpatient wards. Our research makes a valuable contribution in understanding how inpatient wards cope with a changing background and offers service planners useful information in how they can continue to try and improve service users’ experiences while also being cost-effective. We hope others will take forward our ideas and findings and that our large databases will provide fruitful ground for future investigations to aid our understanding of the factors that mediate and moderate ward interventions, their implementation and their sustainability.