Overview of results and interpretation
The Engager research programme has run over 7 years and has incorporated several substudies within the two phases and across the four workstreams. The results of the RCT inevitably dominate the findings. The primary outcome showed no difference between arms at 3, 6 or 12 months. The secondary outcomes measured in the trial showed a pattern of null effect. The small number of statistically significant changes were most likely due to multiple testing. There was no evidence for the Engager intervention being cost-effective.
Despite the development of a substantive IDP, implementation of the Engager intervention was suboptimal. Reasons for this included some individuals becoming disengaged from the intervention and delivery of intervention being affected by illness among practitioners, staff changes, conflicts within team cultures and temporary gaps in the practitioners’ capacity to deliver the intervention. Challenges in delivering in the prison setting (e.g. lockdowns, violence and staff cuts) and the well-evidenced underperformance of community rehabilitation companies are also likely to have had an impact on outcomes achieved by the Engager intervention.
The in-depth case studies broadly supported these findings, with suboptimal fidelity to the model and most intervention participants not achieving positive outcomes. However, when Engager was delivered optimally, a significant minority of participants engaged well with the intervention and made important steps towards rehabilitation. However, the positive outcomes achieved by individuals did not correlate with the standard measures when the case study outcomes and trial outcomes were compared after both analyses had been completed. These individual-level case studies also provided evidence that, although most components of the intervention were helpful for at least some individuals, there were also weaknesses in the intervention theory in that it had not proposed specific practitioner activities in relation to individuals who were overoptimistic or resigned in hopelessness.
Overall, we found evidence for three contributors to the neutral results, that is, suboptimal implementation, weak delivery in certain areas of the intervention and the inability of standard trial outcome measures to detect small unpredictable steps in recovery resultant from the intervention for individuals with lifelong experiences of adversity.
Reflections on achievements and challenges
Although we succeeded in many areas, such as recruitment, retention and our iterative intervention development, we are particularly proud of the work the academic researchers produced in partnership with the peer researchers.60
Phase 1: intervention development and trial science
The main output of phase 1 was the development of a coherent, acceptable and feasible complex intervention for prison leavers with common metal health problems. This iterative process, with realist-informed syntheses, offers other complex intervention developers a blueprint for how to build evidence-based programme theory and evaluate it. This detailed worked example is, to the best of our knowledge, one of few research studies that have used realist-informed approaches in the initial throes of the complex intervention development cycle advocated by the Medical Research Council. Box 3 summarises the key methods of intervention development used.
Our programme theory separated out the theory of what practitioners should deliver and the theory articulating how this should be achieved. For the latter, a substantive IDP was developed. The Engager manual provides evidence-based guidance for carrying out person-centred and integrated care for prison leavers. The rich detail in the manual provides agencies with descriptions of what supports to offer and also how the supports should be delivered to optimise engagement with different people in differing situations and circumstances. The IDP includes a set of measures detailing how practitioners should be supported throughout delivery, and this includes having a team supervisor, a meta-supervisor (i.e. a clinical lead) who takes overall responsibility for the intervention delivery and top-up training, and MBA top-up training and supervision. The manual, along with a detailed description of the products used to develop it (e.g. narratives, logic model), and our proposed changes, are available on the Engager website [URL: www.plymouth.ac.uk/research/primarycare/criminal-justice/engager (accessed 9 June 2022)].
We documented the developmental processes used to build a theory. Although significant efforts were made to ensure transparency, it was not possible to document all decisions made in the course of developing the theory for the intervention. Therefore, owing to the number of data identified and obtained, and the quality of those data, it is possible that some learning and/or relevant theories were missed, minimised or overemphasised.
Our trial science work was extensive and in-depth. We incorporated two consensus procedures, a review of the literature and a significant period of testing to select the most appropriate outcomes (a process not undertaken by many other programmes). Our pilot demonstrated feasibility in terms of recruitment, follow-up and trial procedures, and also highlighted that traditional gold-standard trial methods, such as blinding, can be difficult to manage within a prison context. However, this does not mean that trials of prison leavers should be avoided.
Phase 2: intervention evaluation
The delivery of the trial was a significant logistical achievement. High proportions of people eligible consented to screening and were recruited to the trial, enhancing generalisability. This achievement was based on our substantial feasibility work30 and piloting.61 Credit must be given to our motivated and dedicated team of researchers who worked assertively with flexible operating procedures to maintain engagement. We successfully followed up 66% of our participants at 6 months, and this is considered an impressive rate for this population; however, the rate is still lower than that of other community-based depression studies.25,26
There is currently very little literature on calculating QALYs for people in prison, and this is one of the first studies, to our knowledge, to collect patient-reported preference-based outcome measures in this population. We have collected costs across a range of sectors to provide a more complete picture of where costs fall for people with common mental health problems in prison. Our cost–consequence analysis allowed us to examine the impact further on in the patient care pathway, including facilitating access to services, which allowed us to examine the impact of improving access to planned care on unplanned costs, something not commonly carried out in economic evaluations.
The parallel mixed-method process evaluation, one of the most in-depth completed for a complex health intervention, was a significant success and was especially important given the neutral trial finding. The process evaluation not only provided evidence of breadth and depth, and from multiple perspectives about what was delivered to participants, but also allowed us to hone in on how team dynamics and underlying beliefs and values affected implementation. Subsequently, this allows us to propose what might be done to support practitioners further to optimise delivery. Documenting suboptimal implementation, especially of the MBA and the use of the ‘shared understanding’ formulations, was important for trial result interpretation and development of future practice.
The realist-informed methods allowed us to interrogate the intervention mechanisms by assessing if delivering the specified intervention components produced the hypothesised outcomes, and this gave us insight into how the intervention can have a sustained effect when delivered well. We showed how consistent delivery across time could lead to several mechanisms being activated, often repeatedly, to achieve incremental but sustainable change. The process evaluation also enabled us to identify areas that need refinement in the intervention theory, such as the tools to build a therapeutic relationship for those who might dismiss the need for support, as well as the importance of developing the therapeutic skills of practitioners delivering the intervention.
In the process evaluation, our analyses were limited by the completeness of data collection and longitudinal changes in the context of the intervention delivery across wide geographical areas. Completeness of data collection was particularly challenging for audio-recordings of practitioner/participant sessions and of supervision sessions. There was also a lack of data completeness in the practitioner time logs and session notes. Therefore, greater emphasis was placed on the content of retrospective interviews than we would have preferred.
The trial had some key limitations. Despite standard randomisation procedures being applied, there were imbalances between the two arms at baseline, which indicated that people in the Engager intervention arm were more socially excluded but less emotionally distressed. These differences were controlled for in the analysis; however, there is a chance that people in the Engager intervention arm may also have been more criminogenic, but, unfortunately, we were unable to access CJS data (e.g. offending history) from the Ministry of Justice to examine this (and control for it if needed). In addition, mental health trials of prison leavers will suffer from the problem that distress is relieved for most prison leavers following release, making assessments of benefit more complex.
The outcomes selected are potentially the greatest weakness of the trial. The deliberately flexible intervention, designed to support whatever pathway an individual needed, creates problems for measuring small but potentially critical changes for an individual. The weakness was not in our relatively robust process of selection, but in that there are no outcome measures that have been developed to reflect nuanced individual, but critical, improvements in response to person-centred complex interventions. The in-depth cases studies evidenced small changes that were important to, and valued by, participants and their loved ones; however, these changes were not measurable on the trial outcome measures we had selected.
The development of an iterative and substantive IDP was not sufficient to achieve consistent intervention delivery. Intervention participants’ individual circumstances and personalities, and the context within which Engager practitioners were working (i.e. practitioner illness, staff changes, capacity limitations), limited the intervention being able to be delivered as hypothesised. Participants in the usual-care arm reported experiencing meeting with researchers who demonstrated an interest in them and their lives as helpful and supportive and even referred to this as the ‘Engager intervention’. This degree of interaction was necessary to ensure the retention rates required by RCT methodologies. However, receiving proactive follow-up from researchers who were trained in the intervention’s engagement values to achieve retention rates (i.e. listening without requiring participants to challenge their behaviours) could be seen as a different and even more palatable intervention. The positive impact of the follow-up process may have reduced the ability of the trial to show a difference between people receiving the Engager intervention and those in the usual-care arm.
The intervention resources, as delivered, were insufficient to challenge the wider socioeconomic determinants that framed the trial participants’ life experiences. Despite generous funding and resources, the outstanding questions that this programme has not been able to address are as follows:
- Given the overwhelming challenges and disadvantages that this population faces, what would be required to achieve sustained change for this population?
- Would the costs and resources required for this sustained change be economically prohibitive when set against wider national crime, victim and societal costs, as well as familial costs (particularly the legacy for the children of the participants)?
Implications
Considerations for future research
The Engager research programme demonstrated that it is possible to recruit and follow up individuals with common mental health problems into the community following release from prison in a RCT. However, the cost per individual recruited and followed up is significant. At trial mid-point three researchers were required at each site to manage recruitment of final participants alongside follow-ups. In addition, outcome measures for mental health interventions for individuals with complex needs and, in particular, individuals in contact with the CJS, need to be selected carefully along with the length of follow-up. There is a very real question as to whether or not it is possible to measure changes in outcomes using standard outcome measures for individuals with such great heterogeneity of need. Therefore, it may be that new outcome measures or systems of measurement need to be developed. However, it is also uncertain whether or not interventions of a relatively short nature can overcome years of adversity and maladaptive behaviour.
There were additional challenges to trial integrity. We were not able to ensure that researchers were blinded. We had a significant proportion of follow-ups located in prison, potentially contributing to bias. We needed (and had) caring dedicated researchers to achieve follow-up and they were often seen as helpful by control participants (with some mistakenly believing that they were receiving the intervention). It is unclear whether or not RCT methodologists can overcome all these challenges when attempting to evaluate complex interventions delivered within the CJS and, therefore, we suggest that future trials should allow for creativity alongside rigour.
Alternative methods of research using whole-system approaches, for example focusing on quantification of access and examining pathways over time, alongside qualitative approaches to understanding the likely but not definitive effectiveness of various approaches, is an alternative overarching method to the RCT. Such approaches may also be more easily integrated with system redesign and continuous quality improvement. Whether using a RCT or alternative methods including routine data, we suggest that there is sufficient evidence to support an evaluation of the deployment of Engager intervention components by existing criminal justice, housing and substance misuse services.
Implications for practice and policy
The Engager research programme points to a range of potential implications for practice; however, given the mixed results, none is definitive.
Mentalisation-based approaches
The MBA and psychological formulation underpinning the shared understanding and plan were suboptimally delivered, but showed promise. Mentalisation has a good, but still relatively small, evidence base, and the Engager intervention demonstrated (and evidenced at some points for some people) the potential for applying the approach flexibly to a population with complex needs outside the traditional hour-long therapy sessions. The integration of MBA alongside a ‘shared understanding’ formulation showed promise in some of the in-depth cases, indicating an area for further development. In addition, as has been found in other recent interventions aimed primarily at offenders,62 working as Engager practitioners led to increased competencies and transferable skills. The research showed that some practitioners working in Engager support worker roles were able to use mentalisation-based and formulation-type approaches outside mental health systems. If MBA and other similar psychological approaches are to be embedded into other service settings, such as housing, substance misuse and probation, then this needs to be carried out by staff with suitable capabilities and supported by excellent training and ongoing mental health supervision. It is important to identify the key components of a MBA that is essential in working with this client group, recognising that development in practice may best be carried out with in-depth research, including analysis of interactions.
‘Through the gate’ practices
Engager showed that it is possible to consistently carry out practical ‘through the gate’ work, which would often take most of a day. The working practices used within the Engager programme included systems for reliably meeting individuals, safely using a car for transport, protocols for leaving individuals who may still be without accommodation, supporting attendance at two or more meetings and the use of this work to develop trust for ongoing work or for transfer of trust to other services.
Pre-release work
Pre-release practical and psychological work was perhaps the least successfully delivered component of the Engager intervention. The process evaluation suggested that pre-release practical and psychological work was not accepted when it was tried, and often it was only intermittently attempted (especially for people who were randomised months before release). Practitioners may have been unclear about the importance of the pre-release practical and psychological work and may not have felt confident about what to focus on. The Engager programme, therefore, provides little new evidence on how to best carry out pre-release psychological work, including regarding if focus needs to be placed on developing a trusting relationship to optimise opportunities for engagement post release. We also found little evidence of other mental health support occurring in the prison in line with the lack of progress of delivering the IAPT in prisons, and all of these issues point to the need for a review of psychological support leading up to release as a whole.
Supervision
Team-based working and supervision was shown to be important. If team-based working and supervision was not operating effectively, then it contributed to poorly functioning teams and suboptimal delivery of intervention components, particularly of psychological components. There was also evidence that effective team-based formal and informal supervision helped practitioners in this very challenging work. For example, as well as supporting use of MBA and formulation with individuals who were engaged, supervision supported decision-making about when not to pursue work that was futile. In particular, the Engager supervision structure of having the mental health supervisor not just providing supervision through discussion, but also often seeing individuals and leading a team of professionally unqualified practitioners, is a model that is aligned with national clinical policies and shows promise.
Supporting mental health care within other services for individuals with complex needs
In our view, training existing probation and third-sector staff to deliver the core Engager psychological functions provides the most promising avenue for policy development. These staff, already commissioned to carry out the practical support of resettlement, and often providing emotional support informally, would need additional supervision from mental health experts. A model of embedding mental care into existing non-health teams for individuals with complex needs is the opposite of the current practice of aiming, but nearly always failing, to secure an onward pathway to standard mental health services. It is, however, in keeping with current policies being developed for community mental health provision, which aim to address inequities across voluntary and community sectors and primary care, breaking down barriers between teams, as well as supporting positive mental health function across the system.
High-quality delivery of the mental health functions within existing services offers the possibility of a whole-system person-centred response for individuals with enduring complex needs and one that is substantially less expensive than the stand-alone Engager team with caseloads of 10–12 individuals. If just a few prison sentences could be avoided, then provision of mental health care in this integrated way could be cost-effective.
Conclusions
The Engager research programme allowed for substantial theoretical development and trial science work, building up to a fully powered trial, cost-effectiveness evaluation and an exemplar in-depth multimethod process evaluation in an area of health care that has very limited previous research and development. The Engager research programme saw the realist-informed development of a comprehensive evidence-based complex intervention for prison leavers with mental health problems, which was underpinned by a person-centred approach and supported by a flexible team-based delivery.
Although the Engager intervention, as delivered, was shown not to be effective using standard mental health trial outcome measures, conclusions are not straightforward. The weakness in measurement of improvement for personal goals is a weakness of current methodology for complex intervention evaluation. Our in-depth process evaluation showed that, although implementation problems may also have contributed to the neutral result, when delivered well, there was activation of mechanisms that produced short-term outcomes, such as engagement and emotional competencies, which we had hypothesised. The delivery of MBA and the ‘shared understanding’ were both particularly challenging to achieve, but were also both shown to be potentially critical for the generation of such outcomes. Other promising elements included the through-the-gate function and team-based supervision. Overall, although investing in new stand-alone Engager teams is not justified by the results, we conclude that it is worth testing whether or not core aspects of the intervention could be delivered by existing substance misuse, probation and homeless services, if supported by supervisors with skills in psychological therapy.