Overview
What was already known:
- It is currently not possible to evaluate effectiveness or cost-effectiveness of peer-supported social care.
What this chapter adds:
- This chapter provides an evaluation guide that outlines operational, cost and outcome data that needs to be collected to enable regular monitoring and/or evaluation in future.
Introduction
Currently, peer support services for social care are implemented in different ways (see Chapter 5), have associated risks and benefits (see Chapter 7) yet are not being routinely monitored and evaluated at a national level (see Chapters 8 and 9). Without being able to monitor these services and measure their outcomes, it is not possible to tell how much they cost, whether or not they are cost effective, whether risks are being mitigated and whether outcomes for both recipients and buddies are being improved by these services.
This chapter therefore aims to triangulate findings from Chapters 3–9, to develop guidance to facilitate routine monitoring of peer support services for social care and robust evaluations of these services in future. It also addresses the research question:
- How could these outcomes (of peer support initiatives for social care in adult prisons in England and Wales) be measured.
In this chapter we define ‘monitoring’ as the continuous supervision of activities related to peer-supported social care in prisons and checking whether plans and procedures are being followed.
‘Evaluation’ refers to the systematic assessment of peer-supported social care in prisons in comparison to alternative modes of delivering a service to the same prisoners, including providing no service at all. Both monitoring and evaluation rely on good data and an evaluation also needs well-defined and measurable outcomes.
A unified approach across England and Wales
To monitor peer-supported social care services in prisons, it is important that sufficient data are collected. Locally, this would provide clarity in knowing what care needs are being supported, helps with identifying and mitigating risks, and provides alerts when problems occur. Standard methods of data collection and good communication between the different organisations would also help provide continuity of support when prisoners are transferred between prisons or released into the community. A nationally agreed approach would also facilitate evaluation of services to compare different modes of delivery and to help understand what works best and under what circumstances.
Relevance to prison service instruction for peer support
As described in Chapter 1, PSIs provide guidance on peer support services.45 In terms of monitoring and evaluation, the PSI45 highlights that regular informal peer support should be escalated to social care needs and that boundaries must be explained in care plans or recorded locally.45 However, findings indicated that there is very little collection of monitoring and evaluation data across prisons (see Chapters 8 and 9). Findings from the documentary analysis (see Chapter 4) and empirical study (see Chapters 4–9) found that care plans are infrequently reported for social care needs (though some prisons from the empirical study did have care plans in place). If monitoring and evaluation data are not being captured, this may make it difficult for prisons and local authorities to monitor peer support schemes, and therefore it is unlikely that the provision of informal peer-supported social care will always be escalated to local authorities.
Monitoring of peer-supported social care schemes in prisons in England and Wales
Roles and responsibilities for the collection and monitoring of data
Any guidance on collecting and using data will be ineffective unless the corresponding roles and responsibilities of the different organisations are clear and resources and motivation are both available. In addition to these, training of individuals involved in data collection is essential not only for ensuring the accuracy, consistency, integrity and efficiency of the data collection process but also for allowing data comparability across different organisations. However, by adding extra burden to an already stretched resource there is a risk that organisations will not agree to take part. It may therefore be useful to prioritise the data to be collected while being clear about the overall aims of the service, the purpose of collecting data, and how they would be used in the wider system. Indeed, anyone commissioning or collecting new data would need to understand this alongside the proportionality of extra reporting. Any data collection needs to comply with ethics and legal considerations. Responsibility of monitoring could fall to HMIP (in partnership with organisations such as CQC, where appropriate) in their role as inspectors, and facilitated by operational data collected locally by individual prisons.
Formal evaluation is probably best carried out by research teams directed or sponsored by the service in collaboration with research funders, rather than being based exclusively on routine activity in the service. Local data collected for monitoring would also facilitate these evaluations.
Operational data to prioritise for monitoring
Good operational data are the bedrock of effective monitoring and of any subsequent evaluation. To avoid over-burdening the system and to help services to comply with data collection, we suggest the service prioritises their data needs for monitoring. We provide an example in Table 19 which prioritises the service being offered, who are providing and receiving peer support, and the type of support in each case. Whether collected by the local authority or prisons will depend on the local responsibilities.
| Priority data | Additional data that would be helpful |
|---|---|
| Number of peersNumber of recipientsHours peers work each weekRecords of whether peers have received trainingRecords of the type and amount of supervision providedType of support provided to each recipientRecords of prisoner’s needsFunding sourcesNumber of sessions/hours of peer support received by prisonersLabour costs (e.g. staff and training)Non-labour costs (e.g. hand-outs, uniforms and training materials) | Staff training records (e.g. on social care topics)Number of applications received for peer support rolesRecord of how peers were selectedNumber of individuals rejected vs. offeredInput needed by prison staff to support peer support programme |
Evaluating peer-supported social care schemes in prisons in England and Wales
Selecting outcomes to prioritise
Good monitoring will facilitate effective evaluation. The selection of appropriate outcome measures is key to evaluation, and we have described several potential outcomes of peer support programmes based on their risks and benefits (see Chapter 7). Given the resource constraints and that these programmes are not mandated, a feasible approach may be for outcome measurement and the corresponding data collection to be the responsibility of teams undertaking research studies. The services themselves may then have a useful role in facilitating such collection, and, where possible, with extra dedicated resource. To this end, given the lack of data, it is worthwhile prioritising outcome measures to highlight which data collections are likely to be more important. In Table 20 we present an example that prioritises the impact on individuals in prison that can be potentially measured directly using survey tools or questionnaires.
| Priority | Outcome measure | Suggested data to use | Existence of data | |
|---|---|---|---|---|
| Outcomes for the wider system and society | Higher-priority outcomes | Cost and use of resources | Commissioning data | Existing data in some areas but needs to be in a consistent format |
| Lower-priority outcomes | Crime rates after release | Police National Computer | Existing data but needs to be linked to prisoner IDs | |
| Outcomes for prisons and prison staff | Higher-priority outcomes | Prison safety | IRS | Existing data, individual prisoners not identifiable |
| Financial implications | Regular prison cost forms | Existing data in some areas but needs to be in a consistent format | ||
| Lower-priority outcomes | Prison atmosphere | Surveys of staff and prisoners | Not currently existing | |
| Staff supporting care provision | Surveys of staff and prisoners | Not currently existing | ||
| Outcomes for prisoners providing support | Higher-priority outcomes | Mental health and well-being, personal development, safety and relationships with prisoners and staff | Regular surveys of buddies.Questionnaire measurement tools for measuring well-being and mental health, for example, Guttmann’s self-esteem scale | Not currently existing except as occasional surveys |
| Financial benefit | Regular prison cost forms | Existing data in some areas but needs to be in a consistent format | ||
| Lower-priority outcomes | Employability/upskilling | Post-release surveys of ex-buddies covering personal development, skills and employment | Not currently existing | |
| Financial benefit | Regular prison cost forms | Existing data in some areas but needs to be in a consistent format | ||
| Impact on sentence trajectory | Prison records | Existing data | ||
| Recidivism after release | Police National Computer | Existing data but needs to be linked to prisoner IDs | ||
| Outcomes for prisoners receiving support | Higher-priority outcomes | Social care needs, physical and mental health, personal safety, relationships and social inclusion | Regular surveys of prisoners.Questionnaire measurement tools for measuring well-being and mental health | Not currently existing except as occasional surveys |
| Continuity of care on prison transfer or release | Co-ordinated operational data between prisons, probation services and local authorities | Exists in some areas | ||
| Lower-priority outcomes | Reduction of formal social care support | Local authorities monitoring numbers of prisoners requiring support | Existing data | |
| Physical health inferred from routine data sets | Bedwatch/escort data.Routine hospital administrative data (HES or SUS). Feasibility of methods tested in research studies | Existing data but needs to be linked to prisoner IDs | ||
| Cost of escalation | Regular prison cost forms | Not currently existing |
Time period for measuring outcomes
It is important to measure outcomes at the right times as much as possible. Some effects of peer support programmes on individuals could be observed quickly, whereas others might not be manifest until after some months, particularly if they cannot be measured until after a prisoner has been released. Also, impacts that are observed quickly may not be sustained.
Data linkage and use of existing routine data sets
Routine data sets could have a valuable role in evaluating peer-supported social care services in prisons and we have described in Chapter 8 possible uses of, for example, HES for identifying hospital attendance for prisoners with social care needs. To use this routinely would require ready access to these data sets and the ability to identify the relevant individuals. Also, the feasibility and value of using these data sets would need to be tested first which is probably best carried out in separate research studies.
Establishing comparator groups for evaluation
With most evaluations it is important to have a comparator (or control) group to understand whether any observed results are different to what they would be without a peer support scheme. This raises the issue as to how prisons without a buddy scheme would be supporting prisoners. For example, would they offer informal arrangements with other prisoners or would increased use of professional support services be filling the gap? If using surveys or questionnaires, one option for using a comparator group would be to carry them out in parallel at prisons without such schemes, or at the same prison before a scheme is just about to be introduced. In fact, the best time for such comparisons is when a new scheme is being introduced with surveys taken before and after the scheme has started. Comparisons can also be made between prisons adopting different types of peer support scheme, although a baseline would need to be established to understand whether any observed differences are due to the different schemes or the prisons themselves.
Evaluation of implementation and experience
While this chapter focuses on the monitoring and evaluation of quantitative and economic outcomes, it is also important to explore more qualitative outcomes. The methods provided in Chapter 2, and the findings presented in Chapters 5–7 provide examples of how to evaluate implementation and experience.
Testing approaches
An initial way forward for both monitoring and evaluation might be to test approaches at a local level before any national adoption, perhaps with a mix of local areas with different peer support schemes in place. Monitoring approaches could be piloted in a sample of prisons to determine feasibility and inform amendments required for a national programme. As mentioned above, evaluation would probably be best carried out by research teams who decide, in collaboration with the service, ex-prisoner groups and other stakeholders, which outcome measures to focus on, data collection approaches (e.g. surveys, established questionnaires, and routine data) and how to identify control groups.
Conclusions
This evaluation guide addressed the research question as to how outcomes of prison peer support social care services could be measured. To enable effective monitoring and evaluation of such outcomes, considerable progress needs to be made in gathering the appropriate data. Because of the variety of ways in which they are being implemented across the country (see Chapter 5), data collection should be at a national level if there is an interest in identifying what works best. The barriers to this appear considerable and require co-ordination between different bodies (local authorities, prisons and HMPPS) to gain a common purpose. It also requires identifying responsibilities for collecting information that is not currently reported. Monitoring may be best being the responsibility of HMIP (in partnership with organisations such as CQC where appropriate) but supported by extra resource in prisons to help collect data. Initially, because of the practicalities, we suggest a few data items are prioritised, and the process is tested in a few prisons. Evaluation is best carried out by research teams. It may be possible to evaluate some aspects of these services with existing data collections, but their feasibility for measuring effectiveness would need to be tested.