Economic study

Were the APPRAISE intervention to demonstrate effectiveness, the next barrier to widespread adoption would be cost efficiency. While stated policy criteria exist in the UK health sector to guide the design of health research,76 the APPRAISE intervention encompasses a wider stakeholder group, including the prison and wider criminal justice system (CJS) stakeholders, and the prisoners themselves. These groups additionally require convincing of value from their own perspective to help alleviate implementation barriers. Hence, a standard health and social care perspective alone may not provide sufficient evidence needed by all parties in their independent decision-making, and a broader taxonomy of outcomes is necessary. Conducting any analysis in a prison setting adds challenges to data collection and analysis, hence the feasibility of collecting data to inform data analysis requires exploring and refining prior to any significant investment in a larger definitive trial. To that end, we present the findings of the APPRAISE economic substudy, including a micro-costing methodology protocol, and a revision of the Economic Form 90 for a UK context,60 and some provisional scoping of routine data sources to support future cost-effectiveness analyses (CEA) in this area. We anticipate this work to be valuable as we believe this to be the first of its kind for an extended ABI in this setting and that the methods we have discussed will likely be relevant and transferable across other ABI settings.

Full details of the wider trial methodology are given elsewhere in this report and in our published study protocol.1 This chapter focuses exclusively on the economic evaluation pilot study. Specifically, we discuss efforts undertaken to address the study Objectives 1d, 1e and 1f.

• 1d. Can we collect economic data needed for a future definitive RCT?
• 1e. Can we access recidivism data from the PNC databases for trial participants?
• 1f. Can we access health data from routine NHS data sources for trial participants?

To aid UK policy relevance, the base-case costing perspective was based on the NICE recommended NHS and personal social services (PSS) perspective,76 with the addition of criminal justice and prison service costs to future support prison sector buy-in. The economic study focuses on the opportunity cost of intervention implementation from these perspectives. Further planned analyses aimed to widen this to a whole societal perspective with the further inclusion of employment, welfare, and third-sector addiction services, as well as including future cost savings from the NHS and PSS perspectives. For this pilot feasibility study, the base year for all costs was set to the 2020–1 financial year, selected at the latest financial year from which any prisoner provided data on balance internal and external validity. Discounting and inflation adjustments were not necessary as the time horizon for analysis was under 1 year, and all price estimates related directly to the base year. A Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist indicating the location of key economic principles and methods can be found in Appendix 12.

Methods

The economic evaluation pilot study aimed to evaluate the feasibility of collecting economic data to inform design of a potential future RCT of the APPRAISE intervention in a UK prison setting. This had three principal components:

1. development and pilot testing of an activity-based costing (ABC) exercise adapting recommendations from Drummond et al.,77 which aimed to estimate the resources used to deliver the ABI programme, and equivalent usual care, and associated financial cost of these
2. pilot testing of the collection of self-report data on prisoners’ health care, social care, criminal justice, welfare payments, employment, and third-sector addiction-service utilisation at baseline and at 6- and 12-month follow-ups using an adapted Economic Form 9060
3. an examination of the feasibility of accessing recidivism data from the PNC databases, and health data from routine NHS data sources for research purposes within a prisoner population.

Of note are overlaps between potential data items obtainable via components 2 and 3. We sought to compare data quality to both validate our survey data instrument and identify the preferred data-collection approach to provide an informed balance of data quality and participant self-report burden. We initially planned to publish a health economic analysis plan prior to analysis; however, disruptions due to the pandemic necessitated a number of ad hoc adaptations.

The activity-based costing exercise

The proposed ABC exercise was based on the recommendations by Drummond et al.,77 and consisted of four principal steps:

1. Step 1 – a taxonomy of intervention activities was created. A 1-day consultation exercise was held, with both the staff delivering the intervention and the research staff involved in the APPRAISE pilot study, to identify all of the resources that would be required in the implementation of the APPRAISE intervention, at scale, in a future hypothetical roll-out, as opposed to those only required for research purposes. A detailed narrative was constructed, for each study site, which documented the key components of the intervention delivery and the process. The intervention activities were listed in the sequence performed, in relation to each participant, and the resources that were used for each activity.
2. Step 2 – resources used for each activity were measured. The staff delivering the APPRAISE interventions in each study site were asked to note the duration of each intervention activity and the necessary materials and commodities that were required before, during, and after each stage of intervention. Examples of these resources measured included: the time taken to locate participants using their preferred method of contact; the preparation and printing of the intervention materials; and the actual time taken for the delivery of the intervention sessions. Security protocols and procedures prohibited the use of use electronic equipment for recording this information. In order to comply, handwritten research diaries were instead compiled. These were either transcribed verbatim or were scanned and shared retrospectively within the economic research team.
3. Step 3 – identification of unit cost. Where possible, unit costs for each resource use item were identified based on the study invoices and targeted (non-systematic) literature searches or sourced from expert knowledge within the team. Priority was given to the estimates considered to be best fit to a hypothetical future roll out (at scale) where multiple estimates were located.
4. Step 4 – calculation of costs. The resources identified in Step 2 were combined with the unit costs from Step 3 to generate estimations of the costs of each activity.

These calculations are not intended as an accurate estimation of costs of a hypothetical future roll-out, but rather as a demonstration of how these might be calculated in a future trial. Of particular note is that differences in prison procedures would be expected to yield differences in durations and costs of each type of activity, and that a larger study would need to account for this. To this end, separate outputs from Steps 1 and 2 were generated and compared for each site.

Adaptation of the Economic Form 90

The items of the Economic Form 90 were systematically examined in consultation with a UK economist and experts in the prison and addiction-services fields to identify UK equivalents for each entry and anything missing, and remove anything not relevant in a UK context. Questions were additionally rephrased for clarity to a UK reader and PPI input was solicited.

Unit costs for each item were identified by a non-systematic literature search, alongside those from the ABC exercise, with the same preference given to national-level unit costs to maximise relevance to a hypothetical future roll out (at scale).

Examination of Police National Computer and National Health Service data sources

Permission to access PNC and PHS data sets was requested from prison authorities. Upon receipt these were to be examined for data quality and compared to results of the Form 90 survey to cross validate. Results from this comparison would inform the optimal balance of data items to be obtained from each source in a future definitive trial. Consideration was to be given to any missing items, and if these were estimable via other sources, such as literature or modelling.

Results

Disruption due to coronavirus disease discovered in 2019 pandemic

Follow-up data collection for multiple parts of the study became disrupted as researchers were unable to access prison services due to revised safety protocols. This had the following notable impacts on the economic substudy:

• It was not possible collect the anticipated number of Economic Form 90 surveys, preventing analysis and validation efforts. However, the adaptation process was completed prior to onset of the pandemic, and scoping of potential sources for price weights was possible remotely.
• Prison staff experienced sharp increases in workloads which they needed to prioritise over research efforts, leading to lower participation rates and possibly different duration of intervention sessions than might be anticipated in a live roll-out.
• Time spent locating prisoners for post-liberation intervention sessions increased due to additional complexities of doing so remotely that may not reflect future methods.

Activity-based costing exercise

Row headings on Tables 35–37 display the items identified in Stage 1 of the ABC process via day consultation. Tables were then populated in Steps 2–4 detailing all activities, staff costs and the unit cost sources. Separate estimates are provided for the Scotland and England study sites to account for differences in services and local salaries of staff providing services.

Table 35 breaks down staff costs associated with delivering the intervention. The intervention service providers (CGL) at each study site were asked to provide a typical salary cost for the different roles delivering the APPRAISE intervention so that the intervention staff salary cost could be estimated. Values obtained for these, correct at the time of the pilot study undertaking (2020), are included in Table 35. Participant locating time proved more difficult to determine and hence it was more difficult to provide a meaningful estimation of the staff cost associated with it. In the pilot study these tended to be lower-paid staff, such as support workers or administrative staff, attempting to make contact with a participant. The time taken to schedule a follow-up intervention session or to successfully make contact to start the intervention session remotely (by telephone) often varied significantly. Some individuals could be reached quickly (within 10–15 minutes) or it could be ascertained quickly that the listed provided contact methods could no longer be used, for example their phone number was no longer in use or e-mails to their address were bouncing back etc. This was in direct contrast to those who were unable to be reached after frequent attempts, by different methods of contact, or those who would reply once and then would stop replying. Successful completion of follow-up interventions was low, even with large amounts of time spent attempting to make contact. However, it was difficult to establish how much of an effect the COVID-19 pandemic had in this lack of successful contact; thus it needs to be an area to explore further in a future cost analysis, rather than attaching a cost estimate to the participant-contacting time with limited insight.

Table 36 documents the one-off and the ongoing staff training costs that would be associated with implementing the APPRAISE intervention in practice. Initial APPRAISE interventionist staff training consisted of a one-time in-depth training session lasting approximately 2 hours. This was followed by refresher courses every 6 months typically lasting 20–30 minutes, and weekly 5–10-minute debriefs to allow the intervention staff to catch up with the research staff around their progress and to provide updates and to discuss any concerns or queries. However, disentangling research-only costs from those that would be present in a real-world setting may require separate sampling, and/or may be better handled by attributing varying proportions of activities to each in sensitivity analysis in a next-stage RCT. These staff costs, associated with the intervention delivery training, would vary dependent on the numbers of the different staff roles that were required in each location, and across the two study sites.

Table 37 presents other resource-utilisation costs required when delivering the APPRAISE intervention. As several of the questionnaires form part of the screening and recruitment process, and prison security processes and procedures prevented all non-paper-based data-collection options, printing costs totalling £6.45 per participant were unavoidable.

Depending on which session was being undertaken a plethora of different methods may be used to contact participants for follow-up intervention sessions. These included the cost of using a telephone or text messages, sending written letters or using the internet for social media or WhatsApp. In the pilot study, all of the required printing was conducted and costed by the research staff at the University of Edinburgh, which included the follow-up letters. Any further attempts to contact participants or to conduct phone-based follow-up sessions were conducted using a researcher’s phone, but with a new pay-as-you-go sim card. The sim card was free to obtain, but the phone costs have been quoted from the listed provider’s pay-as-you-go costs at the time of the study. Costs of these in future studies or eventual roll-outs may vary, and users of these results may wish to substitute their own local estimates.

Factors, such as rooms in prisons needed to undertake sessions were provided free and so were not included in the pilot ABC exercise. While these are anticipated to have opportunity costs associated with them, costs at a given prison will vary substantially and arguably only incur an opportunity cost if the room was needed for another purpose. Financial values for these are challenging to estimate reliably and may be misleading if they appear overly specific. Nevertheless, their need is noted in a more qualitative sense, with recommendation that future work or roll-outs include these costs and use local estimates for decision purposes where possible.

Adaptation of Economic Form 90

The adapted version of the Economic Form 90 survey developed as part of this study can be found in Appendix 9. While the COVID pandemic prevented data collection, it was still possible to identify candidate sources of price weights for items included in the survey. These are listed in Table 38.

Note that this table provides potential price weight sources as opposed to specific price weights and values for a given base year. This is to provide a guide for future survey users to source weights appropriate to their analysis which may not have been published at time of writing.

While the specific content of each may change in a given year of publication, standard UK price weight sources, such as the Personal Social Services Research Unit (PSSRU) unit costs of health and social care series78 and the NHS reference cost report series79 are likely to be suitable for the majority if not all of the health and social care aspects of the survey. PSSRU additionally has some content for substance misuse including alcohol services, albeit with more limited content.

A small number of items were identified as being likely to require bespoke or local pricing to best reflect the differences in regional availability of services and prices, namely those relating to housing costs and third-sector alcohol services. While a similar argument for local costing could be made for health and social care aspects, housing and third-sector alcohol services are likely to be part of costing perspectives falling out with a national programme or organisation (such as the NHS), but rather on the individual or a programme regional budget.

Price weights for criminal justice aspects may warrant some minor rewording of the survey to better match the granularity of the available sources in the literature. For example, Ramponi et al. break down costs by form of crime (e.g. assault/robbery/criminal damage) rather than as components of system use (e.g. arrest/custody/prison).81 In the absence of a recurring national publication for such items, researchers may wish to identify price weights prior to survey issue and adapt the questionnaire to suit.

Response rates of Economic Form 90

While the adapted Economic Form 90 was issued at baseline, its follow-up issues were hampered by the pandemic, as shown by Table 39.

The baseline data completeness and self-reported responses are shown in Table 40 (health and social care), Table 41 (living arrangements), Table 42 (employment status) and Table 43 (criminal justice service utilisation).

Completeness of the survey at baseline was generally very good, with only three questions having over five missing entries.

Feasibility of accessing Police National Computer and National Health Service service use data

Although it is possible that COVID-19 played a role in our inability to obtain these data, we do not think that the pandemic was solely responsible. Two key issues are at play: prisoner confidentiality and data systems. We found that gaining approval to obtain the PNC or NHS identifiers was difficult to impossible due to a combination of prison and data-protection concerns. Although these concerns may fade as more evidence of the benefits of the APPRAISE intervention mounts, we conclude that it is very likely that these concerns will still exist for a definitive trial, and possibly be even more salient for some stakeholders in that context. Exacerbating the confidentiality issues were data-system issues that prevented data access on a large scale. Although this issue depends on the jurisdiction, in each of our two jurisdictions (Scotland and England) at least one of the data sources required case-by-case data extraction at some step of the process. We could not extract either PNC or NHS data on a large scale in either England or Scotland, making the use of such data prohibitively costly for a definitive RCT.

Discussion

We aimed to assess the feasibility of collecting economic data to inform the design of a potential future RCT of APPRAISE in a UK prison setting. We did not seek to provide a definitive measurement of cost, but rather inform methods by which they could be measured in a future trial. This has been achieved with mixed results. While many of the issues faced were a result of the COVID-19 pandemic, it is important to separate ambiguity of findings due to extenuating circumstances from findings of issues in data collection. Data obtained despite such circumstances are still likely to be viable without them, if not more so, and we were able to document the stages of an ABC exercise and provide discussion around how to assess the costs associated with implementing the APPRAISE intervention. We believe this to be one of the first pieces of work in this area and the methods that we have presented can be applied more broadly to determine the costs associated with implementing a similar intervention in practice.

While not the aim of the study, we have also provided early estimates of some of the costs associated with the implementation of the APPRAISE intervention, in two different prison systems. There were important differences between the two different sites, primarily around the differing roles and numbers of members of prison and intervention staff involved, the location of delivery and availability of rooms and the level of access to prisoner data at the time of data collection. Such differences are inevitable given the heterogeneity of prison regimes in general in terms of local service provision, facilities, staff and the management of each prison site. Nevertheless, the ABC process itself as described could be reapplied to new contexts, and elements of the activities observed, such as necessary paperwork and durations of phone sessions would reasonably be expected to be common to other settings.

There were challenges in separating research-specific costs from those which might reasonably be expected in a practice. For example, in the England site the researcher recruiting the participants to the study was unable to access the prison database to identify further potential participants and their current location. This meant that the process of identification was more cumbersome at times and it may have taken longer to locate eligible participants, which would likely increase the staff cost associated with the intervention. Assuming that, in a real-world setting, intervention staff or prison staff would have access to such databases and would not need to spend the extra time locating individuals for their APPRAISE intervention, the time taken may not accurately reflect that needed in a live roll-out. In relation to this, prison staff costs were intentionally excluded in the staff costs listed in Table 35 because, while they were involved and helped with the process of locating participants within the research trial setting, it is currently unclear what role they would play in practice. Such details will need to reflect the final design of an intervention or will need to be explored in future work, though we have no reason to believe the ABC approach could not be used to do so.

The COVID-19 pandemic is also likely to have influenced the follow-up and the retention of study participants. This issue is common to the study as a whole rather than to economic assessment in particular, and is discussed in broader terms in other chapters of this report. Specific to the economic evaluation pilot, substantial time was spent by intervention staff trying to locate participants for their follow-up interventions and often being unable to access the information they required or finding that participant court dates had been moved or postponed due to COVID-19. More time spent locating participants for the post-liberation intervention sessions has direct implications on the cost of the APPRAISE intervention, as it obviously increases the staff costs associated with the intervention. Disentangling the direct impact of the COVID-19 pandemic on locating participants remotely from the normal challenges of this population was not possible in this pilot study. Follow-up in the CJS is often very complicated and it is frequently challenging to retain participants within a study due to a number of logistical reasons, mostly being unable to re-establish contact with them.82 Consequently, this is something to consider when planning for a future RCT with the APPRAISE intervention and ensuring that the lower-paid staff, such as administrative or support workers, could be responsible for locating participants to reduce overall costs, if deemed necessary.

The timing of the first COVID lockdowns in 2020 had a significant effect on the participant recruitment and the study’s data collection after both prison sites went into an immediate lockdown. Hence limited data could be collected for the remaining duration of the pilot, post April 2020. As a result, it also became difficult to work with prison staff due to their sharply increased workloads and new unforeseen pressures. The majority of any further interactions were conducted virtually (via telephone or e-mail), rather than the face-to-face meetings and follow-ups as planned. This is extremely likely to have had an effect on the information we were able to collect around the cost estimates. For some specific items, such as the cost of space in the prison, we were unable to obtain meaningful estimates. Context is critical for this factor, as costs of this are expected to vary substantially between prisons. It may be sufficient to note the duration for which a private room is required, and allow users of this information to apply their own local estimates where necessary.

We were able to adapt the Economic Form 90 to a UK context and provide a list of candidate price weight sources for future use. We were also able to issue the survey to 132 prisoners at baseline with very high completeness, though follow-up time points were heavily disrupted by the pandemic. Of note, the baseline survey was researcher-administrated. It is not possible to know how well the survey would have been completed in follow-up time points without the confounding of the pandemic; however, the completeness at baseline at least implies the form was well accepted and understood. Completeness included questions relating to criminal justice activity, suggesting that any adaptation of the questionnaire to allow for more bespoke or specific price weights to be applied is likely to be viable following a similar format.

Disruptions to data collection prohibited our ability to validate the survey against routine data sources. The matter may be rendered a moot point as PNC and NHS data proved too difficult to obtain at large scale for this population, thus suggesting they should not be included as part of a definitive RCT. Some element of prisoner self-report was always anticipated to be unavoidable; however, the loss of routine data as a potential alternative source places increased importance on validation of the revised Economic Form 90. While the original Economic Form 90 has been validated for a US alcohol treatment population,60 larger-scale future trials would be advised to build in a pilot stage to validate and potentially refine our adapted UK version prior to full-scale administration.