Process evaluation methods

Sections of this chapter have been reproduced from Holloway et al.1 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

In this chapter, we present the methods for Phase II of the study which comprised the process evaluation (addressing Objectives 2 and 3). Elements of the process evaluation, where quantitative data were also collected, were integrated within the pilot trial (Objectives 2a and 2c) with cross-referencing to related results tables for these specific objectives provided.44

Process evaluation aims and objectives

The new MRC framework for process evaluation guided the planning, design and proposed conduct of the process evaluation.32 The process evaluation set out to assess intervention fidelity, to consider the extent to which the behaviour change markers were appropriate to the underpinning intervention and to explore the feasibility and acceptability of the APPRAISE intervention and study measures to staff and for men on remand and following liberation. Objectives 2 and 3 set out the key questions that were to be answered.

Objective 2: to assess intervention fidelity

1. What proportion of the interventions are delivered as per protocol?
2. Is there any evidence of contamination between the two conditions and/or between those workers delivering the intervention?
3. To what extent was the intervention changing process variables consistent with the underpinning theory?

Objective 3: to qualitatively explore the feasibility and acceptability of a self-efficacy-enhancing psychosocial alcohol intervention and study measures to staff and for men on remand and on liberation

1. How acceptable are the trial and intervention procedures (including context and any barriers and facilitators) to the following key stakeholders: men on remand in prison and following liberation; prison staff (including healthcare staff); commissioners; policy-makers and third-sector partners?

Process evaluation methods and design

Table 30 summarises the different components of the process evaluation and how they relate to the objectives of the APPRAISE pilot study. It also identifies the main research question for each process evaluation component and the main sources of data that were employed to answer them.

Both quantitative and qualitative methods were employed to provide the data required to answer the process evaluation questions. The quantitative data providing evidence related to the implementation dose and quality as well as the mechanisms of change associated with the intervention were integrated in the pilot trial. The study participants and wider stakeholder semistructured interviews were conducted separately from the pilot trial. This was due to the impact on access and availability during the COVID-19 pandemic.

In particular, the qualitative interview data analysis sought to provide evidence to verify the feasibility and acceptability of the APPRAISE study intervention to better understand and explain the social processes associated with the intervention through the lens of normalisation process theory (NPT).69–71

Intervention implementation

To confirm intervention implementation, data collection was planned in two ways: intervention booklets (dose delivered) and audio recordings (dose quality).

Dose delivered

The interventionists were asked to use intervention booklets to record contact attempts, sessions offered, sessions delivered, reasons for unsuccessful delivery, session dates, and duration of sessions to assess whether the intervention was delivered as per protocol (see In-prison and post-liberation intervention delivery, Tables 17–19).

Dose quality (fidelity)

For the intervention condition, we had planned to audio record 20% of sessions delivered at each time point (4 sessions in total × 9 participants = 36) and for the control condition, we had planned to audio record 20% of the care-as-usual interventions (4 sessions in total × 9 participants = 36). However, we were unable to secure approval from the England study site to take in audio recording equipment, which meant that the in-prison intervention and control conditions could not be recorded there. Following discussions with the Scotland study site intervention team, recording control intervention delivery was not deemed feasible or practical.

Interventionists at the Scotland study site were aware that we were recording the intervention delivery sessions for the purposes of undertaking intervention fidelity assessment.

The Audio Recordings and Behaviour Change Counselling Index (BECCI)72 was used to provide insight into the Behaviour Change Counselling Skills of the interventionists from the audio recording of the APPRAISE intervention delivery sessions. To calculate the overall BECCI score, the mean across the item responses is calculated to provide the Practitioner BECCI score. The BECCI mean score will correspond to the points given on the Likert scales on the checklist.

The interventionists at both study sites received training (see Chapter 2, APPRAISE intervention training) and self-completed a six-question Likert scale evaluation (see Appendix 8) on completion of the training.

Contamination

During the qualitative interviews with the participants and interventionists and those delivering care as usual, we explored the possibility of contamination. This was with the purpose of identifying any evidence of contamination between those in the intervention and control arm and/or the intervention and non-intervention (care as usual) workers. We had hoped to note the wing/area in the prison where the participant was located and which other participants in the study they were likely to have most contact with. However, this was not feasible in terms of the practicalities within the running of the estates at each study site and the impact of COVID-19.

Mechanism of change

Quantitative and qualitative data were obtained to explore the mechanisms of change and the extent to which the intervention changing process variables were consistent with the underpinning theory.

The six proposed mechanisms of change in the logic model are: self-efficacy, outcome expectations, goals, plans, social support. These quantitative measures are discussed in APPRAISE intervention: theory, elements and logic model. In the pilot study the causal mechanisms were measured to provide increased understanding of how the delivered intervention influenced change through the quantitative assessment of key behavioural markers (mediators) of change. These were to be assessed at TP0, TP1 and TP2 and an exploratory prognostic analysis undertaken to explore the nature of change within these domains and their relationship to observed outcome.

In addition, we also conducted a qualitative exploration of study participant and stakeholder responses to, and interactions with, the intervention, providing the opportunity to identify acceptability, barriers and facilitators.

Context

We undertook qualitative interviews with study participants and wider stakeholders to contribute to our understanding of the context within which APPRAISE was delivered, to enable the delivery and or impact of the intervention at scale. We also recorded any differences in delivery in and across study sites where local organisational and devolved country contexts may have had implications for future intervention design and delivery.

Eligibility and recruitment to interviews

Those eligible to be interviewed were drawn from those registered in the pilot trial and from a range of stakeholders who had been involved in different aspects of the study or where, if this intervention were to be implemented at scale, they would be involved in some capacity.

Recruitment to all interviews was significantly impacted by the ongoing COVID-19 pandemic. The challenges of being able to locate men on remand hampered our ability to recruit them to take part in the process evaluation interviews. Similarly, the availability and capacity of some stakeholders (commissioners, policy-makers and prison healthcare professionals), as a result of the ongoing COVID-19 pandemic, meant it was not possible to secure interviews. Interviews were conducted at a time convenient to the participant and were conducted by JF and VG.

Sampling

Participants for all the interviews were selected via purposive sampling within three main groups.73 The proposed number of participants to recruit was 40 interviews in total across the two study sites: men on remand from intervention (n = 8 at each site) [n = 16] and control arm (n = 8 at each site) [n = 16], N = 32; stakeholders (interventionists and those delivering usual care, prison governor, prison officers, commissioners, policy-makers, third-sector partners and prison healthcare professionals) (N = 8). However, due to the impact of COVID-19 we were unable to reach this target.

• Participant group 1: men on remand who had been enrolled onto the APPRAISE pilot study and had been randomised.
• Participant group 2: interventionists, who were invited to take part if they had delivered the APPRAISE intervention to study participants.
• Participant group 3: stakeholders, who were invited to take part if they had been involved in different aspects of the study or where, if this intervention were to be implemented at scale, they would be involved in some capacity.

Informed consent

Each participant was required to provide their informed consent before taking part in an interview. We reinforced to participants that taking part in the interviews was voluntary, even if they had been involved in the main study or had delivered the interventions. It was also imperative to ensure that a participant’s decision to take part was based on a clear understanding of what was involved. All participants, therefore, received a verbal explanation of the study, at the time of the qualitative data collection, even in the case of male remand prisoner participants or APPRAISE interventionists. The verbal explanation included the provision of a PIS and then the interviewer went through all the elements in the PIS. When this had been completed, those in Stakeholder 1 group did not have to provide further written consent at this stage as they had already provided written consent on entering the study, which included taking part in future interviews. For those in Stakeholder 2 and 3 groups, each had to sign a consent form.

All participants were given the opportunity to ask questions or to clarify anything that was not clear, or they did not understand. In addition, all participants were reassured that they did not have to answer any questions that they did not understand or that made them feel uncomfortable and that they could withdraw their consent to taking part, at any time during the interview, and even following data collection.

Data management

During the timeline of the study, the COVID-19 pandemic came into play and therefore many of the interviews had to be conducted virtually using videoconferencing software or via telephone with Dictaphone facility as all interviews were to be transcribed verbatim. In Scotland as COVID-19 restrictions eased in the prison three of the four remand participant interviews were able to take place face to face and were digitally recorded, with the fourth taking place by phone.

Audio files were uploaded securely to a password-protected site by the researchers, from which the approved transcription company was able to access the files. On return, a quality check was conducted to ensure the accuracy of the transcribed files and then all audio recordings were securely destroyed, and the resulting transcripts were stored on password-protected drives at the University of Edinburgh and Teesside University. Upon storage, all interview transcripts were anonymised, with any personal or identifiable participant information being removed, and each transcript was assigned a unique participant number for analysis purposes.

Any pieces of hard-copy data from the Scotland site, such as the completed consent forms with a participant’s name on, were sent by secure courier to the England study team.

Qualitative data analysis

Following the transcription of the interviews, Braun and Clarke’s approach to thematic analysis was used, supporting a structured yet reflexive analytic process.74 Because of the atheoretical nature of Braun and Clarke’s approach, we chose to use NPT as a lens to support thematic identification.74

Thematic data analysis is often underpinned by theoretical concepts. In this study, we employed the NPT, an implementation theory, to better understand what the qualitative data could tell us around implementing the APPRAISE intervention for future work.NPT was developed to explore and develop the understanding around the factors affecting whether an intervention can be incorporated into practice, and the context in which the work of the new intervention happens.69–71 NPT aims to facilitate a better understanding of the challenges arising during the embedding of a new practice by considering the components of four different constructs; Coherence (‘sense-making’), Cognitive Participation (‘relational’ work), Collective Action (‘operational’ work) and Reflexive Monitoring (‘appraisal’ work).69–71 The theory lends itself well when considering qualitative data around implementing a new intervention in different contexts and in this work was used as a tool to structure the thinking around the different factors arising that would affect the future implementation of the APPRAISE intervention.

The interview data in the APPRAISE study were analysed within each of the three main groups; remand participants, CGL team and wider stakeholders. The first stage consisted of data familiarisation, essentially reading through the transcripts to elicit any initial thoughts and feelings emerging from the data. The next stage involved forming early codes from the data, the process of which is referred to as ‘coding’. A code refers to ‘the most basic segment, or element, of the raw data or information that can be assessed in a meaningful way regarding the phenomenon’.75 Each transcript was read and coded by JF and was then double coded by GW at data analysis meetings to ensure consistency and refined further or developed further as necessary. The next stages involved the codes being assimilated and emergent data themes were mapped against the four main NPT constructs. Areas of divergence and discrepancy were explored and the data checked for fit.