Methodology

This chapter introduces the key methodological elements of the study, in particular the study design, feasibility parameters being tested and outcome measures assessed. Information is also given about patient and public involvement (PPI), ethics considerations and changes from original protocol. The study protocol can be accessed at: www.journalslibrary.nihr.ac.uk/programmes/hsdr/130748/#/.

Study design

This mixed-methods study used two main approaches to assess feasibility: (1) a process evaluation to enable evaluation of the feasibility of implementing the CLECC intervention and (2) a pilot pragmatic CRT to lay the foundation for a future evaluation of CLECC’s effectiveness. The design draws on the Medical Research Council (MRC) guidelines for evaluating complex interventions.89 The MRC guidelines highlight the importance of a robust theory-based intervention design coupled with a thorough understanding of the mechanisms of change and the contexts in which implementation is possible.89 It is essential to undertake this design and evaluation prior to and during evaluation of effectiveness, so that the impact of the intervention can be optimised, and so that eventual findings on outcomes can be explained by how change happened (or not). In addition, the guidelines also highlight the importance of detailed groundwork on the implementation of experimental design and methods before a definitive evaluation of effectiveness is undertaken. This includes selection and testing of outcome measures; testing the feasibility of proposed methods for recruitment, data collection and analysis; and calculations of effect size to inform future sample size calculations.

Areas of feasibility that merited study here included implementation of the CLECC intervention into practice, contamination of practices from intervention to control wards, extent of intraward clustering of outcomes, rates of participation and attrition, and the performance of the selected outcome measures. Through piloting these procedures on six wards in two English hospitals, this study aimed to reduce uncertainty in designing and executing a definitive evaluation. We aimed to test a range of parameters to provide an evidence base from which to make decisions about evaluation design to be implemented in other centres in a future trial, including CLECC implementation, feasibility of ward-level randomisation, contributing to sample size calculation and selection of outcome measures.

The feasibility of implementing the CLECC intervention into practice was assessed through a process evaluation, using normalisation process theory (NPT) as a framework.90 Qualitative interviews with nursing staff and managers during implementation and follow-up phases, observations of learning activities, and ward leader questionnaires aimed to identify and explain the extent to which the CLECC intervention was implemented into practice, enabling an assessment of its workability and integration into existing work practices. In addition, data were also gathered on the feasibility of conducting qualitative interviews with staff, patients and visitors to inform qualitative evaluation accompanying a future definitive trial, with the purpose of explaining trial outcomes.

An experimental design in the definitive evaluation is the most appropriate design to establish effectiveness.91 In order to prepare for a definitive multicentre evaluation, we undertook a randomised pilot study to assess the feasibility and pilot procedures for a pragmatic CRT of effectiveness. In randomised pilot studies, all or parts of the future trial are conducted on a smaller scale to see if it can be done.92 Given that the intervention was targeted at a group of staff rather than at the care of individual patients, cluster randomisation of staff and patients at nursing ward level was undertaken.91

Ward teams (i.e. clusters) were randomly allocated following baseline data collection to participate in the intervention or act as control. We theorised that implementing CLECC on four wards across two hospital sites was likely to provide sufficient diverse contexts within which its feasibility could be assessed, leading to further refinement when needed. The inclusion of a small number of control wards aimed to generate insight into the likely acceptability of randomisation in the main trial and differential compliance with study procedures between intervention and control.

The CLECC intervention was implemented over a 4-month implementation period on four wards from June 2015, with two wards acting as control. Outcomes were assessed at baseline (2 months before intervention and before randomisation to groups) and 4 months after completion of the 4-month CLECC implementation period (i.e. follow-up was 8 months after randomisation). There is no single validated measure for compassionate care and we assessed its measurement across three complementary core outcomes: (1) researcher-rated observations of the quality of staff–patient interactions, (2) patient-reported evaluations of emotional care and (3) nurse-reported measures of empathy. Baseline and follow-up data were also gathered on individual and ward team characteristics.

The economic component of the study aimed to explore how costs and benefits might best be measured in a definitive evaluation. Standard health technology assessment economic evaluations rely on patient-based outcome measures, the feasibility of which was explored in this study by use of EuroQol-5 Dimensions, five-level version93 (EQ-5D-5L), (a health status measure) at the ward level. In addition, the study explored the likely costs of both the CLECC intervention (training) and its implementation (qualitative interviews with nursing staff on intervention wards).

Data collection took place between March 2015 and May 2016.

Process evaluation

The process evaluation aimed to identify and explain the extent to which the planned intervention was implemented into existing nursing practices, to enable conclusions to be drawn about how CLECC can be optimised to support sustained compassionate care delivery in acute care settings. NPT was used to guide the process evaluation, shaping the interview topic guides and informing the framework for analysis.90 NPT focuses on the dynamic processes that support the integration of innovative practices into everyday work, and so is a helpful way to evaluate what actually happens when complex interventions are introduced into practice, and how and why the desired outcomes are achieved (or not).94 It explores social processes and the work that people do individually and collectively, in terms of cognitive and behavioural work. NPT has four core constructs: (1) coherence – making sense of the intervention, (2) cognitive participation – investing in the intervention, (3) collective action – the practical work of implementation and (4) reflexive monitoring – modifying and embedding the intervention.95 These constructs were used to define the areas that formed the focus of the evaluation.

The process evaluation focused on:

1. exploring how and in what ways the new practice was initially received, how individually and collectively people practically conceptualise and make sense of it (coherence)
2. assessing the degree of ownership of and participation in the new practice by key individuals and teams (cognitive participation)
3. identifying the work that individuals and teams do to enact the new practice (collective action)
4. exploring the perceived impact of the new practice on staff work and on patient outcomes (reflexive monitoring).

Exploration of these areas was aimed at informing conclusions about how CLECC could be optimised for impact and sustainability beyond the implementation period.

Pilot cluster randomised trial outcome measures

There is no single validated measure for compassionate care and we assessed its measurement across three complementary core outcomes at cluster level and at individual participant level: researcher-rated observations of the quality of staff–patient interactions, patient-reported evaluations of emotional care and nurse-reported measures of empathy.

Quality of staff–patient interactions

The quality of staff–patient interactions was assessed using the QuIS. QuIS is a time-sampling tool that measures both the volume and quality of interactions, through observation, enabling a calculation of how many patients experience one or more negative interactions during an observation session.42 QuIS interactions between staff and patients are coded as positive social, positive care, neutral, negative protective or negative (Table 5).

Although other observational tools have been developed for educational and service improvement purposes, they have not been validated as research instruments. QuIS has been used in a number of studies in NHS acute care settings for service improvement and evaluation, including use by the Health Advisory Service in their seminal evaluation ‘Not because they are old’.96 Other work has demonstrated that it is sensitive to changes in service quality.42,97,98 The selection of this outcome reflects a concern to measure patient-based outcomes rather than, as is often the case in compassionate care intervention evaluations, staff-based outcomes or process measures.16 Because it does not require any capacity to perform by patients, it is also potentially inclusive of people who would usually be excluded from research, for example people with dementia, people who do not speak English and people with communication difficulties.

Inter-rater reliability studies for QuIS have generally reported high levels of agreement, but these studies largely tested reliability by asking a second rater to categorise interactions based upon written descriptions by the first observer.99 In contrast, Dean et al. 42 tested rating reliability between two observers present during interactions (in a long-term care setting) and found that agreement was more variable than reported in other studies, although still acceptable (κ = 0.60–0.91). Prior to our own work, no other studies have examined the reliability of the QuIS in acute care settings using a method similar to that used by Dean et al. 42 No studies prior to our own research in acute care have directly demonstrated a relationship between QuIS ratings and patient experience.

The QuIS was originally designed for long-term settings, and so extended acute care definitions of the original five QuIS categories were developed (see Table 5 ) and tested for their validity and reliability by the team in early piloting work, together with guidance for using the instrument in acute care and an associated training protocol (see Appendix 3 ).99 The results from this early work are reported in Chapter 9, Quality of Interactions Schedule .

Patient-reported evaluations of emotional care

Patient-reported evaluations of emotional care were measured using the Patient Evaluation of Emotional Care during Hospitalisation (PEECH) survey tool.100–102 Although a number of survey instruments are now available that measure patient experience, most are limited in their capacity to assess patient experiences of the more complex relational aspects of care.100 Designed to address this gap, the PEECH tool focuses on the nature of interpersonal interactions with hospital staff and patient-reported assessment of the extent to which therapeutic emotional care has occurred.101,102 Originally developed in Australia, PEECH has since been validated for use in English hospital settings and can be completed by patients during a hospital stay with or without assistance.100 The subscales are security, knowledge, personal value and connection. PEECH is sensitive to changes in service quality and in ward environment.103

Nurse-reported empathy

Nurses’ self-reported empathy was measured using the Jefferson Scale of Empathy104 (JSE) (physician/health professional version), a 20-item inventory in a seven-point Likert-type format ranging from strongly disagree to strongly agree, with higher scores reflecting a more empathic orientation.104 Although caregiver empathy is recognised as an important component of compassionate care, the JSE is the only scale focusing on this concept that is designed for use in patient care contexts. Developed and validated for use by health-care workers, including nurses, the scale is sensitive to changes in individual empathy over time and context.105,106

Ward team characteristics

Baseline and follow-up data were also gathered to enable a description of the characteristics of the ward nursing teams involved in the study. These included qualitative interviews with nursing staff and the administration of a number of instruments through written survey. We assessed staff local working climate using Climate for Care (CC) and Factors that Enable Climate for Care (FECC) questionnaires,26 39-item and 19-item questionnaires, respectively, with answers on a 5-point Likert scale developed as part of a toolkit from a NIHR-funded project measuring culture change and quality of NHS acute hospital care for older people, with the ability to identify distinct nursing team climates.26 We also administered the Matron’s Assessment of Quality of Care (MAQC) and the Carer Experiences of Care (CEC) questionnaires from the same toolkit.26 We assessed nursing staff perceptions of workload using items from the International Hospital Outcomes Study (IHOS) battery,107,108 including (1) enough nurses on staff to provide quality patient care and (2) ratings of core care activities that were deemed necessary but left undone. This survey has been widely validated internationally and subjective ratings from it correlate with objective measures of both staffing and quality. We also measured levels of nursing staff burnout using the 22-item Maslach Burnout Inventory (MBI).109

Pilot cluster randomised trial randomisation

Ward teams (i.e. clusters) were randomly allocated following baseline data collection to participate in the intervention or act as a control. Randomisation was stratified by hospital and by ward type [medicine for older people (MOP) or surgical]. Randomisation was accomplished using the ralloc command in Stata^® 12 (StataCorp LP, College Station, TX, USA) and conducted by the study statistician. We planned for three wards (one surgical and two MOP) in each hospital. Wards were allocated to achieve two intervention and one control in each hospital, with one of the surgical wards allocated to intervention and the other to control. This strategy was to ensure that intervention implementation could be tested in two ward contexts in each hospital, and also to ensure that we gained experience of intervention and control conditions in both MOP and non-MOP ward specialties. We performed simple randomisation. First, we allocated the surgical wards to control or intervention. The MOP wards of the hospital with the surgical ward allocated to control were both intervention wards and, therefore, we did a second randomisation to allocate the MOP wards in the second hospital to intervention or control.

Pilot cluster randomised trial allocation concealment

Ward teams (i.e. clusters) were identified and recruited before randomisation. Clusters were randomly allocated to a group following baseline data collection by team members not involved in data collection. At follow-up, researchers conducting observations of the quality of staff–patient interactions were recruited from outside the core research team and not informed of allocation. Researchers gathering questionnaire data at follow-up and involved in qualitative interviewing were aware of ward allocation. It was not possible to conceal allocation from ward team nursing staff. Patients and visitors were not informed of allocation. The success of allocation concealment was tested as part of the feasibility work.

Progression to a definitive evaluation

As outlined above, the study was designed to lay the groundwork for a future definitive evaluation of the CLECC intervention. In the first instance, we planned that the process evaluation would enable an assessment of the CLECC intervention’s workability and integration into existing work practices, providing important information to guide its further refinement and implementation. Second, findings from the study would enable assessment of the feasibility of a future definitive evaluation against a number of important parameters. Results from the assessment of these parameters would then inform the design and implementation of a definitive evaluation, including sample size, level of clustering and selection of outcome measures.

The study took place on busy acute care wards and our target samples included frontline nursing staff and patients and visitors during the period of their hospitalisation. We set an explicit goal of maximising the participation of patients often excluded from research, that is, older people with complex needs including cognitive impairment and communication difficulties, as it is often people with these characteristics who are in greatest need of compassionate care. The study enabled us to develop and test a number of approaches to successful recruitment and participation. However, there is little in the literature to guide the recruitment of these groups in acute care settings, especially in relation to experimental studies. It was therefore not possible at the outset to quantify fully the study’s success criteria in relation to the number of patients or staff to be recruited. We instead set target recruitment rates that were reviewed and refined as data were collected and analysed.

The success criteria set for the study were:

1. completion of a process evaluation into CLECC’s workability and integration into existing work practices, sufficient to inform refinement and future implementation, and to inform future process evaluations
2. recruitment of sufficient wards (n = 6) to assess the feasibility of a CRT design to inform the design of a definitive evaluation, including information on participation and attrition rates, blinding strategies, mitigation of contamination, baseline rates and intracluster correlations for core outcomes, data collection and analysis procedures
3. recruitment or refinement of target numbers of staff, patients and carers to enable collection of data estimated to be sufficient to inform the selection and use of primary and other outcome measures in definitive evaluation.

Study progress against success criteria was externally monitored by the Study Steering Committee. A Study Advisory Group advised on the CLECC intervention implementation and a PPI group also advised the study team.

Patient and public involvement

Our consultative PPI work over 3 years on this topic confirms public recognition of the need for research of this kind and that the research addresses a topic of primary concern to the general public.

We have consulted with service users about this research since 2013, while we developed the CLECC intervention, and throughout this feasibility study. We recruited older members of the public who had experienced a hospital admission, or had been a carer of someone who had experienced a hospital admission. A PPI group (n = 5) started meeting in February 2013 to help develop the application and plan staff training for the feasibility study, and two members sat on the Study Steering Committee. A leading figure (Lesley Carter) from national charity Age UK (London, UK) also sat on our Study Advisory Committee. The initial PPI group input helped to identify priorities for the research outcomes, clearly showing that improvements to compassion in hospitals, and the involvement of patients and family in care were important outcomes for the PPI group. They also advised on the development of the CLECC intervention and its evaluation. Changes made as a result of PPI input include extending classroom sessions (part of CLECC intervention) to include registered nurses as well as care assistants, developing researcher guidance on approaching family carers to participate in research, and developing ways to include people with communication difficulties in the research. We also included PPI input in training our core research team. One of our PPI representatives (Jan Gollop) provided a half-day of training to research team members on conducting research on people with dementia. Her expertise from caring for her husband with severe dementia, and from her network of other carers, and experiences visiting hospitals gave her a unique insight into what researchers needed to know.

We also held a public consultation event in November 2015 to get feedback on the work to date, test our plans for definitive evaluation and recruit more PPI input. People who attended this event (n = 6) provided advice regarding the dissemination of feasibility findings, which informed the dissemination plan for both the feasibility study and the definitive evaluation proposed as the next step. People who came discussed the ‘value’ of a definitive study of CLECC, in particular its value for money in times of austerity, and concluded that compassion in hospitals was of such personal significance that a definitive evaluation did merit funding. These consultations have confirmed strong public support for the CLECC programme of work. The study addresses a topic of primary concern to the general public and our consultees have demonstrated very strong support for this work. Involving PPI representatives from both hospital sites has been particularly helpful in involving members of the public whose locality and experiences of the hospital in question help ensure not only patient involvement but also local involvement.

Ethics considerations

Our concern in this study was to keep the best interests of participating patients, visitors and NHS staff at the centre of what we did. We included a number of measures to help ensure this and carefully consulted with our PPI group and nursing representatives about our proposals over a 2-year period. We aimed for a proportionate approach that did not place undue burden on participants at any part of the process and also represented what we judged to be achievable as a research team working within limited resources. There was also the opportunity to pilot and adapt procedures within the feasibility framework.

One key ethics issue was the recruitment of and proposed data collection from patients, staff and visitors at what can be an already stressful time. We addressed this by developing concisely written information accompanied by verbal explanation and the chance to ask questions. We ensured that we included in written and verbal information the clear statement that people were not obliged to take part and that their care or treatment would not be affected in any way if they declined to take part or withdrew. We allowed people as much time as they needed to make up their minds about taking part. We ensured that research team members had the skills to identify distress caused by recruitment and/or data collection processes and had clear plans of action to follow if this happened.

Another ethics issue was the desire to include people who lack the capacity to make the decision to take part in the research. This is important because this group is often excluded from research and yet evidence suggests that they are most vulnerable to not experiencing compassionate care. We developed and implemented clear procedures to ensure that the principles of the Mental Capacity Act 2005 Code of Practice 110 and process consent were adhered to.111,112

A further potential issue was the participation of ward staff as research subjects and the concern that may be raised about their rights to refuse to take part or to withdraw from the study. Our communication strategy aimed to ensure that everyone who should know about the study was informed about it and their right to not take part. Researcher training emphasised issues relating to the anonymity of staff, visitors and patients. Clear procedures were developed to guide the reporting of unsafe practice.

Ethics approval for the study was granted by the national Social Care Research Ethics Committee (REC) (reference number 14/IEC08/1018) in December 2014. We originally applied for NHS REC approval, but were advised by the manager for the local REC that the study did not merit NHS REC scrutiny because the proposed intervention was not clinical. We appealed this decision, but it was upheld by the Health Research Authority regional manager. We requested an alternative, but equivalent form of review and were referred to the Social Care REC.

Research team and training for data collection

The core research team involved in data collection were the chief investigator (JB), two research fellows (LG and WW), the senior research assistant (HB) and the research assistant (PL). They conducted the qualitative interviews and were supported in screening, recruitment and data collection by others. Research nurses and clinical trials assistants (n = 18 over the course of the whole study) at the participating NHS trusts screened and recruited patients for observations. Research nurses, clinical trials assistants and eight other research assistants screened and recruited patients and visitors for the questionnaire survey and helped to complete questionnaires when this was needed. Core team members and research assistants undertook the patient observations. All staff involved in the research received classroom and field training, as set out below:

• recruiting for patient observations – 4 hours in the classroom and 4 hours in the field
• conducting patient observations – 4 hours in the classroom and 6 hours in the field
• recruiting for and conducting patient and visitor questionnaires – 4 hours in the classroom and 2 hours in the field.

Changes from original protocol

Two key changes were agreed with the Study Steering Committee and implemented. The first change was to the target sample sizes specified for questionnaire surveys with nursing staff, patients and visitors. This was made after baseline data collection when the feasible recruitment rates became clear. Tables 6 and 7 illustrate original and revised target recruitment rates.

The second change was to the economic evaluation. As the study progressed and the membership of the health economics team developed, it became clear that the health economic dimension to the evaluation would be a more helpful foundation to the definitive evaluation if it focused on how the costs and benefits of CLECC could be measured in a future definitive evaluation.

The original study objective related to the economic evaluation was:

To estimate the costs of the intervention and quality of life.

This was changed to:

To inform the measurement of costs and benefits of CLECC in a definitive evaluation.

Chapter summary

This chapter has included information on the study design, the feasibility parameters being tested and the outcome measures being assessed. PPI, ethics considerations and changes from the original protocol have also been explained.