Methodological frameworks
Integrated care raises major challenges for evaluation, reflecting the general pressure within health services research whereby increasingly complex service redesign requires rapid and rigorous evaluation.41,42
The evaluation of health technologies has been heavily influenced by the Medical Research Council (MRC) Complex Interventions Framework,43 and each of the mechanisms of integration in the SICP (MDGs, health coaching, community assets) would probably meet the conventional definition of a ‘complex intervention’ (i.e. ‘interventions with several interacting components’).
The SICP itself may be best characterised as a ‘large-scale transformation’, defined as ‘interventions aimed at co-ordinated, system-wide change affecting multiple organizations and care providers, with the goal of significant improvements in . . . outcomes’.44 ‘Complex interventions’ are nested within the SICP, but the large-scale transformation is not simply the sum of those interventions, but instead involves wider structural, organisational and cultural changes, which may serve to help or hinder the translation of individual mechanisms of integration into improved outcomes.
To answer our research questions, we adopted a mixed-methods approach using conventional health services research methods:
- We used qualitative methods (interviews and observations) to explore the implementation of the SICP, both at the level of leadership and management of the major organisations involved (implementation 1), and at the level of managers and clinicians involved in the everyday delivery of the intervention (implementation 2).
- We used analysis of routine data sets [Hospital Episode Statistics (HES)] and appropriate comparators and non-randomised methodologies45 to explore effects of the SICP on outcomes (‘outcome 1’).
- We conducted a formal randomised controlled trial (RCT) within the cohort (‘outcome 2’).
In addition, to provide flexibility to assess a variety of aspects of the SICP, we also adopted a patient cohort. This cohort provided an assessment of patient-reported outcomes (health, quality of life and experience of services), which are missing from many integrated care evaluations that use routine data.
The planned cohort also provided an opportunity to explore the innovative cohort multiple randomised controlled trial (cmRCT) design, which at that point had not received significant practical application.46 In this design, a large population cohort is recruited and followed over time. As well as providing an assessment of the impact of the SICP over time, natural variation in the exposure of patients within the cohort to different mechanisms of integration allowed more sophisticated modelling of effects.
In addition, the cmRCT provided a good conceptual ‘fit’ for the evaluation of health coaching within the CLASSIC study. One of the criticisms of RCTs is that they test innovations in a very selected group of patients, which then fail to ‘scale’ because of low rates of acceptability among the wider population. Pragmatic trials are in part a response to this, but they are still selective, as patients are selectively recruited on the basis of their willingness to engage with the intervention. A pragmatic trial may show effects, but can still be unacceptable to many patients who refuse to engage.
This is less of a problem in the evaluation of interventions where patients are seeking help. However, it has less relevance when an intervention involves services proactively identifying patients on the basis of risk. In the cmRCT, engagement in the wider population is, in principle, built into the design, alongside the usual impact of variable adherence (which is already built in to pragmatic trials).
The cmRCT was felt to be a relevant test of health coaching as applied in integrated care as a population health strategy, aimed in a preventative capacity for those at risk of poor outcomes (rather than a population identified on the basis of previous high utilisation).
Conceptual frameworks
A number of conceptual frameworks are of relevance. First, we drew on realist evaluations,28,47 which move beyond simple questions concerning whether or not an intervention ‘works’ to a more detailed assessment of ‘context’, ‘mechanism’ and ‘outcome’:
The complete realist question is: ‘What works, for whom, in what respects, to what extent, in what contexts, and how?’ In order to answer that question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context.
The process of a realist evaluation involves developing initial programme theories, conducting empirical work to test those theories and then analysing the relationships between context, mechanism and outcome to provide insights for those commissioning programmes.
An understanding of context is thus critical, as context can function to make particular mechanisms more or less potent. However, context is a complex concept.26,27 An analysis49 of the process of managing ‘strategic change’ highlighted the importance of what was being implemented (content – equivalent to realist ‘mechanisms’), how this implementation was being undertaken (process) and the context surrounding change, with a distinction between ‘inner context’ (including concepts such as strategy and culture) and ‘outer context’ (the wider economic, political and social situation). This work also identified a number of features relating to a ‘receptive context’ for change:
- quality and coherence and policy
- availability of key people leading change
- intensity and scale of long-term environmental pressure
- supportive organisational culture
- effective managerial–clinical relations
- co-operative interorganisational networks
- simplicity and clarity of goals and priorities
- fit between the change agenda and the locale.
There is already a significant body of literature on large-scale change. As noted previously, a model of ‘large-scale change’ has been developed, which summarises five ‘rules’ underlying such transformations.44
- ‘engage individuals in leading the change efforts’, highlighting the importance of both ‘designated’ and ‘distributed’ leadership
- ‘establish feedback loops’ concerning the collection and use of measures of progress (which can both help and hinder transformation)
- ‘attend to history’, in the sense of understanding previous efforts at change and their implications for current programmes
- ‘engage physicians’, as they are likely to be crucial to transformation efforts owing to their relative power and autonomy
- ‘involve patients and families’ to enhance outcomes.
In the area of integrated care, previous analyses have outlined important issues that need to be considered. A systematic review50 of factors that supported successful integration of health and social care for people with long-term conditions found seven studies and limited evidence overall, but highlighted a number of themes:
- colocation of staff and teamwork
- integrated organisations
- management support and leadership
- resources and capacity
- information technology.
The evaluation of the SICPs also included a detailed analysis of > 200 interviews conducted to drive a ‘bottom-up’ model of barriers to and facilitators of integration,51 many of which were felt to be common to any large-scale change:
- structure and characteristics of organisations and interventions:
- size and complexity of the intervention
- information technology
- relationships and communication
- professional engagement and leadership, credibility and shared values
- contextual factors:
- public service bureaucracy
- resources allocated to the programme
- external policy reform
- organisational culture.
As noted earlier, the ‘large-scale transformation’ of the SICP has complex interventions nested within it, which can be viewed as distinct health technologies using a more granular approach. The development and analysis of health technologies can draw on a number of conceptual frameworks, which often relate to the particular logic model underlying an intervention.
For example, normalisation process theory (NPT)52,53 offers a framework to investigate how complex interventions become embedded and become sustainable over time, based on four generative mechanisms: (1) coherence (what is the work to be done?), (2) cognitive participation (participants have to ‘buy in’ to the work, individually and collectively), (3) collective action (what work has to be done to enact and enable new practices?) and (4) reflexive monitoring (what work can be done to help appraise new practices?).
The analysis of complex interventions within the SICP can also draw on frameworks more specifically related to the particular interventions under test. In the current context, this would include psychological models of behaviour change underlying self-management, which use related concepts such as self-efficacy and patient activation54 to understand the mechanisms by which patients undertake self-management. It can also include clinically focused models, such as ‘patient centredness’,55 to explore how mechanisms of integration impact on patient experience of care, as discussed in Chapter 1.
Timeline
An illustrative timeline of SICP and CLASSIC activities is provided in Figure 3.
FIGURE 3 Illustrative timeline of the SICP. PROTECTS, Proactive Telephone Coaching and Tailored Support. (Figure not reproduced here; see the original article.)
Patient and public involvement in SICP and CLASSIC
Funding bodies require patient and public involvement (PPI) to ensure that research is relevant for its intended beneficiaries and that it prioritises issues of importance.
Where applied health research involves the development of interventions, and PPI is often focused on ensuring that those interventions are sensitised to the needs of patients. In the case of CLASSIC and the SICP, the interventions within the SICP involved patient input to the service development outside the formal research context of CLASSIC.
We now describe patient involvement in the initial development of the SICP and the more conventional PPI within the design and delivery of CLASSIC.
Public involvement in the SICP
The SICP aimed to improve person-centred care. Public engagement was undertaken throughout its formation (2011–12), via engagement activities undertaken by public governors and links with existing organisations (Salford Link Project and HealthWatch). As the SICP aimed to integrate health and social care, the programme was able to draw on existing reference groups.
Engagement with the wider community was required and in July 2012 an event was held with older people, which identified their priorities:
- reducing emergency and permanent admissions to nursing and residential homes
- enabling people to have more control over daily life
- improving satisfaction with care
- supporting people to die in their place of choosing.
Having developed these priorities, a ‘driver diagram’ was used to promote the SICP.
Public involvement was also used to modify the concepts behind ‘Sally Ford’, a character developed to help provide patient focus. By being able to comment on various iterations, group members aimed to make ‘Sally Ford’ more representative.
Additional input meant that ‘Sally Standards’ developed, which outlined how older people could help health and social care providers achieve their outcomes by taking a more active role in their own health and well-being. These underpinned the ‘My Well-being Plan’ developed in collaboration with the community assets team. Public involvement was central to the community assets workstream, with a mapping exercise identifying the unmet need for social groups for older people within neighbourhoods. Older people were encouraged and supported to apply for funding to set up and run local groups themselves. The community assets workstream was steered by its own patient group (the Community Assets Work Stream Group), who began their work by asking three simple questions: (1) ‘What motivates you?’, (2) ‘What makes you feel good?’ and (3) ‘How do you find out about things?’. This identified potential barriers (which included limited physical activity, lack of access to information, not eating well and being socially isolated), thereby forming the focus for ongoing work.
Patient and public involvement in CLASSIC
Ahead of our original NIHR application submission, we consulted with members of a Citizen Scientist Project (www.citizenscientist.org.uk; accessed 16 August 2018) based at Salford Royal NHS Foundation Trust, and other interest groups. In April 2013 we held an engagement event to discuss strategies to ensure older people’s active involvement in CLASSIC. This event highlighted the need for telephone support, engagement through social events (not just written materials) and the importance of peer networks in dissemination of information. The event also helped inform the development of the CLASSIC health coaching intervention.
The CLASSIC Study Advisory Group was formed following assistance from the engagement in research manager from the trust who attended our initial Study Steering Committee. The CLASSIC study was promoted on the Citizen Scientist website, which included an advert for advisory group members. For further meetings, two members of Primary Care Research in Manchester Engagement Resource (PRIMER) [http://research.bmh.manchester.ac.uk/PRIMER (accessed 3 April 2018)] (a local PPI group of long standing) were recruited specifically to work with the CLASSIC team. Its remit included overseeing management of the research, providing a patient voice and commenting on the emerging results and dissemination strategy.
Specific patient and public involvement activity within workstreams
CLASSIC cohort
Our researchers presented the cohort to a local group. In response to their feedback, we made changes to the survey, including increasing font size and type; improving questionnaire layout; providing an indication of time to complete; providing an explanation of why we are asking the questions; adding a statement regarding confidentiality, especially around data sharing; including examples of question completion; providing name and telephone details of a contact to assist with completion; and adding free-text boxes to enable people to add their own comments. The group also provided a ‘critical friend’ approach to letters and participant information sheets being developed to send to potential participants.
In March 2015, we presented to the PRIMER group, and its members provided advice on encouraging people to stay in the cohort and around how we fed back the results from CLASSIC. It was agreed that providing ongoing incentives would help retain participants.
Health coaching
At the meeting in March 2015, PRIMER members discussed the health coaching model, providing guidance on participant recruitment and retention with the telephone-based intervention.
Members of local groups were consulted in 2015 about recruitment methods and gave feedback that many older people were unlikely to answer their telephone to an ‘unknown’ caller. It was therefore agreed that a letter would be sent to potential participants, which included the telephone number that they would be called from, helping increase uptake.
Dissemination of CLASSIC evaluation results
Our two PPI representatives (PB and MM) have commented on the summary findings from the CLASSIC study and assisted with the Plain English summary. Dissemination is ongoing, and we anticipate writing a summary piece in collaboration with our PPI representatives for inclusion on the website (via the SICP communications team) and for inclusion in a local newsletter.
Our website [www.classicsicp.org.uk (accessed 3 April 2018)] will be a repository for publications arising from the CLASSIC study.