What is realist evaluation?
The realist evaluation approach was developed by Pawson and Tilley23 in response to the recognised complexity of evaluating new policies and has increasingly been used in health services research. It applies a realist philosophical stance to policy research. Realism is often described as sitting between positivist and constructivist approaches. While realists accept that a ‘real’ world exists, they also recognise that our understanding of reality will only ever be partial, and this understanding can be refined as new information is obtained. Realists recognise the interplay between structure (e.g. the way ED work is organised) and agency (the decision-making of individuals involved), in understanding causality.
In applying realist philosophy to evaluation, Pawson and Tilley23 advocate a theory-driven approach, recognising that interventions work differently when applied in different settings, generating a range of expected and unexpected results. Therefore, realist research questions explore what works, for whom and in what circumstances. Researchers then develop, test and refine theories which explain how and why interventions work.
A core premise in realist evaluation is that interventions work or do not work because of the way people respond to them that is, outcomes are generated by what realists call ‘mechanisms’. Furthermore, there is recognition that these responses to the resources an intervention offers may differ depending on the context into which interventions are delivered. Many factors such as individual views and preferences, the existence of competing interventions and the history of the organisation are recognised to have the potential to influence the success or failure of a new programme.24
Realist research therefore focuses not only on what outcomes occur in which contexts, but also on exploring how and why those outcomes occur in those settings by identifying the generative mechanisms. These mechanisms are often described in two parts, recognising that they include both the resources offered by the intervention and also the reasoning processes of the individual’s involved.25
The product of realist research is a programme theory which describes the way in which interventions delivered into a given context produce outcomes via a mechanism. Results of realist research are therefore usually expressed as context–mechanism–outcome configurations (CMOCs).
The overall aim of realist research is to generate transferable findings. While CMOCs configurations may initially be generated at the level of an individual or site, during analysis, researchers work to increase the level of abstraction away from individual data extracts and towards developing theories that are more ‘middle range’ in nature – that is, they may be relevant across a number of different settings.
‘Methodological pluralism’23 is typical of realist research. No single method is favoured; rather, methods are chosen for their ability to help develop, test and refine theories. For example, observations may provide useful contextual information about communication patterns between different professional groups, interview data can help to explore the reasoning processes used by staff selecting patients for review by a GP and waiting times data might be used to understand the desired outcome of moving patients efficiently through the department. No single type of data collection method is seen as inherently superior to another; data are instead judged based on their relevance to the theories under development and the rigour of the approach used.26 This approach allows for ‘nuggets of information’27 gained from a variety of sources to be brought together to offer a more complete explanation of causality.
Why use realist methods here?
Introducing GPs to work in EDs is a complex intervention, comprising multiple strategies (e.g. streaming and triage processes, individual clinical decision-making) and actors (including staff and patients). While several potential outcomes for GP-ED models have been suggested (see Chapter 1), we did not identify any existing logic model or theory of change which set out how the intervention was expected to create its effects. It is accepted that the evaluation of complex interventions benefits from a theory-driven approach, mapping out the expected pathways through which interventions are expected to generate outcomes.28 Exploring local contextual factors, at clinician–patient level (microsystem), team and organisational level (mesosystem) or wider level (macrosystem), is becoming an increasing focus of healthcare quality improvement to explain the outcomes.29
We therefore selected realist evaluation as an approach which would allow us to explore how a range of stakeholders expected that GP-ED models might work, and then to test these ‘folk conjectures’30 in a detailed and systematic fashion. Realist evaluation is well suited to using data from multiple sources and perspectives and synthesises these to produce explanatory theories.
We identified the importance of highlighting and exploring the complexity of implementing GP-ED models, as part of our evaluation, to allow commissioners and policy-makers to understand the range of potential impacts that such an intervention might have in their own setting.
By focusing on understanding the role of context, readers can assess the transferability of our findings to their own settings, potentially pre-empting barriers identified, and targeting delivery to settings where they may be most likely to succeed (see Project web page).
While the realist approach encouraged us to explore context in detail to understand causal pathways, the process of theory generation, testing and refinement also aimed to produce explanations that were adequately abstracted to be broadly useful and informative. See Table 2 for a list of realist terminology used within this report.
| Context (C) | Pre-existing conditions which may be at the individual, team or organisational level and influence the success or failure of different interventions |
| Mechanism (M) | The way in which the intervention works, generally comprises the resources that the intervention offers and the reasoning responses it generates |
| Outcome (O) | The intended or unintended results of the intervention caused by mechanism/s operating within a particular context |
| If–then–because statement | Statements developed during realist analysis as an accessible way to start thinking about causality when working with data. For example, if staff are experienced, then they are more likely to stream patients to a GP, because they have developed a good understanding of cases appropriate for GP care |
| CMOC | Realist theories are described as context–mechanism–outcome configurations (CMOCs) to demonstrate how mechanisms ‘fire’ in certain settings to generate the results described |
| Initial rough theory | An early theory, informed by available evidence, about how, why, for whom and in what circumstances the intervention is thought to work, described using a CMOC. Requires further testing |
| Refined theory | An initial theory that has been refined using primary or secondary evidence |
| Programme theory | An overall high-level theory summarising how the intervention works, using the theories developed from the data |
| Formal theory | Existing social theories used as a lens through which to examine the data. Also known as middle range theory or substantive theory (e.g. Crosskerry’s dual process model of clinical reasoning31 used in Chapter 4) |
| Realist interviews | Use a ‘teacher–learner’ approach in which the researchers’ theories are openly discussed. Teaching the interviewee about the theories being tested allows the researcher to learn from the interviewee’s response |
How we have taken a realist approach to the evaluation
We have applied a realist lens throughout the evaluation process, which influenced data collection and analysis in a number of ways highlighted below. The specific details of the methods used at each stage of the study are described in detail in the chapters that follow.
We designed a mixed-methods study with several phases, including in-depth case studies at multiple sites to adequately explore context. The wide range of data sources used (highlighted in Figure 1) recognises the value of different types of data for both theory building and theory testing.
We engaged early and often with key stakeholders. At the outset of the project, we harnessed their understanding of how GP-ED models work ‘in real life’ to aid theory development, while two national stakeholder events during the data collection and later data analysis stages meant their input continued to shape theory development and refinement. This should help to maximise the relevance of our findings in practice, ensuring we are addressing the questions that are important to staff involved in implementing these models.
The realist approach encouraged us to focus on causality, trying to understand how and why outcomes were produced throughout the study. This meant that we started our theorising and analysis early in the project and continued several iterations of the realist evaluation cycle throughout the study (theorising, data collection and testing, refining and further theorising).23
For example, undertaking our realist review (see Chapter 3) enabled us build theories which were taken forward for testing at the case study sites. These sites were purposively selected, not only to represent a range of different settings (urban and rural, different GP-ED models etc.) but also on the basis of their ability to provide rich explanatory data. This was accomplished by interviewing CDs before case study site selection, targeting sites which in the survey had described33 particular successes and failures they had observed using GP-ED models (see also Chapter 3).
Our initial theories provided the researchers with a way to structure their interactions with staff, using a teacher–learner approach in which our developing ideas were shared for staff to expand upon or challenge.34 As the project progressed, and the researchers collecting the data were also involved in ongoing data analysis and theory development, it was possible to focus later observations and interviews towards trying to expand upon and contextualise findings from earlier sites. Our ongoing focus on understanding causality encouraged the researchers to tailor data collection to explore relevant areas of theory emerging, for example, exploring how teamwork and communication was encouraged across different GP-ED models.
Our theory-driven qualitative data analysis (described in detail in Chapter 4) used both deductive reasoning (by comparing the data gathered to our initial rough theories) and inductive reasoning (developing new theories from the data), as well as the realist process of ‘retroductive reasoning’. Retroduction is an interpretive process used by realist researchers to understand causality. It involves identifying or suggesting the causal mechanisms which could explain events or patterns seen in the data.33,35 As causal factors may often be unobservable (e.g. status, power) researchers must interpret the data to make sense of the observations made. This process was done both by individual researchers as they developed if–then–because statements from the data, and also collaboratively as a team when refined context–mechanism–outcome (CMO) statements were developed.
We also chose to use formal theory as a further analytical lens during the data analysis stage.36 Pawson and Tilley advocate for the application of existing formal theory in realist evaluation.23 Formal theory can be useful to confirm the coherence of the theories developed during the evaluation.37 When a match can be identified between the empirical data collected and the relationships already described in a formal theory, the causal explanations generated can be strengthened.38 Generally, realist evaluations utilise theories developed to explain a specific phenomenon39 (e.g. Croskerry’s dual process model of clinical reasoning;31 see Chapter 4) and we used several different formal theories to aid our interpretation of the data (also principally Chapter 4).
Chapter 7 describes how we used the quantitative data described in Chapter 5 to further test our developing theories. This triangulation process helped to increase our confidence in some of the theories developed, while challenging others and identifying areas where we could not be as confident in our findings.
Finally, as with all realist studies, a key output is the refined programme theory of GP-ED models, described in detail in Chapter 7. The RAMESES guidance on the reporting of realist evaluation studies has been followed in the production of this report.
Marker conditions
At the outset, we identified that many of the routinely collected outcome data we planned to analyse were at a departmental level, for example overall attendances and admissions. However, one of the key premises of GP-ED models is that changes at a departmental level may occur because GPs might manage certain groups of patients presenting to the ED with particular ‘marker conditions’ differently to ED clinicians (e.g. differences in use of investigations, time spent observing patients in the department, admission rates). The aim of selecting marker conditions was to provide further data to test our theories that some conditions may be managed differently by GPs and ED clinicians and to explore whether the management was influenced by the GP-ED model in place. We also chose conditions that contributed information about resource use (investigations/consultation times/admissions) and patient experience to supplement the routinely collected outcome data. We wanted to consider both conditions that GPs managed well in the ED setting and others that they may not manage so well.
We planned to use these marker conditions at our case study sites to identify patients (or parents/carers) to interview (see Chapter 4). The marker conditions were occasionally used when discussing theories during the staff interviews. We also collected local data about investigation use and hospital admissions for these conditions at one site for health economic analysis (see Chapter 6).
To identify potential marker conditions, we initially undertook a scoping review of the literature on ED relevant ambulatory care sensitive conditions (ACSCs) (described below). The shortlist generated was then subsequently refined using additional data from our national survey, a ranking exercise within the co-applicant team and a further consensus exercise at our first stakeholder event (all described in Chapter 3). We looked to identify marker conditions that could potentially be seen by a GP or ED clinician in any GP-ED model.
Scoping the literature on emergency department relevant ambulatory care sensitive conditions
Ambulatory care sensitive conditions are conditions where effective community care and case management can help prevent the need for hospital admission. This work was conducted in July and August 2017 by a medical student (FH) with support from the team (AC and FD) and an information specialist.
A MEDLINE search generated 477 hits, of which 347 were relevant after abstract screening. Of these 347 papers, 224 used a defined list of ACSCs, with most referencing 9 core papers.40–48 From the 224 papers, 206 ACSC terms were identified. These 206 conditions were reviewed by 2 academic GPs (AC and FD) to identify potential marker conditions. Acute confusion was not included in the list but was included following discussion. Similar conditions were grouped together, for example, acute exacerbation of asthma and chronic obstructive pulmonary disease (COPD). Nine groups of diagnoses were generated. These were matched with an appropriate presenting complaint described in the ECDS and are shown in Table 3.
| Diagnosis | ECDS chief complaint |
|---|---|
| 1. Acute confusion | Drowsy (altered level of consciousness)ConfusionHallucinations/delusions |
| 2. Acute exacerbation of asthma Acute exacerbation of COPD Influenza and pneumonia | Short of breathDifficulty breathingNoisy breathing |
| 3. Musculoskeletal back pain | Pain in back/trunk (no injury) |
| 4. Cellulitis | Localised swelling/redness/lumps/bumps |
| 5. Anxiety/depression | Depressive disorderAnxiety disorder |
| 6. Otitis media Tonsillitis Upper respiratory tract infection | FeverEar painSore throat |
| 7. Gastroenteritis | Abdominal painDiarrhoeaVomiting +/− nausea |
| 8. Migraine | Headache |
| 9. Pyelonephritis Urinary tract infection | Pain on passing urineFrequent urination |
This list of conditions was used as a starting point for discussion during Phase 1 and was subsequently refined at our first stakeholder event (described in Chapter 3) seeking to identify up to five conditions. We intended that this number of conditions would be adequately diverse to represent differing experiences for patients, while also providing adequate data about each specific condition to allow for helpful comparisons to be made within and between GP-ED models.
Conclusion
This chapter has outlined the rationale for using realist methods and described our application of a realist approach. The chapters that follow now describe the specific methods used at each stage, alongside the results.