Review methods

We followed a realist approach to evidence synthesis to understand the contexts in which remediation interventions are most effective. Realist review is a practical, methodological approach designed to inform policy and practice. The realist review approach is distinct from other types of literature reviews, as it is based on an interpretive and theory-driven approach, synthesising evidence from qualitative, quantitative and mixed-methods research.28 The unique contribution of this approach is that it yields transferable findings that explain how and why context can affect outcomes.28 It does so by developing programme theories that explain how, why, in what contexts, for whom and to what extent interventions ‘work’.29,30

Realist review methods are particularly suited to research on the remediation of doctors, as they focus on the contextual factors that determine the outcomes of an intervention.31 Like other interventions that seek to promote behavioural change, remediation is highly context dependent (i.e. the same intervention will vary in its success depending on, for example, who delivers it and how it is delivered, the characteristics of the learners, the circumstances surrounding it, and the tools and techniques used). Research designs that seek to ‘strip away’ this context limit an understanding of ‘how, when and for whom’ the intervention will be effective.31 A realist review takes context as central to any explanation by exploring how an intervention manipulates context to trigger mechanisms that cause behavioural change.

The research design is illustrated in Figure 1.32 The plan of investigation followed a detailed protocol based on Pawson’s33 five iterative stages for realist reviews: (1) locating existing theories, (2) searching for evidence, (3) selecting articles, (4) extracting and organising data and (5) synthesising the evidence and drawing conclusions. The review ran for a 22-month period from April 2018 to January 2020. The review was registered with PROSPERO (CRD42018088779) and the protocol was published in BMJ Open.34 The review was informed by the quality and publication standards and training materials for realist reviews that were developed by one of the core research team members (GW).35 We were granted ethics clearance by the University of Plymouth, Faculty of Health and Human Sciences and Faculty of Medicine and Dentistry Research Ethics and Integrity Committee.

FIGURE 1 Flow diagram of the project. Dashed arrows to indicate iteration where necessary. Reproduced with permission from Wong et al.32 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure. (Figure not reproduced here; see the original article.)

Stakeholder group

A diverse stakeholder group was recruited for the RESTORE review to provide subject knowledge for programme theory refinement, to optimise dissemination plans and to aid the generation of feasible and practical recommendations. A total of 12 people were consulted throughout the review, including doctors who have undergone a remediation programme, personnel who identify underperforming doctors and initiate involvement in remediation programmes, personnel involved in the delivery of remediation programmes, responsible officers, remediation coaches, remediation researchers, patient and public representatives, and members of relevant medical bodies. Two-hour meetings with stakeholder group members took place at regular intervals throughout the project. Individual telephone calls were also held if stakeholders were not able to attend meetings (n = 4) and e-mail exchange was also used. A list of the meetings that took place is presented in Table 1. The table includes the number of participants who attended each meeting and the topics discussed. The review team members also attended stakeholder meetings.

In the early stages of the study, the review team struggled to recruit to the stakeholder group any doctor who had undergone remediation. Members of the review team and stakeholder group used existing contacts to try to recruit remediated doctors. The professional support unit at Health Education England South West (Bristol, UK) was also approached by NB and they e-mailed all of their trainees undergoing professional support to ask if they were interested in being involved in the stakeholder group. As undergoing remediation is a sensitive issue, we offered the remediating doctor the option of feeding into the stakeholder group via individual telephone calls. Finally, we recruited a remediated doctor to the group in January 2019. TP had two meetings with this stakeholder who provided valuable insight into the remediation process.

We organised stakeholder meetings to take place at the University of Exeter Medical School (Exeter, UK). At each meeting, a brief slide presentation was given by the review team at the start of the meeting to introduce stakeholders to the topic under discussion. The programme theory was presented to the group in the form of diagrams and statements in order to obtain stakeholders’ feedback. The final stakeholder group meeting focused on the dissemination of the findings. The meetings were facilitated in an inclusive way, providing everyone the opportunity to contribute and voice their opinion, whether or not they were in agreement.

All stakeholder meetings were audio-recorded. The recordings were not used as a form of data, but were used to enable the project team to focus on the meeting, as opposed to being distracted by note-taking. Stakeholders were informed of the purpose of the recording and verbal consent to audio-recording was gained at the start of each meeting. Detailed summaries of the meeting were produced by TP and shared with all members of the stakeholder group, regardless of whether or not they had attended the meeting. The meeting notes were used to orientate the review and to inform programme theory development, and were not used as primary data for analysis. The report does not include any direct quotations from these meetings.

The input of stakeholders in the review provided a reality check based on their ‘on the ground’ experiences of remediation. The use of realist review terms was kept to a minimum in the meetings to avoid discussions focusing too much on methodological concepts. Stakeholder involvement also contributed significantly to refinement of recommendations and dissemination of findings in accessible formats.

Patient and public involvement

The RESTORE review included strong patient and public involvement (PPI) throughout the project. LW, our PPI representative, was a co-applicant and a member of the review team and attended all of the team meetings, meaning that she had direct input into the management of the study from start to finish. As well as LW, we also had further PPI input in the stakeholder meetings from a representative from an existing PPI forum that we had established for two other studies on revalidation, funded by the GMC and the Department of Health and Social Care. In the stakeholder meetings, patients and members of the public provided significant input to programme theory development, often highlighting unique aspects of the remediation process.

Steering group

A steering group was set up for the project and comprised the review team plus representatives from the finance department and research and innovation department at the University of Plymouth. The steering group met four times throughout the study (June and October 2018, and May and November 2019). During these meetings, the steering group was updated about the progress of the study, provided scientific and budget oversight, and made sure that the project was delivered as proposed in the protocol. The steering group ratified all changes that were made during the study (e.g. the change of principal investigator and other staff changes).

Step 1: locate existing theories

The purpose of this step was to locate existing theories that explain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work. This involved identifying the theories that explain how remediation interventions are supposed to work to bring about behavioural change in clinical settings. Having already established from previous systematic reviews on the topic25–27 that there is limited theory underlying existing remediation interventions, the realist review approach allowed for the literature net to be cast wider to include literature from other fields and other professions, where potentially similar mechanisms may be in operation.

An initial programme theory was devised by TP and NB during the funding application phase of the project (see Appendix 1). This initial programme theory included some early thoughts on the mechanisms that may interact with important contexts to produce certain outcomes. The elements incorporated in this programme theory were identified through literature included in the personal libraries of TP, NB and JA, which had been carefully and purposefully collected on the topic area. This initial programme theory was then shared with the review team and the stakeholder group for further refinement. The informal searches carried out in step 1 were different from the more formal searching that was carried out in step 2 (see Step 2: search strategy), as their purpose was to quickly identify the kinds of theory that may be relevant. Therefore, exploratory and informal search methods, including citation tracking and ‘snowballing’, were used.36 We carried out these informal searches during April and May 2018. We used terms such as ‘remediation’ and ‘underperformance’ in Google Scholar (Google Inc., Mountain View, CA, USA) and MEDLINE databases. Once the theories had been identified, we built an initial programme theory to test in the review.

Step 2: search strategy

Formal search

In step 2, the goal was to find a body of relevant literature to further develop and refine the initial programme theory developed in step 1. The searches were designed, piloted and carried out by AW, an experienced information specialist with expertise in carrying out searches for realist reviews. AW used MEDLINE (via Ovid) to develop the search strategy. An iterative process was adopted, whereby search terms were added, removed and refined to achieve a balance of sensitivity and specificity in the results. The overall aim was to capture a broad range of relevant literature while minimising irrelevant literature.

As the search developed, sample sets of results were screened by TP and NB. This facilitated the selection of relevant search terms. In addition, highly relevant studies that had already been identified by the review team through their own personal libraries were used to test the search strategy (i.e. if these papers were returned in a particular search it confirmed the effectiveness of the search). The final search strategy used a range of search terms for the concepts ‘remediation’, ‘performance’ and ‘doctors’, which were combined using the AND Boolean operator (see Appendix 1).

The following databases were searched on 4 June 2018: MEDLINE (via Ovid), EMBASE (via Ovid), PsycINFO (via Ovid), Health Management Information Consortium (HMIC) (via Ovid), Cumulative Index to Nursing and Allied Health Literature (CINAHL) (via EBSCOhost), Education Resources Information Center (ERIC) (via EBSCOhost), Applied Social Sciences Index and Abstracts (ASSIA) (via ProQuest) and Database of Abstracts of Reviews of Effects (DARE) (via the Centre for Reviews and Dissemination). The search syntax and indexing terms were amended where needed from the original MEDLINE for use in these databases.

The search results were exported to EndNote X7 (Clarivate Analytics, Philadelphia, PA, USA) and deduplicated using the ‘find duplicates’ function. The search strategies for the main database search are reproduced in full in Appendix 1.

Inclusion criteria

The following inclusion criteria were applied.

• Aspect of remediation, including all documents that focus on the remediation of practising doctors. We defined remediation as ‘an intervention, or suite of interventions, required in response to assessment against threshold standards’.37
• Study design, including all study designs.
• Types of setting, including all documents about primary or secondary care settings.
• Types of participant, including all practising doctors. A practising doctor can be defined as a licensed doctor who has graduated from medical school and is practising medicine. Studies about only medical students were excluded.
• Outcome measures, including all remediation-related outcome measures.
• Language, including studies published in the English language.
• Publication date, including all studies published up until June 2018.

Additional searches

Although we conducted a grey literature search in the main search using HMIC, following a discussion with our third stakeholder group, where some of the stakeholders highlighted the relevance of some key grey literature, we decided to do a subsequent search specifically to identify grey literature. Searches for grey literature were conducted on 25 June 2019. The following databases/websites were searched: Google Scholar, OpenGrey, NHS England, North Grey Literature Collection, National Institute for Health and Care Excellence Evidence, Electronic Theses Online Service, Health Systems Evidence and Turning Research into Practice. The databases were searched with free-text keywords and controlled vocabulary where appropriate, using terms such as remedi*, reskilling, and retraining, combined with the concept of doctors.

Citation searching was undertaken, including searches of the reference list of all included documents and ‘cited by’ searches of certain papers that were particularly rich in terms of building the programme theory. The ‘cited by’ searches were undertaken using Google Scholar. Stakeholders were also asked to identify any literature they thought relevant. Any literature identified through grey literature searching, citation searching and via stakeholders that satisfied the inclusion criteria were included in the review.

The supplementary searches were purposive and undertaken on an ad hoc basis, as and when more data were needed on specific theories regarding different aspects of our programme theory of how remediation is supposed to work (e.g. insight, dissonance, psychological safety, feedback and behaviour change). Google Scholar was searched using topic keywords. Once relevant references were found, backwards and forwards citation searching techniques were used to identify further relevant papers.

Step 3: article selection

Documents were selected based on relevance (i.e. whether or not data can contribute to theory development and refinement).38 We did not assess the rigour of the included studies, as we had already established from previous systematic reviews on remediation that existing literature was of poor quality. An initial random sample of 10% was assessed and discussed by TP and NB to ensure that selection decisions were made consistently. TP then screened all the remaining titles and abstracts to ensure that they matched the inclusion and exclusion criteria. A 10% random sample was reviewed independently by NB to ensure consistency around the application of the inclusion criteria. Very few inconsistencies were identified, and these were resolved through discussion. All of the screening was carried out in Rayyan. Rayyan is a free web application designed to help reviewers manage the screening process. If TP was uncertain over the relevance of an article, then it was initially discussed with NB. If it was still unresolved after discussion with NB, then it was discussed with the review team. The full texts of included articles were screened again using the inclusion and exclusion criteria. Again, a 10% random subsample was reviewed independently by NB to ensure consistency around the application of the inclusion criteria. No inconsistencies were identified. The results were screened in alphabetical order. TP read all of the included papers and ultimately included all documents or studies that contributed to the development of some part of the programme theory.

Grey literature searches were exported into a Microsoft Word document (Microsoft Corporation, Redmond, WA, USA) and screened in Word by TP. NB carried out a 10% check to ensure consistency in article selection.

Step 4: extracting and organising data

Once article selection was complete, TP uploaded the electronic versions of the included articles into NVivo 12 (QSR International, Warrington, UK) for further analysis. Articles were coded in alphabetical order. The coding concentrated on the conceptual level of the data, categorising content into analytical categories to enable these to be refined further. Therefore, we did not code the data in terms of context, mechanisms and outcomes at this stage, but approached coding with an open mind to understand what themes were emerging from the data. Initial coding categories included barriers to and facilitators of remediation, strategies employed by remediation programmes and processes engendering change. Coding was both inductive (i.e. the codes were identified from analysing the literature) and deductive (i.e. the codes generated were informed by the initial programme theory, stakeholder group discussions and exploratory literature searches). For example, insight was identified as an important theme in the first stakeholder meeting and was present in some of the literature. However, the way that insight was described by the stakeholder group covered a wider range of subthemes than used specifically in the literature under the term ‘insight’. We therefore went back to the literature to look for data on processes related to the way in which our stakeholder group had described the concept of insight.

When conceptual coding was complete, we began to consider whether or not the categories (or subcategories within them) contained data that could be identified as contexts, mechanisms and outcomes. In other words, we applied a realist logic of analysis to make sense of the data (see Step 5: synthesising evidence and drawing conclusions). Initially, we focused on the returned literature that was richest in terms of theoretical depth. This allowed us to build context–mechanism–outcome configurations (CMOcs) that could be tested with the data from the wider literature.

To develop and test (i.e. confirm, refute or refine) the CMOcs, TP presented the CMOcs in narrative form on a Word document, along with extracted data to support the CMOcs plus a descriptive explanation of each CMOc. These CMOcs were examined by the review team and further refined. CMOc development was conducted using a realist logic of analysis, whereby we first considered the outcomes (intermediate and/or final desired) that had been identified in the literature and by our stakeholder group, and then worked backwards (using retroduction where needed) to infer the mechanisms that might generate the outcome, and the contexts created by the remedial intervention that might trigger the mechanism. In most cases, although the mechanisms were sometimes alluded to, they were not directly referred to in the literature itself. In these cases, the review team would suggest mechanisms that offered a potential ‘fit’ with the data. At this point, we conducted further supplementary searches to examine the literature relating to these mechanisms and to test whether or not they offered a viable explanation in the CMOc. In some cases, mechanisms were derived from substantive theory (the role of which is described in Use of substantive theory).

Throughout this process of CMOc development, we compared the CMOcs with the evolving programme theory. The programme theory was updated periodically by TP, presented in diagrammatic form with accompanying narrative explanation and discussed at review team meetings. It was important to consider the relationship between the CMOcs and the programme theory. Although, for the main part, the CMOcs would inform the shape of the programme theory, its overall shape enabled us to consider other potentially important outcomes that required further investigation. It also enabled us to consider the linkage between CMOcs and the sequence of intermediate outcomes.

Throughout the review, the programme theory was iteratively refined based on our interpretations of the data included in the literature. The characteristics of the documents were extracted into a Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA) spreadsheet separately for the results of the main search and the supplementary searches (see Appendices 2 and 3).

Step 5: synthesising evidence and drawing conclusions

A realist logic of analysis was used to interrogate the data and develop and test the initial programme theory, which we used to explain what it is about remediation of doctors that works and for whom, in what circumstances and respect, and why. To develop, refine and test the programme theory, we moved between the data (as extracted and coded in NVivo and Word documents) and the analysis, advice and feedback offered by the review team and the stakeholder group. To operationalise a realist logic of analysis, we asked these questions of our data (Box 1).

Throughout the review, we moved iteratively between the analysis of examples, refinement of programme theory and further iterative searching for data to test specific parts of the programme theory. The final realist programme theory is presented in a diagram and through a narrative description of CMOcs.

As mechanisms were often hidden or not articulated very well, we used retroductive reasoning (see Glossary) to infer and elaborate on the mechanisms. Retroductive analyses are analytical processes that seek to identify the hidden causal processes that lie beneath identified patterns or changes in those patterns.39 Therefore, our approach involved repeatedly going from data to theory, to refine explanations about the occurrence of certain behaviours. We tried to construct these explanations at a level of abstraction that would encompass a range of phenomena or patterns of behaviour.

We tried to identify relationships between contexts, mechanisms and outcomes within individual studies, but also across different sources (i.e. inferred mechanisms from one study could help explain the way contexts influenced outcomes in another study). The synthesis of data from different sources was often required to compile CMOcs, as not all parts of the configurations were always present in the same source.

In summary, the process of evidence synthesis was achieved by the following analytical processes:

• juxtaposition of sources of evidence (i.e. where evidence about behaviour change in one source allows insights into evidence about outcomes in another source)
• reconciling of sources of evidence (e.g. when results differ in similar situations, these are further examined to find explanations for these differences)
• consolidation of sources of evidence (i.e. where different outcomes occur in similar contexts, reasons can be developed as to how and why these outcomes happen differently).

Use of substantive theory

In realist reviews, a substantive theory is an existing and established theory within a particular subject area that describes patterns of behaviours at a greater level of abstraction from the intervention than is the aim of a realist review.35 A substantive theory can help to make sense of emerging CMOc patterns and, therefore, aid in their development and/or may be used to provide an analogy (i.e. situate the causal explanation provided by the CMOcs with what is already known from existing research).

In some cases, substantive theories, particularly related to behaviour change, were cited or alluded to in the returned papers. However, much of the literature was not very theoretically rich. In these instances, substantive theory drawn from outside the returned literature became an important component in CMOc development. Some of the theories that informed the development of the programme theory related to the way in which remediation was conceptualised. Early in the review, through discussions among the review team and the stakeholder group, we began to conceptualise remediation as practice change, rather than an educational process. This drew us to theories of behaviour change to help make sense of the intermediate outcomes in the remediation process. In other cases, substantive theory would be drawn from previous realist research that considered interventions with similar contexts.