Overview
This report presents, discusses and reflects on the experiences of the three research centres that conducted evidence syntheses to inform health and social care service organisation and delivery in the UK between April 2017 and June 2020. These responsive and often rapid reviews were commissioned by the NIHR HSDR programme, although the direct customers and audiences of different reviews included policy-makers, service commissioners, service providers and managers, or particular types of care professionals working in the UK health and social care sector.
This discussion and reflection on our experiences of the process of review scoping and review question formulation is based on the following:
- the collation and writing of eight descriptive case studies of selected reviews within each of the three teams
- a broad and basic thematic analysis of key issues or common considerations that shaped the processes of review scoping and question formulation
- a discussion and reflection on the overall process and practice of scoping for such policy, service and practice-responsive evidence syntheses.
The process comprised within-centre reviewing of documents, review team recollection, reflection and discussion (to produce the descriptive case studies) and between-centre discussion and reflection on our shared experience of scoping selected review topics, primarily through consideration of each other’s experiences, as written in the case studies.
We did not explicitly aim to document how our centres and teams differ in academic experience, team organisation or working practices; however, from past collaborations and experience (e.g. on public health reviews and technology assessments) we typically learned that we worked in similar ways in most respects. The HSDR centres at Exeter and York had smaller core teams that conducted most of the work on every review project; however, the Sheffield team typically made use of a wider group of researchers, with different people working on different reviews. Otherwise, we were not aware of major differences in how review teams within the three centres worked.
Choice of case study reviews
In May 2020, all researchers (the co-authors) in each of the three centres were e-mailed and asked to select two or three examples of their reviews, commissioned since April 2017, where ‘the scoping was challenging, interesting, and demonstrates a variety of approaches, or where the teams believe it was particularly critical to the ultimate delivery, quality and usefulness of the review’. The final number provided by each centre reflected the diversity of topics covered and challenges faced, but also the capacity and time resources of the researchers within teams to create these retrospective accounts of scoping. All case studies suggested by centres were written up and included.
Data sources and process
Exploration of the scoping processes for each case study drew on a combination of researcher recollection, review of notes and meeting minutes (e.g. with expert stakeholders) from within teams, e-mail correspondence with stakeholders, scoping searches and search results, from first allocation of a review topic through to review protocol agreement (with the NIHR HSDR programme or the policy customer). However, the extent to which the case study was grounded in or checked against documentary evidence (e.g. notes of meetings with service commissioners or HSDR contacts), rather than the recollections of review team members, was not rigorously documented. In most cases, the initial draft was written by one lead researcher, before elaboration and revision by other members of that review’s team.
Case study content
To simplify the process of writing and to best enable cross-case comparisons, we sought a similar structure and level of content for each case study. A draft case study was prepared by the Exeter team and shared across all three teams with a proposed set of headings and content:
- introduction of the basic context and origin of each review and topic (identifying the review team, the supporting protocol and main academic output)
- a statement of the final review questions or aims
- a description of the key challenges or choices during scoping, plus key decisions made in response to them, to illustrate how the team moved from an original review topic to specific review questions and the detailed review approach
- a (within-case) reflection and discussion section.
This broad structure was agreed by all co-authors for use in all case studies, although identification of key challenges and key decisions was not constrained to the subheadings.
Thematic analysis
Drafts of all eight case studies were shared across the three teams prior to a joint teleconference. One of the co-authors (AB) drafted a provisional thematic analysis based on reading these drafts. This framework of 14 distinct scoping considerations was informed by earlier conversations on the focus of the report, which had identified the combined influence and inter-relationships between stakeholder-/user-related factors and review team/technical factors. It also drew on factors identified in a systematic review of evidence use by policy-makers12 and sought to reflect the twin emphases of an associated report for policy-makers and systematic reviewers.13 Finally, the conceptual model was informed by the RETREAT (Review question, Epistemology, Time/Timescale, Resources, Expertise, Audience and purpose, Type of data) framework of review considerations, with review question, epistemology, and audience and purpose collectively reflecting the user-related factors; and time, resources, expertise and type of data capturing the technical requirements.14 The RETREAT framework was developed (by co-author AB) from an analysis of the attributes of qualitative evidence synthesis methods in 26 methodological papers, and has been shown to both distinguish and inform selection of synthesis approaches.15
The initial tabulation of scoping considerations organised themes according to whether or not they primarily related to:
- consultative issues (i.e. externally generated issues relating to input from commissioners, stakeholders, experts and patient groups to inform the planned evidence synthesis product)
- interface issues (i.e. issues relating to the interaction between the technical processes of the review team and the requirements of the review user)
- technical issues (i.e. internally managed issues relating to the conduct of the review, as experienced within the review team).
The first draft of the thematic framework was shared with all co-authors and commented on using e-mail and tracked changes. It was also discussed in a teleconference. This led to several revisions of wording and clarified definitions, but all 14 originally suggested considerations in the framework were retained and no new ones added. The framework was used by the lead author to allow similarities and differences in scoping processes and outcomes to be more easily identified as a basis for drafting this report’s discussion. Co-authors were also encouraged to consider the framework in relation to reviews that they had contributed to, which enabled further mapping of these scoping considerations to the case study reviews.
Ethics considerations
As this report is not based on any formal process of data collection and analysis, and did not formally recruit any participants from which data were obtained (e.g. interviews), it would not be defined as research and, therefore, did not require research ethics approval. The main contributors were the team members and co-authors of this report. Nevertheless, the scoping processes described in this report sometimes closely involved other individuals within the NIHR, DHSC and other national and regional organisations linked to NHS service commissioning and delivery. Although such individuals are potentially identifiable from the report and the underlying review reports of the eight case study reviews, in all cases, we ensured that the degree of anonymity in this report was the same as in the report already in the public domain.