Background

Systematic reviews for policy and service improvement

For systematic reviews to be useful, rigorous and deliverable within given resources, they need to articulate a well-defined review question. The bold advice in one of the more authoritative methods textbooks in the field is ‘Never start a systematic review until a clear question (or clear questions) can be framed’.1 Useful, deliverable systematic reviews also require an appropriately bounded set of inclusion criteria that together describe the nature of the evidence and types of studies that should answer the review question.1–3 The review question and inclusion criteria form the core information of systematic review protocols, driving the specification of subsequent aspects of the methods (i.e. search strategies, data extraction plans, tools for study quality assessment and strategy for evidence synthesis). In turn, prespecified and registered systematic review protocols seek to assure rigour and transparency in the conduct of systematic reviews.4

Policy-makers and service commissioners frequently express a desire to use evidence, but acknowledge that they often lack the time or relevant skills to explore and specify which type of research evidence would best inform a policy or service commissioning choice. Therefore, teams of experienced systematic reviewers and information specialists are often commissioned to undertake such responsive review work with and for them. The model of having university-based research centres, commissioned for a number of years to conduct highly applied work for the Department for Health and Social Care (DHSC), National Institute for Health and Care Excellence (NICE) or the National Institute for Health Research (NIHR), is a key feature of the applied ‘evidence ecosystem’ to inform UK health services and policy. For example, there are long-term arrangements for commissioning systematic reviews or model-based economic evaluations to support NICE technology appraisals and policy-making in the DHSC (e.g. Policy Research Programme evidence review facilities).

The NIHR’s Health Services and Delivery Research (HSDR) programme first commissioned two evidence synthesis centres in 2013, and then three centres from 2017 to 2020 [the University of Exeter (Exeter, UK), University of Sheffield (Sheffield, UK) and University of York (York, UK)-based teams that produced this report].5 The aim of these centres has been to ‘produce evidence syntheses which will be of immediate use to the NHS in order to improve the quality, effectiveness and accessibility of the NHS including delivery of services’.5 The review topics are specified by the HSDR programme, and:

. . . will be areas of importance to the service, where there is a reasonable level of published evidence but these may be dispersed, with useful lessons for the NHS from other sectors, countries or a broad range of literature. The finished products are designed to summarise key evidence for busy managers and clinical leaders, while evaluating the quality of information and strength of findings. The aim is for an authoritative single-source document, which provides simple top-line messages in complex areas.

. . . The output will be an evidence synthesis – that is, a comprehensive review of published literature with explicit search strategy, appropriate range of sources and critical assessment of quality of evidence and strength of findings.

Between 2017 and 2020, our three research centres were commissioned to conduct evidence syntheses that respond to specific health policy-makers’ and service commissioners’ needs. Typically, we were tasked, at short notice, to conduct a systematic/rapid review of evidence on a service delivery/design health-care topic, following the identification of a need for evidence on that topic by a policy or commissioning lead or team within the DHSC or NHS England.

Scoping within systematic reviews

The scoping stage of a review comprises those initial processes that aim to establish or refine the review questions and determine the review’s scope, such as its area of focus, key terms and types of studies to be included. (It is potentially confusing that a scoping review is a particular type of review method and evidence end-product.) In general, the scoping stage seeks to reconcile the twin goals of asking the ‘right review question’ (to best address user needs) and making the best use of available research and other evidence. To be useful, systematic reviews frequently have to negotiate a compromise between these two goals, answering questions that are close to the user’s needs, but also being confident that evidence of adequate quantity and quality exists. This report aims to share the lessons learned from our varied experiences of managing this compromise within the NIHR HSDR programme’s remit, primarily to inform health-care commissioning and delivery in the UK.

This report's focus was initially suggested by the lead author (RA) at our annual HSDR Evidence Synthesis Centres meeting, in May 2019. After further discussion by e-mail, we chose this focus for our joint final report because we felt, collectively, that question formulation and review scoping are (1) critically important stages and processes in ensuring the quality and usefulness of responsive reviews, (2) review stages for which few explicit ‘methods’ or detailed guidance exists (see next section), and they are also (3) rarely described in journal articles or reports of reviews and systematic reviews. We also wanted to explore, for the benefit of our own teams and others that conduct policy-/service-responsive reviews, whether or not it is possible to specify ‘best practice’ principles and approaches to question formulation and review scoping on the basis of our experiences.

Established principles for scoping and developing review questions

This section summarises the guidance from established textbooks and guides on the methodology and conduct of high-quality systematic reviews in the health-care field or more broadly in the social sciences. Table 1 shows the degree of coverage of methods or principles for question formulation and scoping provided in the established textbooks and guides most used by and familiar to the members of the three review teams.

Although most methodological guidance on the conduct of systematic reviews offer coverage and consideration of how to develop review questions and agree the scope of a review, others start with the review question as already given. For example, both the Centre for Reviews and Dissemination guidance9 on conducting systematic reviews in health care and the Cochrane handbook by Higgens et al.8 give very little guidance on scoping and developing review questions, beyond the need to conduct searches to confirm that an identical or overlapping systematic review is not already published or in progress.

Petticrew and Roberts1 indicate the importance of asking the ‘right review question’ when deciding whether or not a new systematic review would be appropriate and useful. The authors also list situations when conducting a systematic review may not be appropriate (Box 1).

• High-quality systematic reviews already exist on the same topic.
• A systematic review on the same topic is already being conducted.
• The review question is too vague/broad.
• The review question is too narrowly scoped and, therefore, unlikely to be useful.
• There are insufficient resources to conduct a reliable systematic review.

More positively, Petticrew and Roberts1 describe the importance of finding and using previous systematic reviews and, if resources allow, conducting scoping searches to see what sorts of studies and what number of primary studies exist in relation to a potential review question. They go on to assert that policy questions may often be quite broad and that work is usually required to decide which questions it would be most useful to answer. The authors imply that this often involves ‘working back’ from the types of available evidence and the disciplines that have produced them.1

Advice offered by these texts often also includes breaking down the question into common components. Typically, for effectiveness review questions, these are its PICOS [population (or patient type), intervention, comparator, outcomes, study types] components.2,3,7 This exploration and breaking down is also framed as developing a conceptual framework for a systematic review.3 To give direction and corroboration to these decisions, several authors strongly suggest setting review questions jointly with the intended users of the review and also involving them in developing the review protocol.1,3 However, beyond such general advice, the texts offer limited specific, practical guidance on the principles or process of negotiating a path to the ‘right’ review question and inclusion criteria. Our experience as systematic reviewers within our teams also underlines that we have never been ‘taught’ or learned ‘formal methods’ for conducting these important stages of reviews, but have mainly learned through experience of conducting many commissioned and responsive reviews and engaging directly with the intended review users/commissioners.

An initial model for understanding the scoping process

To help guide our descriptions, discussions and reflections on scoping processes, we devised a simple model to depict the interaction of the stakeholder and other main considerations that determine review questions and review scope (Figure 1). Essentially, this combines the drivers associated with rigour (i.e. technical quality) and relevance (to policy-maker and user needs), alongside logistic concerns relating to deliverability (i.e. time/resources) and empirical considerations (i.e. amount and quality of relevant evidence). Acknowledging that quality extends beyond technical quality to include the perceptions of the policy-makers and users of the utility of each review product, the model can be seen to partly embody a specific application of the reconciliation of time, quality and money (as previously identified in Booth et al.7).

FIGURE 1 Basic model of key factors in systematic review scoping. (Figure not reproduced here; see the original article.)

Aim of this report

The aims of this work and report were to:

• describe six or more varied examples of recent scoping processes that were required to shape and specify responsive systematic reviews conducted by the three HSDR evidence synthesis centres
• provide reflective commentary on the different choices, sources of information and advice that shaped the review question(s) and scope for each review
• provide an overarching summary and set of principles or lessons for effective scoping of rapid and responsive systematic reviews.

We intended that these principles would apply to reviews when they are commissioned at short notice and with limited time for completion, and to meet specific health-care policy/commissioning needs. We sought to compare these principles and lessons with current authoritative advice on developing review questions and the scoping stage of systematic reviews.

Discussion of these aims revealed a shared interest in the need to reconcile, on the one hand, technical or ‘data-driven’ aspects of scoping the evidence for potentially answerable questions and, on the other hand, the collective learning processes required to develop an understanding of a new topic, and to build relationships with stakeholders and potential users of the review. As one co-author framed it, we sought to cover:

. . . technical issues of scoping (e.g. scoping searches and preliminary desk research) and the softer issues of consultation as they specifically relate to topic identification and scoping and could include our relationships with variously the generator of the initial topics (as appropriate), the HSDR team, patient/public representatives and those delivering services.

At the same time, we were mindful that the previous report and paper, which was based on these centres’ work (from 2014 to 2017), had mainly reflected on the role of stakeholder engagement in such responsive and service-/policy-oriented reviews.11 The main findings from that previous review of the HSDR centres’ earlier review projects are shown in Box 2.

Rapid production of high-quality outputs is facilitated by initial evidence mapping and topic scoping.

Barriers to prioritising the topic and defining the scope were: review team knowledge of the wider NHS/policy context; ability to define a scope that was both relevant and manageable.

Staying on track with the review was facilitated by: the ability in the team to deal with unexpected findings or problems; the commitment of individuals to support the project (especially external stakeholders).

Responsive working with stakeholders was also facilitated by: producing and disseminating appropriate outputs; the timeliness and topicality of outputs; producing or capturing evidence of impact.

Involvement of stakeholders at key stages maximises value and potential for impact but the impact of evidence on decision-making remains poorly documented.

Responsive evidence synthesis programmes should seek the optimum balance between decision-makers’ needs for rapid and efficient evidence synthesis and the time and resource requirements of rigorous systematic reviews.

We aimed for our new collective methodological reflection to complement rather than duplicate that work. We believed that new and valuable insights might emerge from a systematic focus on the processes of topic scoping and review question formulation, which are rarely described in academic outputs of systematic reviews.

For the purposes of this reflective exercise, the scoping processes of a review were specified as those that occurred between the time of first notification of a new review topic and the final agreement of the review protocol (with the NIHR HSDR programme and policy customer), including final review questions, inclusion criteria and the type of planned evidence synthesis.