Some parts of this chapter have been reproduced from Chambers et al.1 © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY. Published by BMJ. This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/. The text includes minor additions and formatting changes to the original text.
Main findings
The literature search identified 29 publications describing 27 studies that met the inclusion criteria. Studies were published between 2006 and 2018 and were diverse in terms of their design and methodology. Four items of grey literature were included in view of their importance to the topic, although three were produced by the developers of specific systems and should be interpreted with that fact in mind.26,27,34 At the time of writing this report, NHS England’s evaluation of four different systems in different regions was still in progress but an informally published draft report was located online7 and we have used that in our review while acknowledging that it represents an incomplete preliminary data set.
The overall strength of the evidence base varied between outcomes (see Table 14), but in absolute terms the evidence is weak, being based largely on observational studies. A substantial component of grey literature of uncertain quality complicates the interpretation of the evidence. Interpretation of the evidence should also take into account the risks of bias in individual studies, as discussed in Chapter 3, Risk-of-bias assessment.
We found little evidence to support the hypothesis that digital and online symptom checkers are detrimental to patient safety. Although the evidence was comparatively strong relative to other outcomes assessed in the review, it falls far short of being an adequate assessment of the safety of these systems. The studies that reported on the outcome were mostly short term and involved relatively small samples. Some were limited to people with specific types of symptoms (e.g. influenza-like illness23 or respiratory infections17) and others recruited from specific population groups (e.g. students22) who were not representative of typical users of urgent-care services. This body of evidence should, therefore, be interpreted cautiously and not be extrapolated to the possible impact of a nationally available digital urgent-care service being used by millions of people annually. Anecdotal reports of informal tests circulated via social media show symptom checkers making serious errors, but we were unable to locate any research to substantiate these claims.
The evidence on patient satisfaction with digital and online systems also had some limitations, but these findings appear more likely to be generalisable. Study participants generally expressed high levels of satisfaction, albeit in uncontrolled studies. For example, in the NHS England pilot evaluation, 70–80% of users were satisfied with their experience at each of the pilot sites.7 This was based on a sample of > 1500 users, the great majority of whom were involved in the London pilot using the ‘babylon check’ app. Despite the unbalanced sample, the findings on satisfaction suggest that any national, digital urgent-care service may be well-used, particularly given the increasing reliance on digital technology in all areas of life. Satisfaction was measured in a number of ways, but generally the studies appeared to be rating usability rather than satisfaction with the advice received or the degree of reassurance provided. Failure to provide the patient with reassurance could result in them seeking further help elsewhere in the urgent and emergency care system.
The evidence from the studies included in the review suggests that digital and online systems have yet to achieve a high level of accuracy in the diagnosis of specific conditions. This finding applies both to ‘general purpose’ symptom checkers and to those limited to particular conditions. Although the evidence was classified as relatively strong, several caveats should be applied. For example, some of the included studies did not recruit representative populations and others were based on standardised vignettes rather than real-world data. Finally, those studies that compared symptom checkers with health professionals tended to use the doctors’ clinical diagnosis as the reference standard, which would bias the comparison in favour of the health professionals. There was an indication from one study that the symptom checker performed better when additional information was available.29
The accuracy of signposting patients to the most appropriate level of service is closely related to diagnostic accuracy but results for this outcome were inconsistent between studies. Studies researching the accuracy of signposting used similar methods to those evaluating diagnostic accuracy. A review of 23 symptom checkers found that triage level was appropriate overall for 57% of patient vignettes, with considerable variation between systems and conditions.6 The NHS England evaluation also found differing levels of agreement with clinical expert opinion across the four systems evaluated.7 In general, algorithm-based triage tended to be more risk averse than that of health professionals, with 85% of respondents being advised to visit their doctor in one study.21 Although there is considerable uncertainty about the magnitude of the effect, a national, digital urgent-care service could result in considerable numbers of patients receiving inappropriate advice to visit the ED or request an urgent general practice appointment. Middleton et al.27 claimed that the ‘babylon check’ app had a high degree of triage accuracy for vignettes when compared with health professionals, but this report was not peer reviewed and so requires further validation.
We also found inconsistent evidence on effects on service use, but there was some indication that symptom checkers can influence the pattern of service use. The strongest evidence comes from a randomised trial of an intervention that was specifically designed to promote self-care and covered respiratory symptoms only.17 In this study, the intervention group had less contact with doctors (but more contact with NHS Direct) than the control participants despite having a longer duration of illness and illness of a greater severity. It is uncertain whether or not these results are generalisable to systems covering the full range of urgent care. However, the NHS England evaluation found a small shift towards self-care with digital triage compared with telephone triage,7 and Madan34 reported that the webGP symptom checkers diverted 18% of patients from requesting a general practice appointment in that consultation. These are findings from the grey literature that have not been peer reviewed and should, therefore, be interpreted with caution.
There is very limited evidence on patients’ reactions to online triage advice and whether they follow the advice or seek further help or information. This would have implications for service use. Preliminary findings from the NHS England evaluation7 suggest that patients may be more likely to seek further advice for more urgent conditions, but further confirmation is required.
Over half of the included studies considered equity and inclusion issues either directly or by comparing users with non-users of digital triage systems. Not surprisingly, studies revealed a clear consensus that younger and more highly educated people are more likely to use these services, whereas older and less educated people are more likely to prefer telephone or face-to-face contact. This could have implications for health equity if urgent-care pathways prioritise (or appear to prioritise) requests originating from digital sources. In primary care, issues have arisen because patients using e-consultation systems to request an appointment following online triage may be seen more quickly than those contacting the practice by telephone.
In summary, the key findings of this systematic review are that considerable uncertainties surround the diagnostic and triage accuracies of digital and online symptom checkers, their overall effects on service use and whether patients are likely to follow their advice or seek further assistance from other sources. Surveys of patients who have used them report high levels of satisfaction with their usability. Increasing reliance on digital and online services generally suggests that a national digital 111 service may be heavily used and its impact on the health-care system, in whatever direction, could be substantial. This review found limited evidence to suggest whether the new service will stimulate or manage demand for urgent and emergency care.
Strengths and limitations
This systematic review has been undertaken to a timescale determined by the review commissioners using a relatively large team of experienced researchers, including both methodological experts and topic experts. In other words, we have sought to accelerate the systematic review process rather than abbreviate it by omitting some processes. The main exception to this was the use of one reviewer for some processes (with checking of a 10% sample carried out by a second reviewer) rather than full, independent duplication. The evaluation of the screening process suggested that there was only a moderate degree of agreement between reviewers. This reflects a degree of learning by the review team during the screening process. Our initial sift of the search results consciously favoured inclusivity, and items that were found not to meet the inclusion criteria on detailed examination were subsequently discarded. We performed a rigorous search of the literature, including reference checking and citation searching. Rather than a conventional, highly sensitive search (which would have resulted in inefficiencies in the screening process), we combined an initial focused search with subsequent rounds of follow-up searching, including searches for named symptom checker systems. Overall, we consider it unlikely that any significant research in this area was overlooked.
We assessed risk of bias in individual studies using a variety of appropriate checklists, as well as summarising the overall strength of evidence for key outcomes (see Table 14). The protocol was developed in association with NHS England and the NIHR HSDR programme team and was registered prospectively on PROSPERO.
The heterogeneous and descriptive nature of the included studies meant that meta-analysis was not feasible for any of the outcomes of interest. Our narrative synthesis approach used a mixture of description and tabulation to summarise the evidence for each of the prespecified outcomes of interest. In addition, we considered the different symptom checker systems as interventions and extracted data on them using the system rather than the study as the unit of analysis (see p. 23 and Appendix 3); we also used an adaptation of an established intervention-reporting checklist to assess them.8 This process highlighted the lack of information available for some systems and it could be developed to form a template for ongoing evaluations of current and new systems.
This was a review of published, including non-peer-reviewed, literature and the coverage of systems is not exhaustive (e.g. we did not extract data from websites). We also did not carry out any original analyses of raw data even when such data were available. The timing of the review meant that the final results of NHS England’s pilot evaluation were not available to us. We were able to make use of a draft report that was published online,7 but we acknowledge that the findings of the final evaluation report, when available, will supersede those of the 2017 draft.
The review inclusion criteria were relatively broad and findings from symptom checker systems for specific conditions may not be applicable to more general systems and vice versa. We have also included studies of symptom checkers as part of e-consultation systems in general practice, which represent a slightly different setting from a general digital 111 service, and this should be kept in mind when interpreting the results.
Implications for service delivery
The implications of this systematic review for service delivery should be considered in the context that a decision has already been taken to introduce a digital 111 service and that the implementation of the service is in progress. The findings can be used to support the ongoing development and evaluation of the service.
Achieving a high level of diagnostic accuracy will be key to the success of a digital 111 service. Failure to provide an accurate diagnosis may result in outcomes such as patient dissatisfaction and unwillingness to use the service again, increased use of other urgent and emergency care services and possible risks to patient safety (although the cautious approach characteristic of most existing systems that we identified may help to mitigate this).
The studies included in the review suggest a high level of uncertainty about the impact of digital 111 on the urgent-care system and the wider health-care system. Some of these uncertainties can be addressed by research and data collection, as discussed below, but the health service may need to respond to short-term increases (or decreases) in demand and/or shifts from one part of the system to another. This may increase pressure on the system, at least in the short term. In the longer term, if the use of the NHS111 telephone service decreases as planned, there may be opportunities to re-deploy staff to fill other roles within the urgent and emergency care system.
Research on the introduction of the NHS111 telephone service from 2010 was outside the scope of this review. However, the telephone service followed a similar pattern of piloting followed by nationwide implementation to that proposed for digital 111. In brief, the evaluation of the pilot sites (compared with control sites where NHS111 was not available) found that NHS111 did not reduce calls to ambulance services,5 did not improve public perceptions of urgent care provision41 and identified key groups that were less likely than others to use the service (including older people and men).42 However, the majority of respondents in the study of acceptability were very satisfied (73%) or quite satisfied (19%) with the service overall.42 A later study of five NHS111 service providers covering 2014–16 also found high levels of patient satisfaction, but views of other stakeholders were more mixed.43 Decision-makers may be able to use the lessons learnt from the introduction of the NHS111 telephone service alongside the findings of this review to inform their choices around the implementation of digital 111. For example, some respondents had concerns about the relevance of the questions they were asked,42 emphasising the importance of flexibility in the underlying algorithm to prevent users giving up in frustration and seeking help elsewhere in the urgent and emergency care system.
Implications for research
The limitations of the evidence base suggest that further rigorous research is urgently needed. We have identified the following priorities for research (in addition to the ongoing collection of data to monitor the use and safety of the digital 111 service):
- Our research suggests that there is a wide variation in performance between symptom checker systems,6 but we found no studies designed to compare systems directly. Such research will be important for commissioners to make an informed choice on how to implement services at a local level.
- Evidence on costs or cost-effectiveness was classified as very limited; hence, rigorous economic evaluations based on real-world data are needed. A key question for research is whether or not new digital services will reduce overall costs for the health service and whether or not savings are made in one part of the system at the expense of another.
- The introduction of a digital 111 service provides a new point of access for patients needing urgent medical help. Research is needed to investigate the pathways followed by patients using the service and, particularly, if the overall number and level of contacts with the health system can be reduced without affecting the quality and safety of patient care.
- This review located only one study of a service designed to assist parents and other caregivers to assess the acuity of a child’s illness.23 Research is required to ensure that the systems underlying national digital 111 services are suitable for children’s illnesses as well as those of adults, or to evaluate a separate ‘paediatric digital 111’ service.
- Given the example of the Internet Doctor intervention, with a design based on a specific psychological theory,17,25 the role of behaviour change theory in the development and implementation of symptom checkers should be investigated.
Conclusions
The current evidence base covers a diverse range of interventions, study designs and outcomes. The evidence is generally weak as it is based largely on observational studies and includes some studies that have not been independently peer reviewed. There are major uncertainties around the probable impact of digital 111 services on most of the important outcomes, but evidence on patient satisfaction, together with developments outside the health field, suggests that once introduced the use of these services may increase rapidly. It will be important to monitor and evaluate the services using all available data sources and by commissioning high-quality research.