The EAG conducted a broad evidence review to identify the available evidence base for the value of AI as assistance to identifying fractures in urgent care settings. The assessment identified an emerging though limited evidence base for the technology, with meaningful gaps in evidence to inform decision-making on the use of the technology in clinical practice. Almost all of the evidence base identified evaluated the diagnostic accuracy of the technology for identifying fractures, though these analyses were largely not specific to emergency care settings or the staff that were anticipated to typically use the technology in clinical practice. There were also significant methodological limitations in the evidence base, which increased uncertainty in the findings. Aside from X-ray reading time, there was no evidence for the impact of the technology on service outcomes, such as the use of additional imaging, hospital appointments, and patient recalls, and no evidence for the health outcomes of people with suspected fractures. Overall, there was a paucity of evidence for determining the potential value of the technology for use within NHS settings.
In consideration of the limitations of the evidence base, the EAG tentatively concluded that there was early evidence that the technology may have value for reducing the risk of missed fractures but may not improve the avoidance of FPs. Due to uncertainty in the precise estimates reported by the studies, the EAG could not determine a reliable estimate for the reduction of missed fractures that could be avoided with the technology, though noted that the technology did not eradicate missed fractures entirely. Some studies, particularly those evaluating the accuracy of the technology in fracture types that were more difficult to identify, reported high rates of missed fractures that would be unacceptable in clinical practice. The implication of these findings was that while the technology may improve identification of fractures, its use would not remove the need for existing strategies used by urgent care settings to protect patients (such as further imaging, precautionary treatments and fast turnaround times to definitive reports). As might be expected, there was less additional value of the technology when unassisted accuracy was already high, such as when used by senior staff and in fractures that were easier to diagnose. This would suggest that the technology may best be targeted towards the settings and populations where it may hold the greatest value; however, without evidence for the clinical and service outcomes that may be affected by using the technology, the EAG was unable to make this conclusion. For instance, the EAG considered it plausible that a small difference in sensitivity may nevertheless be meaningful if the additional fractures identified would have otherwise resulted in significant health or resource implications. As noted in Evidence gaps and research recommendations, further evidence is needed to explore this in order to inform decisions about if and how the technology could be used.
The current evidence base also did not allow for an understanding of how the potential evidence base might vary according to the target population. The majority of studies identified reported very few details about the study sample demographics, including the prevalence of people with frailty and health conditions that affect bone health. Very few studies reported outcomes specifically in children, where the identification of fractures can be particularly difficult. This was not notable in the results of the studies, however, where diagnostic accuracy both with and without the technology was not substantially different to results reported in adults alone. Given the limitations in the evidence base, the EAG considered that more evidence to evaluate the technology in children and across other key subpopulations would be important.
The economic analyses suggested that most of the AI-assisted diagnostic algorithms were associated with a positive INHB compared with unassisted diagnosis at NICE’s lower and upper threshold band of £20,000–30,000 per QALY. The data were of insufficient quality to enable a robust comparison of one algorithm against another in a fully incremental analysis, which would be required to ensure the varying diagnostic accuracy of the different algorithms are matched to their prices, and to encourage a competitive market for the benefit of NHS patients. The results were largely insensitive to the different scenario analyses considered, except for diagnostic accuracy of the algorithms and unassisted diagnosis.
Overall, the results of this assessment must be considered within the context of the constraints of the EVA methods. Consistent with the aims of the EVA, the EAG adopted a pragmatic approach to the identification of evidence and exploration of this within its assessment. The EAG acknowledge a number of limitations to this approach, which included the use of single reviewer screening and data extraction, the broad inclusion criteria (allowing for the inclusion of lower quality evidence), and the lack of a formal quality assessment. The EAG was also unable to explore heterogeneity in the diagnostic accuracy results in more depth: while groupings of reader experience were made in order to aid interpretation of the results across studies, the EAG considered these to be unreliable, given difficulties in interpreting staff grades and experience as reported from publications of studies conducted in other countries. Given the variation in methods used across studies and the quality limitations of the evidence, further exploration of heterogeneity or re-grouping of studies may not have been meaningful, though this is a limitation of the assessment. An exploratory economic analysis was developed, the objective of which was to establish whether there was a prima facie case for AI-assisted diagnosis of fracture to represent value for money for NHS patients (i.e. not to provide detailed guidance on and precise estimates of the cost-effectiveness of the different algorithms). It should therefore be considered an approximation placing plausible bounds on the likely costs and consequences of the algorithms and not a definitive estimate of the cost-effectiveness. A large number of gross assumptions were required to conduct the analysis within the assessment, for example, the EAG was unable to consider the longer-term costs and consequences of FNs and positives in anything but the most rudimentary manner. More detailed modelling in a full formal diagnostic assessment review is required to consider these issues and nuances, and how they are likely to impact the estimates of cost-effectiveness.
In conclusion, this assessment identified preliminary evidence that AI may have potential value for use within urgent care settings to aid with the identification of fractures. In order to inform a full evaluation of the technology, that could be used to inform decisions on routine commissioning, a significant body of further evidence is needed to establish the clinical, service and economic outcomes associated with the technology in settings relevant to urgent care within the NHS.
Patient and public involvement
Consistent with NICE methods for the evaluation of technologies within its EVAs, patients and members of the public were invited to provide comment on the scope1 for the assessment, the report and draft committee recommendations. Specialist committee members recruited by NICE also included a combination of clinicians and lay members and their contributions were discussed by the NICE committee. To inform its own assessment, the EAG consulted with three clinicians who treat patients with suspected fractures in NHS settings (all authors on this report). Consultation responses are available on the NICE website:51 in general, there was support for the conclusions and evidence generation plan reported, though consultees raised concerns about the quality of the evidence available and considerations for implementation of AI in NHS settings. During the appraisal, it was noted that the way in which clinicians engage with the technology could have an impact on its accuracy and the way it is used to inform decision-making. Furthermore, the local protocols and equipment available in NHS settings may also affect the way in which the technology is integrated into care and the associated cost and resource. For these reasons, further engagement with stakeholders will be important as part of the evidence generation plan for these technologies.
Equality, diversity and inclusion
During scoping for this assessment, NICE conducted an Equality Impact Assessment to consider how equality considerations may be considered.52 Considerations raised included the potential for some patient groups to experience fractures that may present differently and/or where diagnostic accuracy of the technology may vary, or where the implications of missed fractures may be greater. It was also considered that the accuracy of AI technologies may have limited accuracy for certain groups if training sets used to develop the algorithm are not sufficiently diverse. Equality considerations were also discussed by the NICE committee during its deliberations. Overall, due to the limitations in the evidence base for the technologies, it was not possible to obtain accurate data for how the diagnostic accuracy of the technology may vary across population subgroups. This is included in the evidence generation plan for this technology.