Economic evaluation

Quality appraisal of selected studies

Consistent with the methods for an EVA, no formal quality appraisal of included studies was undertaken.

Relevant economic models

Four papers relevant to the topic area were identified and used as a basis for this assessment, specifically providing information on the payoffs [cost and quality-adjusted life-years (QALYs) accrued] from the diagnostic outcomes of true and FP and negative. An overview of these publications is provided in Table 17.

• Rua35 conducted an assessment for the wrist, evaluating the cost-effectiveness of immediate magnetic resonance imaging (MRI) in managing patients with suspected scaphoid fractures. The clinical outcomes were sourced from the SMaRT trial, which recruited participants with negative initial radiograph findings at the ED of a central hospital in London.
• Nwankwo34 calculated the incremental cost-effectiveness of a removable brace versus plaster cast in the management of adult patients with ankle fractures. The outcomes were derived from the Ankle Injury Rehabilitation trial, a UK-based pragmatic multicentre RCT. Eligible patients presented to the hospital with an ankle fracture, whether treated operatively or non-operatively, for which the clinician deemed a cast a reasonable management option.
• Low32 analysed the utilities associated with hip fractures, comparing the costs and QALYs of different imaging strategies for diagnosing occult hip fractures. The study compared MRI with dual-energy computed tomography (DECT), single-energy computed tomography (SECT) supplemented with DECT, and SECT alone.
• Judge33 evaluated the cost-effectiveness of three models of secondary fracture prevention for all patients with hip fractures admitted to NHS hospitals.

Further details on how the data were used are described from Foot and ankle to Hip.

Economic model

The EAG developed a de novo model to explore the potential cost-effectiveness of AI-assisted diagnosis of fracture compared with unassisted diagnosis of fracture in an urgent care setting from the perspective of the NHS and Personal Social Services (PSS).

Given the heterogeneous nature of the subsequent costs and consequences of different fracture locations, the EAG divided the analysis into three separate decision problems, focusing on diagnosis and treatment of (1) wrist and hand fractures, (2) ankle and foot and (3) hip. These three fracture sites were chosen as being both where the EAG considered there to be opportunity for greatest benefit from AI-assisted diagnosis and where the costs and consequences of the fractures differed substantially, warranting separate modelling. The outputs of these were then weighted based on the case mix of a typical urgent care setting to estimate the overall change in costs and QALYs accrued within that setting attributable to AI-assisted diagnosis. Scenario analyses explored additional adjustment for use of AI in diagnosing fractures other than the three types explicitly modelled.

The model described below represents a very rapid overview of the likely costs and consequences associated with use of several commercial AI algorithms to assist in the diagnosis of fracture, and unassisted diagnosis, in an urgent care setting. The purpose of this analysis approach was to explore whether there was a prima facie plausible case for any of the technologies to represent value for money for the NHS/taxpayer within the context of a NICE EVA. The results should therefore not be used as a definitive estimate of the costs, effects and cost-effectiveness of the interventions: a more thorough analysis may lead to different conclusions.

Model structure

The model structure (Figure 1) comprised a decision tree incorporating the prevalence, sensitivity and specificity and cost per diagnosis of a strategy (i.e. AI-assisted or unassisted diagnosis). The model structure allowed up to two reviews of a radiograph, although the model could be simplified to just a single review (e.g. where input data either represent just a single view, or where accuracy data were presented for a double-reading process alone, without disaggregation by reader). Allowing for two reviews generated eight terminal nodes, two each for four possible states of the world: TP, TN, FP and FN. Payoffs in terms of costs and QALYs were attached to these, drawing on previous relevant studies in the literature. The prevalence in this case was defined as the proportion of patients arriving in the urgent care setting with a suspected fracture who have a fracture. Payoffs attached to each terminal node are described in Foot and ankle to Hip. Details on how the three models were combined to estimate the overall impact on a ‘typical’ urgent care setting are in Overall impact of AI-assisted diagnosis in an urgent care setting.

Model assumptions: External Assessment Group base-case and scenario analyses

The EAG base case assumed that patients present in an ED with a suspected fracture, which can be one of three or four types: ankle or foot, wrist or hand, hip or ‘other’. The base case assumed that AI-assisted diagnosis was limited to ankle/foot, wrist/hand and hip fractures alone and so only considered these fracture types. The typical patient was assumed adult, and the model drew on data sources from adult populations (all 18+ with the exception of Rua et al.35 (wrist fractures) for which the minimum age for entry was 16). A scenario analysis (scenario 9) assumed that the software was used to read radiographs for all suspected fractures (including ‘other’).

As the cost of the ED attendance and X-ray (including the grade of staff reading the X-ray) was common to all comparators in this analysis, these costs were excluded. The difference in cost between different technologies was therefore limited to the cost per scan. This also meant that the cost of reading the scans were directly transferrable to other urgent care settings (i.e. UTC and MIU). Sensitivity and specificity of the diagnosis was assumed to depend on the grade of staff reading the scan, rather than the setting where it took place.

The decision problem addressed in the EAG base case was to explore the cost-effectiveness of AI-assisted diagnosis using four technologies (BoneView, Rayvolve, RBfracture and TechCare Alert) and unassisted diagnosis of fracture in the urgent care setting from the perspective of the NHS and PSS.

Sensitivity and specificity of diagnosis

Two of the studies identified in the literature review were selected as sources for prevalence and sensitivity and specificity of each diagnostic strategy. While all studies had strengths and limitations, and a number of potentially relevant studies for the decision model were identified in Study design and diagnostic tests, Bousson13 provided directly comparative data between BoneView, Rayvolve and TechCare Alert. Furthermore, data were disaggregated by foot, ankle and hand (but not hip). For the base case, the EAG estimated a mean sensitivity and specificity for foot and ankle, assumed hand applied equally to wrist, and assumed the sensitivity and specificity of hip fracture diagnosis was equal to that for ‘all fractures’. Bachmann23 compared RBfracture assisted diagnosis to unassisted diagnosis in a wide range of fracture types, reporting results by ‘mixed’ staff types, ED trainees and trauma care nurses. This was used as the source study for RBfracture and unassisted diagnosis, with diagnosis by ‘A&E trainees’ considered the closest match to that in Bousson (‘radiology residents’ with four years’ experience). Due to concerns about quality limitations in the studies, the EAG did not seek to adjust estimates for differences in the characteristics of these studies, and a full network meta-analysis of source studies is recommended in future work. Instead, the EAG explored optimistic and pessimistic scenarios in the analysis (see Scenario analyses). As data were not disaggregated by fracture type, the base case assumed the same sensitivity and specificity for all fracture types for RBfracture and unassisted diagnosis.

Source studies estimating the sensitivity and specificity of diagnosis assumed a single read of a scan. To ensure as fair a comparison between technologies, the EAG base case assumed only a single read. A scenario analysis including a second read of all scans was included as the EAG considered that this was a more realistic approach for clinical practice. Base-case input data (including parametric distributions for the probabilistic analysis) are summarised in Table 18.

Foot and ankle

Based on the WHO Global Burden of Disease 2019 report,36–38 Chen et al.39 estimated that in the UK in 2019, the age standardised incidence of foot fracture was 202.64 per 100,000 [95% uncertainty interval 142.23 to 278.04]. Globally, foot fracture incidence exhibits a bimodal distribution by age, with peaks among the very elderly (80+) in both men and women and at around age 20–24 in males and 10–14 in females. Overall incidence is higher in men than in women, although is similar among the very elderly.

Two previous economic studies were identified as potential sources for data for this analysis. Nwankwo et al.34 reported the results of a within-trial cost–utility analysis comparing removable brace with cast in patients with ankle fractures aged 18+ over 12 months, while Baji et al.40 reported the results of a within-trial economic analysis over 12 weeks. Health state utilities measured in Nwankwo et al. at six time points showed continuing improvement in both arms, suggesting healing continued over this time horizon. In the absence of data to the contrary, the foot and ankle fracture model assumed a time horizon of 12 months (i.e. implicitly assuming that there will be no difference in cost and outcomes between patients with and without fractures after this point). The Baji40 study was not considered further as a source for extracting payoffs due to its short time horizon.

In this analysis, patients were assumed to present to urgent care with a suspected fracture following a trauma. The two states of the world were for the ankle or foot to be broken, or for it to have sustained soft tissue injury alone. In the case of a soft tissue injury, the patient was assumed to remain in pain for a short while (e.g. 2 weeks), before returning to normal health.

The time horizon of the source study was one year. Payoffs in terms of costs and QALYs for the four possible outcomes are described below.

True positives

Nwankwo34 reported health state utilities for patients experiencing ankle fractures over 1 year using the EuroQol-5 Dimensions, five-level version instrument, yielding 0.723 QALYs for patients with brace and 0.720 for those treated with cast. The EAG assumed that 50% of patients would be treated with a brace and 50% with a cast. As allocation among the trial was almost exactly 50/50, the weighted average QALYs accrued was 0.722 [standard error (SE) 0.06], which was assumed to be the QALYs associated with a true positive diagnosis of a foot or ankle fracture (Table 19).

National Health Service and PSS costs over 12 months were estimated at £995.54 (SE £130.68) in the brace arm and £717.47 (SE £47.70) in the cast arm (2019/20 prices). This included ED visits, in-patient days, medication, community health services, outpatient visits, aids and adaptations to the home and PSS (as described in Nwankwo supplementary material34). Following confirmation with the author, the ED visits measured in the study did not include the index attendance (i.e. initial presentation at ED with a suspected fracture). As per the assumptions for calculation of QALYs, the EAG assumed a 50/50 split between the two approaches and adjusted the price year with the NHS Cost Inflation Index (NHSCII) pay and prices index yielding a mean cost of £1837.23 (SE £71.03) in 2022–3 prices (Table 20).

True negatives

A patient who is correctly diagnosed as not having fractured their ankle or foot was assumed to incur the cost of the index ED consultation and then be discharged (the index consultation is excluded from the analysis as is common to all arms).

To estimate QALYs, patients were assumed to endure the health state equivalent to a fracture for 2 weeks (representing pain and soft tissue bruising from the injury causing them to seek assistance), before reverting to the ‘healed’ health state utility for the remainder of the year. This led to a mean QALYs accrued over 1 year of 0.796 (Table 21). Utilities and durations were entered in the model with QALYs calculated dynamically from sampled values of both. Due to lack of data, correlation between utility at the time points was assumed zero (i.e. independent). This will overestimate the distribution of sampled values of QALYs (i.e. overestimate uncertainty in QALYs). The utility associated with ankle/foot fracture (0.225) was considered to lack face validity for a soft tissue injury. Therefore, scenario analysis explored alternative assumptions.

False positives

A FP was assumed to accrue the same cost as a TP patient and to accrue the same QALYs as a TN.

False negatives

Patients with a fracture who were wrongly diagnosed as not having a fracture were assumed to re-present in the ED after 2 weeks, whereupon additional investigations would be conducted, and a correct diagnosis made. Clinical advice to the EAG was that a patient would either present to their general practitioner or re-present in the ED. A plausible time frame was considered to be within a month. They therefore incur the same costs as TPs, plus a cost for additional examinations. To calculate QALYs, the patient was assumed to remain in the base health state for 2 weeks before following the utility trajectory of a TP patient, trimmed to 52 weeks.

Summary of key assumptions

• Ankle fractures would be healed within 12 months.
• TN incur the cost of an ED attendance.
• TN accrue QALYs based on 2 week’s baseline utility from Nwankwo et al. followed by the remainder of the year at the ‘healed’ utility from Nwankwo et al.
• FP incur the same cost as TP, same QALYs as TN (2 weeks of pain then back to normal).

Hand and wrist

Six previous economic evaluations23,35,41–44 of interventions for hand and/or wrist fractures were identified, of which two were UK-based and therefore of most relevance to this study question.35,44 Both studies reported the results of the same RCT of a second line diagnostic test of immediate MRI versus no MRI in patients with a suspected scaphoid fracture who had a negative X-ray (i.e. the sum of TNs and FNss from the initial X-ray). The full cost–utility analysis35 was used as the primary source for this analysis.

As Rua et al.35 compared immediate MRI with no MRI (control), the results were presented in an aggregate manner, without distinguishing between FNs and TNs. Consequently, the EAG drew on the control arm data to populate the costs and consequences of all four terminal nodes.

Utilities were measured at four time points to a time horizon of 6 months. Resource use comprised primary and secondary care contacts, including fracture clinic appointments, subsequent ED visits, additional diagnostics, surgery, physiotherapy, splits and plaster casts.

The time horizon for the source study was 6 months. Payoffs in terms of costs and QALYs for the four possible outcomes are described below.

True positives

The EAG assumed utilities for TP hand/wrist fractures were one SD lower for 3 months, compared to those reported in Rua et al.35 This assumption was required as the aggregated results encompassed patients both with and without fractures (Table 22). SEs were assumed as per those reported in Rua et al.35

Costs for the TP population are shown in Table 23.

True negatives

The utilities for TNs were assumed to be equal to those reported in Rua et al. 2020,35 as 89.6% of patients in the control arm confirmed the absence of fractures. The EAG noted that this resulted in a moderate underestimate of QALYs accrued due inclusion of 10.4% of fractures (Table 24).

Similarly to the utilities, the costs derived from the publication are assumed to align with a TN population (Table 25).

False positives

As for TNs, the utilities for FPs were assumed to be consistent with those reported in the publication, given 89.6% of patients in the control arm confirmed the absence of fractures (Table 26). This may overestimate utility gains as it may be the case that treating FPs as positives may result in disutilities.

The costs associated with TNs and FPs differ, as the latter group is treated similarly to those with fractures until a subsequent diagnostic test confirms otherwise (Table 27). Therefore, the base-case resource utilisation for this population is assumed to be 10% higher than that of the publication.

False negatives

Regarding quality of life (QoL), this population closely resembled that of TPs. However, it was assumed that patients in this group would be mistreated as negatives for the first 2 weeks following their presentation to urgent care, and therefore would not accrue QoL gains during this period. Furthermore, as observed in the true positive population, it was assumed that the utilities were lower by one SD for up to 3 months compared to those reported in the publication. Finally, it was assumed that the QoL for patients after 3.5 months of treatment aligned with the values reported in the publication (Table 28).

Differences in resource use from the TP population include an additional 5% of patients likely to undergo wrist surgery, as suggested by Rua,35 and the inclusion of an extra visit to the ED (Table 29).

Summary of key assumptions

• All health states:
  • QoL after 3 months aligned with that of the publication as patients can be considered ‘cured’.
  • 6 months is a long-enough time horizon to capture all QoL and costs differences.
  • The utilities and resource use from Rua et al.35 are representative of those of TNs in the UK.
• TPs:
  • Utilities for the first 3 months are equal to one SD below those reported in Rua et al.35
  • Resource use data.
• FPs:
  • The utilities of Rua et al.35 are representative of this population.
  • Resource use associated to this health state is 10% higher than that of TNs.
• FNs:
  • Patients return to urgent care 2 weeks after the initial presentation following which their fracture is correctly diagnosed.
  • Following from the above, the utility from Rua et al.35 at baseline represent the utility for those 2 weeks, therefore not assuming disutilities in that period.

Hip

Six economic evaluations for hip fractures were identified. One study was UK based (Judge et al.33) and provided the lifetime costs of hip surgeries for usual care (UC), fracture liaison nurse (FLN) care and orthogeriatrician (OG) care models for delivery of secondary fracture prevention after index hip fracture. Other studies were conducted across different geographies (EU/Singapore/America) of which Low et al.,32 a Singapore based decision model comparing different diagnostic strategies for occult hip fractures provided utilities for hip fracture by age group and different time periods following the fracture (i.e. immediate, during first year and after first year following fracture) was used, as the study design was aligned with the decision problem scope1 and included a detailed model inputs table (along with the associated uncertainty parameters).

The time horizon for the source study was lifetime. Payoffs in terms of costs and utilities for the four possible diagnostic outcomes are described below.

True positives

Low et al.32 reported utilities for hip fracture for 65- to 74-year-olds and also at different time points following the index hip fracture (immediately following fracture, during first year following index hip fracture and beyond first year). Table 30 shows the mean utilities and uncertainty distributions inserted into the model based on Low et al.

As the Judge et al.33 study was UK-based, the EAG considered it to be the preferred source for resource use and costs. This study comprised a lifetime Markov model with 1-year cycles that simulated the natural history of hip fractures, including progression, major non-hip fractures, and discharge to home or a care facility. The study further provided the intervention, hospital, primary care and care home costs by male and female cohorts and by different care models namely UC, FLN and OG care.

Table 31 provides the mean discounted costs which is an average across all models of care with the assumption that patients were equally distributed across those models of care (namely UC, FLN and OG). This approach was taken as the population concerned was over 60 years typically having fragility related fractures, where FLN and OG care models could also be useful in addition to UC. SE was not available; therefore, it was calculated from 95% CIs (upper and lower limits) provided (i.e. SE = upper limit – lower limit/3.92).

True negatives

A patient who was correctly diagnosed as not having fractured their hip was assumed to incur the cost of an emergency room consultation and then be discharged. They were assumed to be experiencing health state equivalent to a post-fracture immediately for 2 weeks (to account for the pain and short-term impact of injury), before reverting to the baseline health utility (0.79 as shown in Table 32). Utilities and durations were used in the model to calculate QALYs dynamically from sampled values of both (Table 32). Due to lack of data, the utility values at different time points were assumed to be independent.

False positives

As per Low et al. 2021,32 people with false-positive results would be treated as having a hip fracture and would undergo surgery similar to TPs. Therefore, their costs would be the same as those of TPs. However, as they do not actually have the fracture, their utilities would be assumed to be the same as TNs. The EAG noted that FPs were less likely to undergo surgery as the incorrect diagnosis was likely to be spotted before this point. The cost estimate for this analysis may therefore be an overestimate.

False negatives

Low et al.32 mentioned that people with FN results would be discharged only to return to ED within a month for hip surgery, following which they would have the same pathway as TPs. Utilities used for FN hip fractures are shown in Table 33.

In terms of costs, the same costs as TPs were incurred, plus costs for additional ED attendance and further imaging [computed tomography (CT) was assumed], as given in Table 34.

Summary of key assumptions

• Long-term costs and consequences for hip fractures were assumed for lifetime horizon (one index surgery followed by a second surgery assumed) as per Low et al.32
• FNs were assumed to incur the same costs as TPs with additional costs of an ED attendance and any additional investigations (1 CT assumed). Time for return after discharge was assumed to be 1 month or 4 weeks for FNs based on Low et al.32
• TNs incurred the cost of an ED attendance and accrue QALYs based on immediate post-fracture utility applied for initial 2 weeks, following which they were assumed to return to baseline utility for 65–74 years.
• FPs incurred the same cost as TPs as per Low et al.32 and same QALYs as TNs.

Overall impact of artificial intelligence-assisted diagnosis in an urgent care setting

Clinical advice to the EAG was that over 2022–3, there were approximately 25.3 million ED attendances in England and that fractures typically account for 2–4% of all visits, equating to between 506,000 and 1,012,000 attendances. Clinical advice suggested that around 12.5% of all fractures are ankle, 7.5% wrist and 12.5% are hip. Data specifically including ankle and foot, and wrist and hand, were not available. However, the EAG assumed that these proportions represented the base-case distribution to estimate the overall impact of diagnosis in an urgent care setting (Table 35). Scenario analysis included the use of the technology for all fractures.

These figures were multiplied by the number of patients per year expected in a ‘typical’ ED with 350–400 attendances per day (total attendances, not just fracture). This equated to 136,875 attendances per annum, of which 4106 would be for fracture. Including just the base-case fracture types, this equated to 1334 scans per year.

The EAG base case assumed a weighted average cost and outcomes by fracture type. This was scaled up to the number of attendances per year to estimate the overall difference in cost and QALYs accrued to patients attending the ‘typical’ ED. The three fracture subtypes had different time horizons. By merging the three together the assumption was that costs and QALYs accrued after 6 months for wrist/hand and 12 months for ankle/foot were identical across all arms, in other words complete healing has taken place by this time. If any differences remained after these time horizons, then the analysis may underestimate the cost-effectiveness of AI-assisted diagnosis [i.e. overestimate the incremental cost-effectiveness, underestimate the incremental net health benefit (INHB)].

Cost of diagnosis

Cost of scans

Cost per scan was extracted from company RFIs where reported. Some companies operated volume-based pricing. The EAG base case used a cost per scan based on 1334 scans per annum. Base-case prices are reported in Table 36 and minimum and maximum prices are explored in scenario analyses 3 and 4. Where no pricing data were supplied, the EAG inserted a notional cost per scan. The maximum economically justified price per scan for each software compared with unassisted diagnosis was also calculated.

Cost of accident and emergency attendance

As the index presentation at A&E, and the X-ray, was common to all arms, the costs of these were excluded from the analysis. However, some cases (e.g. FNs) were assumed to require an additional presentation after a period of time. This was costed on a mean cost of £149.04 (NHS Reference Costs 2022–3, code VB11Z: Emergency Medicine, no investigation, no significant treatment), and varied by ± 10% in a uniform distribution in the probabilistic analysis.

Approach to analysis

As there were multiple comparators, the EAG reports the INHB of each technology compared with unassisted diagnosis at willingness-to-pay (WTP) thresholds of £20,000 and £30,000 per QALY gained. This facilitates a fully incremental analysis as the option with the highest net health benefit is the option yielding an incremental cost-effectiveness ratio (ICER) at or below the threshold after taking into account dominated and extended dominated options; incremental comparisons between each technology can also be made by comparing the INHB directly. However, given the level of uncertainty and the rapid and approximate nature of the modelling for this assessment, the EAG cautions against comparing each AI software against each other, instead considering a ‘class effect’ for the software as a whole.

The EAG reports the results for each fracture location individually, followed by an overall weighted average for all three fracture subtypes based on case mix. This was multiplied up to show the expected impact on cost to the NHS and QALYs accrued to patients versus unassisted for each software. Finally, the maximum economically justified price per scan was calculated as the cost per scan associated with an ICER of £20,000 per QALY and £30,000 per QALY.

Scenario analyses

The EAG planned to conduct scenario analyses on the use of the technology in different settings (e.g. ED vs. UTC). However, the EAG considered that the grade of staff reading the radiograph was a more important determinant of the diagnostic accuracy than the setting, and any differences in resource use across the settings may largely be equal between all comparators. The EAG therefore conducted an optimistic and pessimistic scenario to represent different settings as described in Scenarios 1 and 2 below. Other scenarios are described below.

Scenarios 1 and 2: optimistic and pessimistic diagnostic accuracy

The EAG base case for this EVA used a naive, unadjusted comparison of arms from two studies to inform the sensitivity and specificity of each technology. The EAG therefore explored an optimistic and pessimistic scenario, based on a review of all source studies (see Tables 11–15). The optimistic scenario assumed the lowest sensitivity and specificity for unassisted diagnosis and highest for each technology, and the pessimistic the reverse (Table 37). Where a source suggested a lower or upper bound that was inside the EAG base case, the EAG base case was used as the estimate at the bound.

Scenarios 3 and 4: cost per scan

Several companies price their software on the basis of annual volume. These scenarios explored a low volume (i.e. highest cost per scan) and high volume (lowest cost per scan) analysis for those technologies pricing based on volume (confidential information has been removed, Table 38).

Scenarios 5 and 6: reduced time to interpret radiograph

Data on reduced reading time for X-rays are reported in X-ray reading time. These studies generally reported time savings of between 7 and 13 seconds per scan assisted by the technology. For the purpose of this scenario analysis, the EAG assumed a notional 10 seconds reduction per X-ray and subtracted the cost of this from the cost of the AI-assisted strategies under two scenarios: scenario 5 assumed the radiograph was read by a junior/trainee radiologist and scenario 6 by a consultant-level radiologist (Table 39). The EAG noted advice from clinical experts that best practice use of the technology may lead to increases in reading time, though considered the lack of evidence for this, the EAG did not explore this scenario in the analysis.

Scenario 7: health state utility for negative ankle and foot fractures

The base-case analysis for an ankle or foot fracture assumed that a patient without a fracture experienced an equal health state utility to a fracture, but only for 2 weeks before resolving (i.e. assuming a soft tissue injury which healed in 2 weeks). However, the EAG considered the health state utility to be lower than was plausible for such an injury and therefore lacking in face validity. The EAG therefore explored a scenario with a utility of 0.727, equivalent to EuroQol-5 Dimensions utility for a person with some mobility problems and some pain (overall profile 21121).

Scenario 8: use of technology in all fractures

The EAG base case assumed that the technology was only used in the three types of fractures considered in the analysis. However, the EAG considered it plausible that the software would be enabled for other fracture types. The EAG therefore conducted a scenario representing an additional cost associated with those other uses but assumed that zero benefit was gained from those reads. This therefore represents a pessimistic scenario of the broader use of the technology (Table 40).

Scenario 9: second read of all scans

The EAG base case used data on diagnostic accuracy as extracted from the two source studies. These comprised a single interpretation of the X-ray, although a second review of diagnoses was considered to be more reflective of real practice. To represent this, the EAG conducted a scenario where all scans were read a second time. It was assumed that the diagnostic accuracy of the second reader was the same as the first. As this was common to all arms, the cost was excluded from analysis.

Results from the economic modelling

Base-case results

Overall, the majority of the AI-assisted diagnostic algorithms were associated with a positive INHB compared with unassisted diagnosis at £20,000 and £30,000 thresholds, although 95% credibility intervals in most cases crossed zero, both for all fracture types and when considered together (Tables 41–46).

As noted previously, due to data limitations, the EAG advises against comparisons between the individual algorithms. The minimum economically justified prices were somewhat above the proposed per-scan prices proposed by the companies. However, the EAG also cautions against use of these data to inform pricing decisions, as the modelling was only considered suitable for an indicative ‘signal’ of cost-effectiveness. More detailed analysis would be required to estimate a suitable maximum price per scan, along with fully incremental analysis to compare the benefits and costs of all the algorithms against one another.

Scenario analysis results

Results are summarised in Table 47. Scenario analyses suggested that the decision was sensitive to the optimistic and pessimistic scenarios (scenarios 1 and 2), implying there was much uncertainty in the data. However, the results suggested that there was the potential for AI-assisted diagnosis of fracture to be cost-effective. Results were broadly insensitive to the low and high pricing scenarios for Confidential information has been removed (scenarios 3 and 4), when accounting for time taken to interpret X-rays (scenarios 5 and 6), adjusting for the health state utility of negative ankle and foot fractures (scenario 7), use across all fractures rather than just the three locations analyses (scenario 8), and whether scans are read once or twice (scenario 9).

While the scenario analyses of this EVA suggested that the only parameters to which the results were sensitive was the sensitivity and specificity of the diagnostic scans, the EAG advises caution in interpretation as more detailed modelling and analysis may lead to differing conclusions.

Summary and interpretation of the economic evidence

The early modelling for this EVA suggested that, at the proposed prices charged by the companies, AI-assisted diagnosis had the potential to represent a value-for-money investment for the NHS at typical thresholds of £20,000–30,000 per QALY gained. However, this conclusion was considered highly uncertain at the present time. The EAG noted that the cost-effectiveness appeared to be driven by reductions in costs rather than a gain in QALYs, but the impact was variable across fracture types. The EAG cautions against using this analysis to compare one AI algorithm against another due to data limitations, and instead to consider whether AI-assisted diagnosis as a class would likely be of value. Future work, however, will require comparison of one algorithm against another to ensure the varying diagnostic accuracy of the algorithms is matched to their prices in a fully incremental analysis.

Scenario analysis concluded that the only parameters to which the decision was sensitive were the diagnostic accuracy of the algorithms and unassisted diagnosis, with the ‘optimistic’ and ‘pessimistic’ scenarios generating diverging conclusions. Other scenarios, including cost and time savings from AI-assisted diagnosis, low and high price-per-scan scenarios, and varying assumptions around health state utilities, use of AI in all fractures and including a second read of all X-rays did not materially affect the results.