Assessment of clinical effectiveness

Some text in this chapter has been reproduced with permission from Lewin et al.21 and Faric et al. 2024.27 These are Open Access articles distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Method for reviewing effectiveness

Identification of studies

The current review was conducted by the BMJ Technology Assessment Group (BMJ-TAG), in accordance with the UK NSC’s evidence review process. A search strategy (see Appendix 1, Table 16) was developed by one reviewer (VW) with experience in systematic database searching. This was based on the search performed for Question 4: ‘What are the social and ethical implications of implementing AI-based tools in screening programmes and would it be acceptable to health professionals and the public?’ of the UK NSC 2021 evidence map.9 In line with the evidence map searches, systematic searches of Ovid MEDLINE (see Appendix 1, Table 17), EMBASE (see Appendix 1, Table 18), PsycInfo^® (American Psychological Association, Washington, DC, USA) (see Appendix 1, Table 19) and Cumulative Index to Nursing and Allied Health Literature (CINAHL) (see Appendix 1, Table 20) were performed to identify primary qualitative studies relevant to the evaluation of the ethical implications of AI-based technologies in medical image classification systems in screening for medical conditions. Review and opinion papers (summarising the views and opinions of the authors, clinicians or professional bodies or reviewing literature non-systematically without undertaking clinical research) were not included, as the current research aimed to determine ethical issues that the use of AI-based technologies in screening interventions may raise, emerging from the qualitative views and opinions of health professionals, patients and the general public.

The search terms included combinations of free text and subject headings grouped into the following categories:

• Intervention: diagnosis, early diagnosis, computer assisted, diagnostic test, screening, imaging, AI, automation, machine learning, deep learning, neural network.
• Outcomes: attitudes, perception, accept, barriers, appropriate, experience, ethic, social.
• Study design: qualitative, interview, survey, questionnaire.

The searches conducted for Question 4 of the UK NSC 2021 evidence map were updated. Thus, searches were limited to the period from the date of the previous search (30 June 2020) to the date of the current searches. That was 18 September 2024. Some additional search terms were included to broaden the original search. All search terms are listed in Appendix 1. The searches were not limited by language, but only studies published in English were considered for inclusion. Reference lists of relevant SLRs identified via the systematic searches were also assessed to ensure relevant papers included in SLRs were picked up by the searches.

Papers included for Question 4 of the 2021 UK NSC evidence map, along with newly identified papers from the updated search, were assessed for relevance to the current review against the inclusion and exclusion criteria of this protocol. After careful inspection, studies included in the UK NSC 2021 evidence map were not considered as relevant to the current review considering the survey study design and therefore they were not deemed eligible for inclusion in the current review.

The search strategy is presented in Appendix 1.

Inclusion and exclusion criteria

• Study design: Qualitative studies such as interviews and focus group studies (including studies using thematic analysis, grounded theory or other appropriate qualitative approaches). If insufficient qualitative studies were identified, the review would have been extended to include questionnaires and/or surveys, but this was not necessary.
• Intervention: AI-based technology applied to medical image classification for screening of any medical condition.
• Comparator: Not applicable.
• Population: Health professionals, providers and users (clinicians and patients) of screening programmes and the general public. Screening population for breast cancer and diabetic retinopathy was the primary focus during inclusion/exclusion, but searching for papers relevant to all target conditions was conducted.
• Outcomes: Themes on ethical issues surrounding the use of AI in screening of medical conditions.

Data abstraction strategy

The following review process was followed.

1. After removing duplicates, the records identified from the searches across databases were imported to EndNote [Clarivate Analytics (formerly Thomson Reuters), Philadelphia, PA, USA] and Rayyan (Doha, Qatar; https://rayyan.qcri.org/) and were combined. Using Rayyan, each abstract was reviewed against the inclusion/exclusion criteria by a single reviewer (MV), with a second independent reviewer (VW) providing input in cases of uncertainty and validating 20% of the first reviewer’s decisions. Any disagreements between reviewers were resolved by discussion. Consensus was reached between reviewers across cases of disagreement without the need to consult a third reviewer to resolve outstanding conflicts.
2. Full-text articles required for the full-text review stage were acquired.
3. Each full-text article was reviewed against the inclusion/exclusion criteria of the protocol by one reviewer (MV), who determined whether the article was relevant to the review question. A second independent reviewer (CD) provided input in cases of uncertainty and validated 20% of the first reviewer’s screening decisions. Any disagreements were resolved by discussion until a consensus was met.

The following strategy was followed for the inclusion of papers in the review:

1. Qualitative papers on the use of AI in screening in the context of breast cancer and diabetic retinopathy were prioritised for full-text assessment and inclusion.
2. As only a few relevant papers were identified, qualitative papers relevant to the use of AI in screening of any medical condition were assessed.
3. It had been prespecified that if few qualitative papers were identified irrespective of medical condition, studies involving surveys/questionnaires would be assessed, but this was not necessary.

Data were extracted independently by one reviewer (MV) using a standardised data abstraction form (see Appendix 2) for each study separately, with 20% checked by another reviewer (CD). Qualitative data on ethics surrounding the use of AI emerging from each included paper were extracted, and using thematic analysis, were summarised as themes for each paper in its corresponding data extraction form.

Critical appraisal strategy

The following tools were used to assess the quality and risk of bias of each study included in the review and the overall confidence in the review findings:

• Qualitative studies: Critical Appraisal Skills Programme (CASP) Checklist.19 CASP (see Report Supplementary Material 1, Table 1) was used to assess included qualitative studies individually, and a statement about methodological limitations and applicability has been included in the data abstraction form for each study (see Appendix 2).

After quality assessment of individual studies was performed, and themes from the included qualitative studies were evaluated and presented using the:

• ‘Confidence in the Evidence from Reviews of Qualitative Research’ (CERQual) Approach developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE)-CERQual Project Group.20

Each theme was assessed for each of the four quality elements.21

• Methodological limitations: The extent of problems in the design or conduct of the included studies that could decrease the confidence that the review finding is a reasonable representation of the phenomenon of interest. Assessed at the study level using the CASP checklist.
• Coherence: The extent to how clear and cogent the fit is between the data from the primary studies and the review finding.
• Relevance: The extent to which the body of evidence from the included studies is applicable to the context (study population, phenomenon of interest and setting) specified in the protocol.
• Adequacy: The degree of confidence that the review finding is being supported by sufficient data. This is an overall determination of the richness (depth of analysis) and quantity of the evidence supporting a review finding or theme.

The GRADE-CERQual was used to assess the body of evidence as a whole through a confidence rating representing the extent to which a review finding is a reasonable representation of the phenomenon of interest. For each of the above components, the level of concerns were:

• no or very minor concerns
• minor concerns
• moderate concerns
• serious concerns.

The concerns from the four components were then used to form an overall judgement of confidence in the finding: high, moderate, low or very low, as described in Table 1. Each review finding started at a high level of confidence and was downgraded based on the concerns identified in any one or more of the four components. Quality assessment of qualitative reviews is a subjective judgement by the reviewer based on the concerns that have been noted. An explanation of how such a judgement was made for each component was included in the footnotes of Qualitative evidence summary: GRADE-CERQual tables where the evidence was summarised as well as narratively under each overarching review theme presented in the Results section.

Results of the quality assessments of individual studies are presented in quality assessment of individual studies; see Report Supplementary Material 1, Table 1.

The level of concern over each GRADE-CERQual component and the overall judgement of confidence in each review finding are presented alongside the narrative synthesis of review findings and Qualitative evidence summary: GRADE-CERQual tables in the Review findings from thematic analysis section in the main body of the review.

Methods of data synthesis: thematic analysis

The synthesis of qualitative data followed a thematic analysis approach. Thematic analysis is a method used to analyse qualitative data, which involves the identification and reporting of patterns in data sets, which are then interpreted for their inherent meaning, thus offering insights into the research question.22 Information was synthesised into the main review findings, that is the review themes as outlined below.

Information relevant to the research was identified from each included paper and was narratively summarised into themes for that paper. Themes emerging across studies were then reviewed by one reviewer (MV), with 20% validated by a second reviewer (CD), and thematic analysis methods were used to identify common patterns of information, which were further synthesised into broader overarching themes. These constitute the main review findings. Each review finding has been presented in the form of a narrative summary detailing the evidence from the relevant papers and how this informed the overall theme plus a statement on the level of confidence for that theme. Considerable limitations and issues around relevance have been specified. For each review finding, a summary evidence table, including a succinct summary statement, the number of studies contributing to the finding, their methodology and the associated GRADE-CERQual confidence rating, has also been presented (see Results section).

Papers selected for inclusion in the present review were first read by the reviewers to familiarise themselves with the data. The data were closely examined, and information relevant to the outcome of interest, ethical issues surrounding the use of AI in screening of medical conditions were noted and given a code/title, for example data confidentiality. The code was then assigned to segments of data within the paper that capture the core message of the code. Identified information across the paper that was relevant to each code was then narratively summarised in the data extraction form for each study under a theme and was given a theme title. These are presented in Appendix 2. Quotes from study participants were extracted and used where appropriate to further illustrate the themes. Once this process was completed for each included study, derived study themes were reviewed for recurring elements, that is common themes across studies and patterns in the qualitative data identified, and were further synthesised into overarching themes that link the review question and the data identified.

Clinical experts

Before database searching commenced, clinical experts were identified and recruited to provide input for the review. As the initial focus was on the use of AI in screening for breast cancer and diabetic retinopathy, consultant radiologists specialising in breast screening or chest imaging and consultant ophthalmologists were contacted. Potential experts were identified using the specialist info directory and were invited via e-mail, with the invitation outlining details of the review, key dates and the expected time commitment.23

To participate, experts were required to declare competing interests, sign a confidentiality agreement form (see Report Supplementary Material 2) and attend a 1-hour teleconference via Zoom (Zoom Video Communications, San Jose, CA, USA) to discuss current UK clinical practice and respond to questions from BMJ-TAG (see Report Supplementary Material 3). They were also asked to review a draft report and provide feedback.

A consultant ophthalmologist and a breast radiologist were recruited based on their availability. Due to a lack of availability of the identified consultant radiologists specialising in breast screening, BMJ-TAG extended an invitation to a mammography expert recommended by the UK NSC, who was available to assist with the project and recruited as a clinical expert. A separate hour-long Zoom meeting was conducted with each expert to address the BMJ-TAG’s questions. Additionally, a second consultant radiologist with expertise in breast cancer screening was recruited as a peer reviewer to provide feedback on the draft report.

The draft report was shared with clinical experts and the peer reviewer on 20 February 2025. Their feedback informed the evidence review and has been incorporated into the discussion of findings.

Dr Nisha Sharma (Director of Breast Screening and Clinical Lead for breast imaging, Leeds Teaching Hospital NHS Trust) and Ms Sarah Meredith (Consultant Ophthalmologist Portsmouth Hospital University NHS Trust and Medical Director of InHealth Intelligence) provided clinical advice via teleconference and feedback on the draft report. Dr Janet Litherland (Consultant Breast Radiologist, West of Scotland Breast Screening Service, Greater Glasgow and Clyde Health Board) provided feedback on the draft report.

Results

Quantity and quality of research available

A total of 2944 publications were identified through database searching. After deduplication, 2191 titles and abstracts were screened against the eligibility criteria of this review.

Figure 1 summarises the volume of publications included and excluded at each stage of the review. Sixty-six publications were ultimately judged to be relevant to the review question after the review of qualitative papers and reviews and were considered for extraction. Fifty-two of those were ultimately judged not be relevant for this review. These publications, along with reasons for exclusion are listed in Appendix 3, Table 35. Publications that were included or excluded after the review of full-text articles are detailed below.

FIGURE 1 Summary of publications included and excluded at each stage of the review. (Figure not reproduced here; see the original article.)

At title-/abstract-level review, 147 studies were selected for further inspection. Of those, the following were considered to be potentially relevant and eligible for full-text screening: 43 primary qualitative studies, 81 survey/questionnaire studies and 23 reviews (that included 2 article papers/literature reviews). Primary qualitative studies were prioritised for full-text screening, and 14 of the 43 examined met the inclusion criteria of the protocol and were included in the review.24–37

Studies on any medical condition were included, given that only a limited number of studies on screening for breast cancer and diabetic retinopathy were identified. In line with the criteria prespecified in the review protocol, a descriptive synthesis of studies involving surveys/questionnaires was not undertaken as sufficient evidence from qualitative studies was identified (> 10 studies). Included studies are presented in Table 2.

All primary studies (n = 14) were qualitative studies. These included studies focusing on patients (n = 4); healthcare professionals such as radiologists, primary care physicians and ophthalmologists (n = 5); a range of professionals and stakeholders involved in the development and implementation of AI in radiology settings, including clinicians, radiologists, software developers and vendors (n = 4); and one study that included a range of professionals and stakeholders, clinicians and patients. Studies on patients included women of breast cancer screening age (n = 3), men with experience of prostate cancer screening (n = 1), patients with pulmonary hypertension (PH) (n = 1). Studies on healthcare professionals included breast cancer screening specialists (n = 3), ophthalmologists (n = 1) or radiologists with experience in prostate cancer screening (n = 1). The studies were conducted in the UK (n = 4; one of which was in Scotland), Sweden (n = 3), Australia (n = 2), Germany (n = 2), and one each in Norway, China and the USA.

The studies examined a broad range of different aims that are summarised in Table 2, and these were relevant to different target conditions, including: breast cancer (n = 5), lung cancer/pulmonary nodules (n = 2), diabetic retinopathy (n = 2), prostate cancer (n = 2), cutaneous melanoma (n = 1), PH (n = 1) or no specific target condition (n = 2). Studies used a range of qualitative methodologies, most frequently semistructured interviews (n = 9). Other studies used focus groups (n = 2) or a combination of focus groups and semistructured interviews (n = 2) and a dialogue group methodology (n = 1). Details of each study are summarised in Table 2. Full data extraction forms for each study are provided in Appendix 2, Tables 21–34. The quality assessment of each included study is provided in Report Supplementary Material 1, Table 1.

Review findings from the thematic analysis

As a result of the thematic analysis of the included papers, the following themes were identified. A detailed narrative synthesis of each theme is presented below with a brief statement of the overall assessment of confidence in each finding. This section is followed by a succinct summary of each review finding alongside a detailed quality assessment rating (GRADE-CERQual tables; see Qualitative evidence summary: GRADE-CERQual tables, Tables 3–13).

Narrative summary of review findings

Review finding 1: human in the loop and inability of artificial intelligence to make holistic assessments

Both patients and professionals across a large number of studies highlighted the importance of maintaining human oversight in screening and did not favour the possibility of autonomous AI. Women participating in breast screening expressed a strong preference for humans over AI alone but viewed AI as a valuable complement to human expertise. They stressed the importance of human oversight in the diagnostic process, recognising the unique skills of radiologists that AI cannot replace, a view that was shared with male patients with varied experience of prostate cancer diagnostics. They expressed that while AI is a powerful tool, it requires human control to ensure reliability. Participants supported a collaborative approach, where AI aids radiologists in detecting abnormalities, maximising the strengths of both for optimal outcomes.25,27,28,30,31,33–37

As stated by one patient: ‘as long as there is a human factor involved, I believe it can be very effective’.30

Some women questioned whether AI could adequately replace the two radiologists who currently assess mammography images, and many appeared to be greatly reassured by having multiple radiologists working independently. While they acknowledged AI’s ability to detect details that the human eye might miss, they remained uncertain about the specific areas where AI surpasses human capabilities.

Patients, including women undergoing breast cancer screening, highlighted several key strengths of humans, including their holistic perspective, ability to connect diverse information and consideration of various factors related to the individual woman. Humans were seen as more capable of investigative thinking, imagination and contextual understanding when compared to AI. Patients strongly believed that human involvement adds an irreplaceable element, particularly intuition and empathy, which AI currently lacks and may never develop. Research skills, professional experience and flexibility were also highlighted as human attributes bringing rigour to screen-reading processes. While patients recognised AI’s potential, they maintained that the human touch remains essential and offers qualities beyond AI’s capabilities. Moreover, some concerns were raised over the lack of empathy of AI systems and their inability to understand emotional issues. These were considered to be particularly important in the context of diagnosing complicated diseases, such as brain tumours or lung cancer, where it was thought that if the findings of radiology reports were to ever be communicated by AI without the presence of medical professionals, this could cause anxiety and negative emotions. While some were hopeful regarding the combination of AI and humans in preventing potential errors, they perceived that there is currently considerable uncertainty regarding the performance of AI and that solely relying on AI would be a substantial leap. Overall, participants found AI implementation to be acceptable as long as a radiologist remained involved, at least in the first round of mammography screening. They expected clinicians to continue reading mammograms, oversee AI decisions and maintain control, expressing the view that AI should not be ‘the ultimate judge’25 despite considering that ‘AI might end up picking up things that humans miss’.25

When asked about AI-supported screening of diabetic retinopathy, general practitioners (GPs) and medical practice staff (MAs) emphasised that AI cannot consider patient care holistically like a doctor can. Given their deep sense of responsibility for patient’s well-being, GPs expressed discomfort in providing advice based solely on AI-generated insights. Breast radiologists as well as stakeholders involved in the evaluation of an AI system targeting breast cancer detection in the real-world setting of the Scottish National Breast Screening Service [Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD) programme] also highlighted that AI cannot function independently without human assistance. For example, assembling different parts of a mammogram for a comprehensive evaluation may be beyond AI’s capabilities, and it might fail to detect technical issues, potentially leading to missed cancer diagnoses or unnecessary patient recalls. In the context of breast cancer in particular, radiologists highlighted that if one radiologist would be replaced by an AI, then the remaining one would have to be a very experienced examiner, because otherwise cancers could be missed. Additionally, professionals highlighted that AI lacks emotions and the ability to communicate with patients or medical professionals to justify its assessments. It may also struggle to provide accurate health projections, as it cannot review a patient’s complete medical history or compare past and recent mammograms effectively. To mitigate these challenges, all participants recommended maintaining human involvement, particularly during the initial phases of AI integration. However, one radiologist in one study appeared to favour the idea of autonomous AI, that is working without interaction with the radiologists, but the vast majority believed that human supervision was necessary. Moreover, despite the fact that their confidence in AI-based software grew with use and experience, radiologists, who had been using AI-based software in the context of screening for pulmonary nodules, had the view they should not be entirely reliant or overconfident in the results of AI and should be double-checking the results of AI systems.

Overall assessment of confidence was high as concerns identified over methodological limitations were minor and there were no further concerns to lower confidence in this theme.

Review finding 2: liability/accountability

Accountability and legal liability issues were discussed by healthcare practitioners, including GPs and ophthalmologists, in the context of diabetic retinopathy screening;25,26,28–30,32–36 by radiologists and radiographers in breast screening services, prostate cancer tumour imaging; by professionals involved in the development of AI for clinical use in early diagnosis of PH; and by stakeholders involved in the evaluation of the iCAIRD programme. Concerns were raised about the unresolved question of responsibility in the interpretation of outputs from AI application, and about who would be viewed as responsible (the doctor or the system), for medical decisions in cases where the implementation of an AI-assisted tools results in the classification of a lesion as benign, that is then shown to be malignant. Stakeholders raised the question of ‘who takes responsibility’ ‘if the machine misses something’34 as an ethical issue for consideration, while GPs and ophthalmologists raised concerns about patients taking legal action and holding GPs accountable in cases of possible harm to patients as a result of the application of AI-supported diagnostic tools for diabetic retinopathy screening. On the other hand, radiologists involved in prostate cancer tumour imaging mentioned unclear liability and ethical consideration regarding AI usage. Moreover, radiologists in NHS breast cancer screening units were less concerned about the potential impact of AI innovations for their professional accountability in diagnosis in clinical decision-making. They were confident that their institution would establish appropriate policies and mechanisms to protect individual practitioners in cases of challenges to AI-assisted diagnostic methods.

Both patients with PH and professionals involved in the development of AI for clinical use on this target condition expressed a clear preference for specialist clinicians having the ultimate responsibility over the custody and control of AI, including the interpretation of outputs from the AI. The idea that specialists should have responsibility for allowing AI to play a role in diagnosis was compared to buying a self-driving car, with participants agreeing that the responsibility for the autonomous vehicle ultimately rested solely with its owner. Within this framework, AI was seen as a resource to inform diagnostic decisions whose acceptability appeared to depend on it having an assistive role in the diagnostic process, assisting clinicians without reducing their capability to take control when required. Furthermore, clinicians were highlighted as having the ultimate responsibility for clinical decisions by healthcare practitioners involved in the implementation, deployment and development of AI-assisted decision-making software for diabetic retinopathy and lung nodules in China. However, when a patient’s care required input from multiple clinical departments, there was uncertainty about who had the authority to approve and take responsibility for reports generated by AI-assisted decision-making software despite AI-generated reports having been designed to support rather than replace the decisions made by human clinicians. Challenges were reported to arise when non-specialist clinicians were asked to endorse these reports. As they lacked the same level of expertise as specialists, they found it difficult to validate the accuracy of AI-generated findings. Concerns about legal liability, alongside differences in clinical knowledge, made them hesitant to sign off on these automated assessments. This emphasised the need to establish an interdepartmental collaborative network between specialty and non-specialty clinical departments and provide clear definitions of the roles and responsibilities within these departments.

From a patient’s perspective, many women participating in mammography screening placed great trust in radiologists, viewing them as ultimately responsible for the outcome of AI-assisted mammography screening.

Artificial intelligence was seen as a tool that cannot be held accountable for its actions, thus some participants believed that any mistakes made by AI would ultimately be the responsibility of humans and that ‘it is always the medically responsible physician, the radiologist, who is accountable’ as “we can’t blame the technology”’.30

Similarly, doctors were seen as accountable by men with varied experiences of prostate cancer (including men with prostate cancer diagnoses, on active surveillance or disease-free) for interpreting, verifying and quality-controlling AI-based test results, ensuring that AI interpretations aligned with their clinical judgement, experience and expertise. They were assigned the role of technical supervisors, whose presence was crucial to intervene if the AI made an incorrect interpretation. The doctor’s role as an AI controller was deemed vital, particularly in high-stakes decisions, such as selecting treatment options. Thus, the need for quality control ‘of sorts’33 to account for software errors with potentially negative consequences was highlighted. However, some perceptions appeared to vary regarding the issue of responsibility. For example, one participant offered a different perspective, suggesting that a radiologist’s mistake could be attributed to a fault in the device or image-computer system, while another participant emphasised the importance of the radiologist taking AI indications seriously, particularly when it flags a potential case of cancer.

Overall assessment of confidence was high as concerns identified over methodological limitations and coherence were minor and counterbalanced by the wealth of information available to illustrate the theme.

Review finding 3: algorithmic bias and comparability/discrepancies of real-world versus training data

Some text in this section has been reproduced with permission from Aquino et al.24 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

The lack of compatibility24,32,34–36 between training data and real-world data and the risk of algorithmic bias arising from this discrepancy was discussed by participants, including clinicians, stakeholders, clinical and industry setting professionals involved in the development and implementation of AI algorithms across a number of studies in the context of screening for lung nodules, diabetic retinopathy, breast cancer, prostate cancer and PH. Stakeholders involved in the integration of an AI algorithm in the Scottish National Breast Screening service for breast cancer detection raised the potential of inequalities in detection if AI is not trained appropriately, using a wide range of population data such as different age groups and ethnic communities.

As highlighted by an AI regulatory expert in one study, ‘there was a bias against African Americans. … if you are Black American … you are certainly more likely to be mislabelled as low risk if you actually were high risk’.24

The risk of errors or inaccuracies of AI software, such as incorrect classifications potentially due to biases in the training data, was also raised by a radiologist in the context of prostate cancer and the application of AI in prostate MRI readings. Potential harms of algorithmic bias raised by clinicians and regulators of healthcare AI ranged from medico-technical harms, such as compromising patient safety, to broader social and public health harms such as contributing to health inequity. It was suggested that such harms may undermine the promise of healthcare AI. It was also highlighted that the misconception that AI systems are objective and infallible may lead people to dismiss the problem of algorithmic bias. Indeed, a small group of AI developers, entrepreneurs and screening programme managers within one study denied the existence of bias, arguing that bias existed in other industries but not in medicine and was not likely to occur in medical diagnosis because ‘imaging technologies are objective’ and ‘the human body is universal’.

For example, as highlighted by one clinician, ‘in radiology … a C-T scan will look exactly the same with black and white pixels. It doesn’t matter if you’re black, or white … a rib fracture is a rib fracture. There is no bias there’.24

On the other hand, a wide range of professionals, including clinicians, developers, regulatory experts and professionals involved in the development of AI for clinical use in PH, also raised the diagnostic biases embedded in data sets, but some stressed that these are part of algorithm development and the risk of diagnostic bias was inevitable and a limitation that one must be prepared to live with in the early design stage of the algorithm.

Within this context, although it was emphasised that patient data can be ‘far from ideal’ they still represent ‘an improvement on what we got at the moment’, a part of any tool that is used.36

Thus, bias in data sets was something that some professionals were willing to accept as long as they are aware of them, as it was acknowledged that ‘there’s going to be pros and cons with any AI approach’ and it can be ‘very dependent on the user of AI using that AI responsibly’. In addition, some professionals viewed bias as ‘unintentional’ and recognised it as a problem to some extent but not as an ethical concern or at least not raising more ethical concerns than the status quo. They highlighted the potential benefits of AI irrespectively of the presence of bias, arguing that bias is an ongoing problem even among clinicians and AI bias is preferable to human bias.

In addition, as raised by professionals from both clinical and industry settings, who were involved in the development of AI-assisted decision-making software for lung nodules and diabetic retinopathy, their performance relied on the availability of high-quality and accurate source data. It was highlighted that AI-assisted software was not highly compatible with, and adequately trained on, data collected during routine clinical practice. When developing software, the data sets used to train machine learning and deep learning algorithms undergo meticulous cleaning to remove ‘poor-quality’ data that create imaging noise. However, in real-world clinical settings, factors such as equipment limitations, patient motion and the proficiency of technicians can introduce imperfections in the imaging data, which the training process does not account for.

Last, but not least, it was raised that creating an algorithm that produces biased results could essentially violate human rights, highlighting that algorithmic bias may not only constitute an ethical problem but also a legal problem.

Overall assessment of confidence was moderate due to minor concerns over methodological limitations, coherence and relevance.

Review finding 4: data confidentiality, cybersecurity and consent

Some patients, healthcare practitioners and stakeholders involved in AI software for clinical use expressed concerns related to data confidentiality, privacy and cybersecurity, while others believed that the potential benefits may outweigh the implications for privacy.24,27,30,32,36,37 When asked about their attitudes towards sharing mammography images with different healthcare professionals, women participating in mammography screening consistently expressed their willingness to share their health information within the healthcare setting. They firmly believed that healthcare units should work together and exchange information more effectively than they presently do, highlighting the importance of such collaboration in safeguarding patient safety. However, when it came to sharing health-related data or personal data with companies, the participants were more reluctant, with such sharing being perceived as a great violation of privacy by some participants. Anonymising mammograms helped ease women’s privacy and personal identification concerns, as they felt that mammogram images of their breasts were less concerning when fully anonymised and disconnected from their personal identities.

As stated by one participant: ‘No, I wouldn’t have any issues with that [sharing mammograms] as long as it can’t be linked to me as a specific individual’.30

However, many viewed giving consent to share their data as essential and associated to their ability to ensure that the mammograms would be used for a meaningful purpose. They were willing to provide consent if the data were used to save lives, develop medical devices or support research, but they were strongly opposed to sharing their data if it were intended for commercial companies or advertising purposes.

Patients with PH, carers of patients and a range of professionals, including clinicians, developers, regulatory experts involved in the development of AI for clinical use in PH in the UK, recognised patient data as the key driver behind earlier diagnoses and more informed treatment decisions enabled by AI implementation and were not overly concerned about data risks and implications for patient privacy. Any privacy concerns were often outweighed by the potential benefits of early diagnosis and earlier treatment opportunities. For example, patient carers thought that the potential loss of privacy was worth the risk and ‘an acceptable cost’36 in the context of PH, considering it is a rare disease and the long length of time often required to make a diagnosis. Nevertheless, similarly to patients undergoing mammography screening, PH patients had concerns over commercial motives and despite being happy about the NHS having access to their data, they felt less assured about access to their data by external organisations and companies outside the NHS, such as insurance companies, that could misuse their data and make biased decisions in insuring people and adjust prices of insurance policies. Professionals, including PHA-UK (a patient group that supports people with PH) representatives, highlighted the breadth and complexity of data risks, including misuse and misappropriations, and its economic value. They noted how they have been approached by pharmaceutical and insurance companies who view patient data as ‘absolute gold dust’ on multiple occasions. However, when asked whether they were happy that their data could be sold by the NHS to foreign companies, some patients appeared to be unaware of the potential financial value of their data and were naive to the risk of misuse.

Patients also raised concerns about the extent to which AI systems could maintain confidentiality so that their health information is protected from potential hacking or data leakages.

The importance of data privacy and security was further highlighted by healthcare practitioners and other stakeholders involved in the implementation, deployment and development of AI-assisted decision-making software for diabetic retinopathy and lung nodules in China. It was highlighted that AI software vendors aiming to collaborate with hospitals in the Chinese healthcare system were mandated to implement strong information security protocols and obtain the required security certifications to meet safety requirements. The lack of such security measures and certification, often seen among innovative technology companies, constituted barriers to successful implementation. It was noted that legislation, such as the cybersecurity law, required a multilevel protection scheme to safeguard data security and uphold patient privacy. Professionals involved in developing, selling, regulating or implementing healthcare AI called for data governance, with disagreement expressed on how to handle variables representing complex social information, such as gender and racial identities, that are typically marked with historical or continuing prejudice and marginalisation, when developing algorithms. In addition, concerns raised related to cybersecurity were also related to data storage requirements, including their associated costs.

Overall assessment of confidence was low due to some concerns over methodological limitations, coherence and relevance.

Review finding 5: transparency over how artificial intelligence algorithms make decisions and over disclosing their use in decision-making

(a) Transparency over disclosing the involvement of AI to make clinical decisions24,25,31,32,36,37

Different views emerged between professionals involved in the development of AI for clinical use in the screening and early diagnosis of PH (including clinicians and computer scientists) and patients or carers of patients living with PH when it came to expectations about disclosure of using AI. While patients expected to be informed about the use of AI in their diagnosis, professionals did not consider it to be necessary to disclose that AI played a role in the decision-making process.

As stated by a biomedical scientist: ‘the more we go into telling people how we’ve collected the [patient] data or made that decision I think the worse that is’.36

The professionals compared it to other medical technologies, where patients have no more understanding beyond knowing that it is a laboratory test or an imaging procedure. The difference in patients’ perspectives suggests that AI holds a distinct epistemic role for them compared to professionals, setting it apart from other diagnostic technologies. Breast screening radiologists also had divergent views on whether informing women about AI’s use in mammography screening would enhance the screening programme. Some believed that detailed information would not improve the process, while others emphasised the importance of full transparency.

(b) Transparency in the AI algorithm’s decision-making24,25,31,32,36,37

Patients, who expressed concerns about the quality and trustworthiness of AI outputs resulting from the use of AI systems to make sense of radiology findings, highlighted the need to increase system transparency by explaining how AI arrived at its conclusions. Women undergoing breast screening highlighted transparency as an important factor than can mitigate the potential disengagement from breast screening services resulting from patients’ lack of familiarity with AI tools. In their view, breast screening services needed to be transparent about the role of AI and its performance, including the benefits and risks. Respectful information sharing appeared to increase women’s willingness to support the idea of breast screening AI. Similarly, professionals involved in developing, selling and regulating or implementing healthcare AI recommended open disclosure of AI limitations as a way to overcome any potential bias in AI algorithms. They emphasised the importance of open acknowledgment of potential biases when communicating predictions or recommendations generated by AI. In addition, healthcare practitioners involved in the implementation and deployment of AI-assisted decision-making software, for lung nodules and diabetic retinopathy in China, encountered challenges in integrating AI-assisted decision-making software into clinical practice due to their limited understanding of AI algorithms. The lack of necessary knowledge and expertise made it difficult for them to grasp the reasoning behind AI-generated recommendations and decisions. This uncertainty led to a lack of trust and hesitation in adopting AI-assisted software. Participants suggested that improving case-specific explanations and enhancing overall transparency (such as clarifying the algorithm’s functionality, strengths and limitations) would help demystify the ‘black box’ of AI technology.

Overall assessment of confidence was low due to moderate concerns over methodological limitations and relevance and minor concerns about coherence.

Review finding 6: risk to doctors’ competencies and impact on the profession

Professionals, including GPs, radiologists and radiographers, expressed varying perspectives on how AI technology might affect their skills, training opportunities and professional accountability and reputation.26–28,30,31,34–36 Concerns were expressed about the potential for AI to automate their work or affect their skill development. Concerns were raised about over-reliance on AI in radiology, with fears that it could jeopardise essential diagnostic skills, particularly among trainees, limit radiologists’ exposure to critical learning experiences and undermine their professional identity. Stakeholders, including radiologists, patients and experts from the iCAIRD programme, highlighted risks such as deskilling, increased false positives and the potential for AI to dictate which images radiologists review, leading to a diminished role for human expertise in breast cancer screening.

Conversely, others thought the role of radiologists would evolve positively with AI or viewed the integration of AI screening (for diabetic retinopathy in particular) as an opportunity to enhance a practice’s image by showcasing technological advancement. Additionally, the GP’s role as a mediator was highlighted, as they may not perform or evaluate the examination but remain responsible for discussing results, explaining their significance and addressing patient questions.

A view expressed by a consultant radiologist about the implementation of AI-based diagnostic decision support systems for pulmonary nodules was that ‘AI won’t replace radiologists, but radiologists who use AI-enabled tools will replace radiologists who don’t’.27

Moreover, although radiologists did not think AI would replace them, some mentioned that it could help upskill some staff and ‘democratise’ image knowledge by assisting less experienced clinicians in making more precise interpretations of imaging outputs. This was in line with the views expressed by a woman undergoing breast screening, who viewed the development of AI as requiring continuous skill improvement among radiologists and acknowledged that to effectively support the training and advancement of AI systems, radiologists must stay updated and further develop their expertise, highlighting the mutually beneficial relationship between radiologists and AI, where continuous skill development and collaboration drive progress in both areas. Radiologists in breast cancer screening did not think that AI threatened their profession at a fundamental level, highlighting that being a radiologist in a screening programme is ‘so much more than reviewing mammograms’31 and thus thought that AI only replaces part of their work. Some radiologists, including those from breast screening NHS units, emphasised their extensive workload and the potential of AI to decrease it. They believed that the high-level and varied character of their roles made them less vulnerable to deskilling and instead viewed AI innovations as a way to reduce routine tasks, enabling them to concentrate on more engaging and complex tasks.

The debate of skilling versus deskilling of healthcare professionals as a result of the implementation of AI algorithms was discussed by professionals involved in the development of AI algorithms (including screening, imaging and biomarker algorithms) for clinical use for the early diagnosis of PH. It was emphasised that AI may produce outputs that do not resonate with their existing understanding of the disease, and questions were raised about whether specialist clinicians had to learn new skills to interpret or explain new characteristics of PH due to algorithms potentially making associations that do not seem to correlate with clinicians’ current understanding of the pathology. Within this framework, three components of new skills that may be required when working with AI were highlighted:

1. the traditional skills of gaining specialised knowledge about PH, such as understanding the disease process
2. the ability to evaluate how well the AI system reveals different aspects of the disease, questioning whether it provides meaningful insights
3. the skill of assessing the system’s trustworthiness, including whether the algorithm relies on surrogate measures and if its calculations can be trusted.

Overall, it was emphasised that working with AI will require knowing when to rely on established knowledge about and experience of the disease and knowing when to integrate and adapt that knowledge based on insights provided by AI.

Overall assessment of confidence was high due to the concerns identified over methodological limitations and coherence being minor and a wealth of information to support the theme.

Review finding 7: trade-off between false positives and lesion detection

Stakeholders associated with the integration of an AI algorithm in the Scottish National Breast Screening service for breast cancer detection raised that AI might overdiagnose (e.g. recognise lesions at a much earlier stage or lesions of insignificance) or misdiagnose (e.g. misinterprets scars as cancers).25,30,31,34 Women participating in a national breast cancer screening programme strongly communicated that failure to detect a case of cancer is ‘terrible’ and highlighted the significance of healthcare professionals making every effort possible to guarantee early detection. They acknowledged the trade-off between raising increased concern as a result of more false-positive cases and the possible harm of not detecting a cancer diagnosis. However, they conveyed a clear preference for prioritising the identification of potential cancer cases over reducing false positives. They considered that it was preferable for more individuals to experience the anxiety associated with a false positive than for a potential case of cancer to be missed and agreed on setting AI to a highly sensitive level to improve cancer detection.

As stated by one woman:

we should be able to live a bit more in reality [not being afraid of bad news]. And if we can get the help [hinder cancer] that it entails, that I have to go for an extra appointment, then I would just be thankful and accept it … I’m not inclined in the other direction … I would rather go for the extra appointment than not do it.30

Radiologists on the other hand, when interviewed on their experience of using an AI tool in mammography screening (as part of a clinical trial), were concerned that AI flagged more cases (for second-stage screening) compared to radiologists in the first screening, increasing the number of cases in the second screening. Hence, radiologists were concerned that AI technology generated a high number of false-positive cases, increasing their overall workload. For example, the AI responded to dense breasts that the radiologists knew were not cancerous, leading to more work in the second review, but not the first. They considered that the AI ‘messed it up’ for them because of the considerable amount of time required to assess the extra cases. They were also concerned that more resources would be required in the second screening, which might not be economically possible. The view that false positives resulting from the introduction of AI in breast cancer screening may put pressure on healthcare systems was also endorsed by a group of patients who emphasised the burden of potential long travel times involved in ‘unnecessary’ follow-up appointments in the case of a false positive.

Overall assessment of confidence was moderate due to minor concerns over methodological limitations and coherence.

Review finding 8: lower tolerance for errors made by artificial intelligence versus humans

The failure to detect cancer was perceived as devastating by women participating in mammography screening as part of a national breast screening programme (Sweden), who consistently emphasised the importance of the ability of AI to detect cancer. Nevertheless, their acceptance of failure to detect cancer differed between radiologists and AIs, being much higher for mistakes made by radiologists than by an AI. Women felt that human error is unavoidable — and thus understandable — but believed that technology, including AI, should not make mistakes when used in health care.30,34

As stated by one participant, ‘[ … ] we all know that a technology [ … ] it should be a hundred percent …’.30

Women thought that for the implementation of AI to be acceptable, it should maintain a high level of security and operate without errors. Women questioned the implementation of AI that does not meet these criteria. In particular, two women highlighted the potentially catastrophic consequences of systematic errors in diagnoses by AI systems when compared to the relatively limited impact of a few cancer cases missed by a single radiologist in a day. Within this context, a range of negative emotions were expressed, including feeling bad, disappointment and a sense of being cheated, if AI were to make a mistake in detecting any irregularities in mammograms.

Stakeholders with an interest in the integration of an AI algorithm in the Scottish National Breast Screening service for breast cancer detection also raised that the public or patients may not accept errors made by AI.

As stated, ‘… a radiologist can miss something, if the machine misses something it would have a very different public perception’.34

On the other hand, a few women participating in mammography screening recognised that neither humans nor AI can be 100% accurate all the time. They acknowledged that AI relies on data and statistical predictions, which inherently includes the possibility of errors. These participants emphasised that no system, whether human or AI, is infallible, and both can potentially make mistakes in their assessments.

Overall assessment of confidence was moderate due to minor concerns over methodological limitations and coherence and adequacy.

Review finding 9: cases of reduced effectiveness

Clinicians and stakeholders,27,30,32 primarily in relation to the chest radiology setting and also breast radiology and diabetic retinopathy, highlighted potential cases of reduced accuracy of AI-based tools. In particular, stakeholders – primarily healthcare practitioners – involved in the implementation, deployment and development of AI-assisted decision-making software in therapeutic areas of diabetic retinopathy and lung nodules, reported that lower test accuracy, high false-positive rates, overestimation of lesion size and misclassification of lesion types were common despite the strong evidence base underpinning AI-assisted decision-making software. Women participating in breast radiology also expressed concerns regarding AI’s effectiveness to detect cancers in more challenging scenarios, for example where a woman has breast implants or dense breast tissue.

Radiologists using AI-based software, in the context of screening for pulmonary nodules in UK hospital-based chest radiology settings, identified specific regions of chest images where the software may not produce accurate readings. For example, an area in the lungs (in the central midline portion of the thoracic cavity), where, due to the presence of blood vessels, the AI may have ‘blind spots’ and not produce precise results. As a result, caution was emphasised when reviewing these areas, with radiologists advised to remain aware of AI’s limitations and carefully assess its findings in regions where it may be less reliable.

As highlighted by a radiologist:

It doesn’t cope very well with identifying masses … in the lungs where blood vessels interface with the mediastinum. Another area that it struggles with is lesions in the central airways. A human will specifically review those areas … as potential blind spot or weakness for the AI.27

It was reported that, over time, users learned how to work around the limitations of the system, identifying which regions of images disproportionately produced erroneous results and which were more reliable and interpreting the findings accordingly. Within this framework, radiologists specialising in cardiothoracic imaging reported using AI-based imaging software differently from general radiologists, focusing on more detailed information such as nodule dimensions rather than just detecting their presence. Differences in the use of AI-based software was reported to vary with experience, with more experienced radiologists feeling better able to identify and disregard the false positives picked up by AI. There was concern that less experienced users might over-rely on the machine’s findings, potentially leading to unnecessary follow-up.

Overall assessment of confidence was low due to minor concerns over methodological limitations and adequacy and moderate concerns over applicability.

Review finding 10: lack of familiarity and trust as a barrier to implementation/the need to address lack of familiarity and trust in artificial intelligence algorithms

When asked about their views on using AI in breast cancer screening,25,29,32 women of screening age were concerned that the unfamiliarity of AI as a tool could change how patients feel about breast screening services, which may lead to disengagement from services. The need to remember that most women undergoing breast screening do not have a scientific background, can be older and thus may find AI technology scary was emphasised. Within this framework, the time required for patients to become familiar with the new technology was emphasised. Some women viewed AI as science fiction or as a technology of the future, currently unattainable in the real world, or emphasised the possibility that the implementation of AI in breast screening could ‘backfire’ if it is forced.

Lack of familiarity with AI algorithms and a limited capability to understand the rationale behind their recommendations and decisions, which may be due to a lack of necessary knowledge and expertise, was also raised as a potential challenge to the implementation of AI-assisted decision-making software by healthcare practitioners involved in the implementation and development of such software in the context of lung nodules and diabetic retinopathy in China. It was emphasised that this could lead to lack of trust and reluctance towards the implementation of AI software implementation in clinical practice. Trust in AI-based decision-making tools and their ability to provide reliable diagnostic guidance were further highlighted as prerequisites for implementation in the healthcare system in the context of cutaneous melanoma. On the other hand, it was highlighted that addressing case-specific reasoning and ensuring overall transparency, such as clarifying the algorithm’s functionality, strengths and limitations, could help clinicians see ‘what is inside the black box’ of AI technology and mitigate such issues. An important element of trust raised by clinicians was supportive clinical evidence on AI tools’ diagnostic accuracy from high-quality trials as well as positive opinions from key opinion leaders in the field in which the tools are to be applied, such as dermatologists in the case of cutaneous melanoma.

Overall assessment of confidence was moderate due to the concerns identified over methodological limitations and relevance.

Review finding 11: systemic integration of artificial intelligence

While the themes above address core ethical principles, this theme brings together practical, system-level considerations raised across studies that may affect the fair and safe implementation of AI in medical clinical screening. These do not represent ethical issues per se but highlight contextual factors, such as infrastructure, resource constraints and workflow integration, that can shape the ethical challenges of implementation in clinical practice.27–29,32–34,36

The integration of AI-assisted tools into clinical screening raises several ethical challenges related to equity, patient safety and the appropriate use of technology. One major ethical concern is healthcare inequality: AI-driven screening systems, which rely on imaging data, can exacerbate disparities in access to health care. In hospitals with limited resources, particularly those lacking essential equipment such as computerised tomography (CT) scanners, AI’s potential to improve diagnostics may not be realised, creating an inequitable distribution of healthcare benefits.

Another key ethical issue concerns patients’ safety and the risk of delays in diagnostic processes. In some settings, poor infrastructure and connectivity caused delays in AI-generated reports, raising concerns about the timeliness of critical diagnostic information. Delays in AI reporting may compromise decision-making and impact patient care, especially in urgent cases.

The ethical principle of trust is also central to AI integration. Healthcare professionals emphasised that, for AI to be effectively and ethically implemented, it must be rigorously tested and its diagnostic capabilities should be unambiguously validated. Transparency in how AI makes decisions is critical as is the development of clear guidelines that define AI ‘s role as an assistive tool rather than a replacement for human judgement. This ensures that clinicians remain responsible for the final decisions, preserving the human element in patient care.

Overall assessment of confidence was low due to concerns identified across quality elements.

Other issues for consideration only emerging from one study each

(a) Tension between GPs and ophthalmologists28

Both GPs and ophthalmologists highlighted that tensions between the two professional groups could potentially arise as a result of the implementation of AI-supported devices for screening of diabetic retinopathy. Tensions may arise due to concerns about reduced referral rates to ophthalmology services, communication challenges, power dynamic in decision-making and concerns about the impact on professional roles and autonomy. More specifically, lower referral rates may cause ophthalmologists to feel undervalued or threatened in their specialised roles. Furthermore, existing tension in the collaboration between GPs and ophthalmologists due to delayed information transfer (such as delays in receiving fundus examination results), the use of many specialist terms by ophthalmologists which GPs often find incomprehensible and a lack of exchange between specialties (leading to feelings of frustration on both sides) may become more pronounced with the wider AI integration, leading to further misunderstandings.

As stated by one ophthalmologist:

What is of course also viewed with a bit of scepticism among the ophthalmologists is when, as a result, far fewer people are referred to the ophthalmologists and the general practitioners think that they can do everything themselves.28

However, others saw the implementation of AI as a potential opportunity for closer and better co-operation between specialists and GPs, allowing for a more integrated approach to patient care.

(b) Customisation of AI to meet clinical needs32

Healthcare practitioners and professionals involved in developing and implementing AI-assisted decision-making software for diabetic retinopathy and lung nodules in China highlighted that the software did not fully meet their clinical needs. Currently, the AI algorithm is primarily designed and trained for general, non-personalised applications. Clinicians found the AI-assisted software to be inadequate for handling complex cases that require a holistic approach and a deep understanding of a patient’s medical history. To better align with the diverse needs of healthcare practitioners across different specialties and clinical settings, it is crucial to incorporate customisation options, enhance algorithm adaptability and demonstrate a commitment to continuous improvement.

The GRADE-CERQual tables providing a succinct summary of the narrative summary of review findings alongside with the detailed assessment of confidence in each finding are presented below (see Tables 3–13).

Qualitative evidence summary: GRADE-CERQual tables

The themes identified through thematic analysis are presented in Figure 2, categorised into overarching ethical concern categories.

FIGURE 2 Thematic diagram. (Figure not reproduced here; see the original article.)