Implementation of an artificial intelligence-based system for mammography in the compulsory medical insurance program: results of a 3-year study

Introduction

The development of artificial intelligence (AI) technologies profoundly impacts healthcare, driven by their ability to rapidly and effectively analyze heterogeneous data alongside growing medical data availability. AI is applied across medical specialties including internal medicine, surgery, and radiology, with diverse uses such as clinical decision support, personalized treatment planning, therapeutic drug monitoring, and more (1).

Mammography represents one of the most promising fields for AI applications due to the high radiologist workload, compounded by an escalating global disease burden. Breast cancer is the most prevalent malignancy among women and the leading cause of cancer-related mortality in this population (2). Consequently, numerous countries implement mammography screening programs, facilitating early detection and timely intervention to reduce mortality (3). However, screening entails substantial resource demands, including double-reading by radiologists (4). Unfortunately, radiologists remain scarce facing widespread overwork and burnout as mammography volumes steadily increase (5,6). Human factors contribute to diagnostic errors like delayed cancer diagnoses or unnecessary biopsies in malignancy-free women (7,8). In contrast, AI enables efficient image analysis with automated quantitative feature extraction (9). Original studies and systematic reviews confirm AI’s value in mammography, primarily through automated and precise image interpretation (10-13). Nevertheless, evidence of AI limitations has been increasingly documented.

Despite active discussions of legal and ethical constraints, clinical and technical limitations—including development, testing, and monitoring for stable performance—receive less attention (14,15). For instance, a recent systematic review of AI errors in mammography noted technical issues in only one of seven studies (16). Local validation (assessing clinical fit and performance) is key for identifying flaws (17), but mammography AI validation is complicated by its “black box” nature, data heterogeneity, subjective reads, diagnostic metric variety, and resource limits, often yielding unreliable results (15,18). External validation on independent datasets occurs rarely and shows accuracy drops of up to 24% compared to internal tests (19-21). Many AI systems lack peer-reviewed efficacy evidence (22).

Current initiatives aim to improve the quality of radiology AI systems and ensure their comprehensive evaluation. Standardized reporting checklists for AI development in healthcare structure research studies but have little impact on the creation of the models themselves, since applied research usually relies on existing technology (23-27). Procurement evaluation methods for medical leaders are also applied post-development (28,29). Diverse testing frameworks exist but lack examples and overlook key factors (30,31).

First, most frameworks overlook AI’s dynamic evolution through retraining, requiring continuous post-update assessment (32). Second, they neglect radiologist involvement throughout development and testing, which risks creating misaligned solutions (33,34). Third, methodologies rarely account for data heterogeneity, volume, and multi-center representation (35-37). Finally, few address real-world integration into healthcare systems like compulsory medical insurance (CMI) programs, which demands comprehensive functional, stability, and compliance validation (38).

In contrast to these existing validation frameworks, which are often static and applied as a one-time assessment post-development, we propose a novel lifecycle-based methodology (39) that directly addresses the identified gaps: (I) it mandates continuous post-update assessment to account for AI’s dynamic evolution through retraining; (II) it embeds radiologist involvement throughout all stages of development and testing; (III) it systematically accounts for data heterogeneity, volume, and multi-center representation; and (IV) it is specifically designed for real-world integration into healthcare systems like CMI programs.

It systematically evaluates functionality, calibration, real-world stability, and clinical efficacy across four iterative stages: (I) functional testing; (II) calibration testing; (III) prospective clinical and technical monitoring; and (IV) developer updates. While theoretically comprehensive, its practical application to mammography AI remains untested. This study demonstrates its real-world implementation, achieving substantial performance improvements and successful AI integration into the CMI program (40).

Objective

To ensure continuous quality control of the mammography AI solution during its implementation in the CMI system. We present this article in accordance with the DECIDE-AI reporting checklist (25) (available at https://qims.amegroups.com/article/view/10.21037/qims-2026-0864/rc).

Methods

Participants

The clinical base was the Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Department of Health. This single institution served as the data source, aggregating depersonalized mammography examinations referred from a total of 238 medical organizations, all of which contributed imaging data to the final analysis. This study represents a secondary analysis of digital mammography examinations previously collected from the broader imaging initiative registered at ClinicalTrials.gov (NCT04489992). The study focused on AI analysis of anonymized mammograms, without direct patient recruitment. Digital mammograms in craniocaudal (CC) and mediolateral oblique (MLO) views from women aged ≥18 years were included. For the retrospective phase (functional and calibration testing), inclusion required histological confirmation of breast cancer (target pathology). Low-quality exams were excluded retrospectively (e.g., incomplete breast visualization, improper exposure, artifacts). Prospectively (clinical and technical monitoring), no exclusions applied; the AI analyzed all mammograms from participating sites (continuous sample). This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.

AI system users were radiologists divided into experts and end-users. Expert radiologists annotated the test dataset and assessed AI outputs during prospective monitoring (detailed below). Inclusion criteria for experts: (I) >5 years’ experience in mammography subspecialty and (II) introductory training on AI in radiology, covering basic output requirements (Tables S1,S2) (41,42). End-user radiologists from participating sites performed second reads during prospective monitoring (with AI as first reader).

Data Anonymization

All included mammograms were anonymized per Digital Imaging and Communications in Medicine (DICOM) confidentiality profile attributes (43). Four anonymization methods (removal, cleaning, replacement, identifier generation) were applied to 433 DICOM tags per study. The method was documented in tag “[0x0012, 0x0063]”. Additionally, pseudonymization replaced original study IDs with randomly generated unique numbers, preserving data linkage while protecting patient privacy and preventing direct participant identification.

AI system

We evaluated Third Opinion Mammograms^®, a convolutional neural network-based AI system previously developed by Third Opinion Platform. This solution simultaneously performs computer-aided detection (CADe) to localize target findings and computer-aided diagnosis (CADx) for classifying them. Its clinical task was detecting and classifying breast cancer signs on mammograms, analyzing images only. The training dataset of approximately 4,000 mammograms was collected from several partner clinics of the Third Opinion Platform, ensuring broad population and ethnic diversity. The data were characterized by heterogeneity in patient age, Breast Imaging-Reporting and Data System (BI-RADS) categories, American College of Radiology (ACR) breast density, mammography equipment, and image quality [according to the Perfect, Good, Moderate, Inadequate (PGMI) criteria]. This diversity was designed to mitigate potential bias in the AI system. The ground truth labels for the training dataset were established through a consensus reading by three expert radiologists. The dataset consisted of both biopsy-proven cases and those diagnosed on imaging alone, which is representative of routine clinical practice. Initial version 1.0.0 comprised two modules: (I) calcifications segmentation using Unet++ architecture (44) with EfficientNet-B1 encoder (45); (II) mass, asymmetry, and architectural distortion segmentation via Mask2Former (46) with EfficientNet-B3 encoder (45). Internal testing yielded: area under the curve (AUC) =0.84, accuracy =0.81, sensitivity =0.87, specificity =0.78, F1 =0.75. Inputs were DICOM mammograms auto-converted to NumPy arrays via Pydicom, with preprocessing [slope/intercept, volume-of-interest look-up table (VOI LUT)/windowing, normalization]. Outputs included Digital Imaging and Communications in Medicine Structured Report (DICOM-SR) and AI-generated supplemental images (Figure 1). The example shows correct segmentation and classification of microcalcification clusters in the upper outer left breast quadrant (dashed turquoise contour). The software was registered with Russia’s Federal Service for Surveillance in Healthcare (Roszdravnadzor) under certificate No. 2022/16534. This regulatory approval confirms that the medical device has met national standards for safety, quality, and clinical efficacy, serving as the local equivalent of the European Conformité Européenne (CE) mark. See Appendix 1 for details.

Figure 1 Example of supplemental image series and text protocol generated by the artificial intelligence-based system.

Implementation

Objectives, outcome measures, and success criteria for the testing and monitoring stages are presented in Table S3 and detailed below.

Stage 1: functional testing

One expert radiologist verified core AI functions against Basic Requirements for AI Systems in Radiology. Basic functional requirements (Table S1) applied universally across radiology tasks, while basic diagnostic requirements were specific to mammography-based breast cancer detection (Table S2). Datasets for each implementation stage are described in Results section.

Stage 2: calibration testing

This stage evaluated the AI system’s core diagnostic performance against predefined success criteria (Table S3). Diagnostic metrics, including accuracy, sensitivity, specificity, and AUC, were calculated from the confusion matrix to assess model discrimination. Success criteria for this phase were: (I) AUC and accuracy >0.80; (II) relative deviation from developer-claimed metrics <10% (39). The minimum sufficient sample size of 100 studies was determined based on recommendations for statistical power in pilot studies of medical diagnostic systems (47).

Stage 3: technical and clinical monitoring

Unlike prior stages, monitoring was prospective: AI analyzed all mammograms from participating sites without exclusions. High volumes (>6,000 studies per month) precluded full expert review, so technical defects were automatically logged and classified as: (I) complete analysis failure; (II) partial failure (processing 1–3 of 4 standard images: bilateral CC and MLO views). These defects could result from AI errors, mammography equipment faults, or other technical or operator-related issues.

The clinical monitoring strategy employed a two-tiered approach to ensure both real-world applicability and rigorous performance evaluation. The first tier involved all end-user radiologists at participating sites using the AI system as a decision support tool during their routine practice. This phase was crucial for understanding general usability and integration into the clinical workflow. However, given the large number of participants and the subjective nature of individual feedback, a quantitative analysis of their diagnostic decisions was not suitable for providing an objective measure of the AI’s accuracy. Therefore, to achieve a robust and unbiased assessment of the AI’s performance, an independent expert radiologist implemented a second tier of validation. This involved the rigorous assessment of agreement with the AI’s outputs on a randomly selected sample of cases, using a four-point scale (agreement, incorrect assessment, false positive, false negative; see Table S4). Due to the limited availability of experts, this in-depth review was conducted only at the start and end of the prospective phase, as well as after each AI system update, with a controlled sample size of 80 cases per month (39,47).

Stage 4: AI system updates

Developers updated the AI system in two scenarios: (I) unsatisfactory testing or monitoring results; or (II) voluntary updates regardless of performance. Update effectiveness was evaluated as follows: group 1 updates triggered repeat testing or monitoring of the relevant stage; group 2 updates allowed continuation of ongoing testing or monitoring. Updates were also classified by change type: (I) architectural (changes to AI architecture or core functions); or (II) non-architectural (e.g., graphical user interface improvements). Architectural updates required repeat calibration testing for validation; non-architectural updates proceeded with routine testing or monitoring.

Stage 5: AI implementation in clinical practice

Implementation into routine clinical healthcare within the CMI program was permitted only after the system met predefined acceptance criteria for safe and effective deployment. These criteria, which were set to ensure sufficient diagnostic performance, reliability of claimed metrics, and technical robustness for real-world use, included (I) a diagnostic accuracy >80% (to ensure a baseline level of clinical utility); (II) a relative deviation from claimed performance metrics <10% (to guarantee model robustness and generalizability); (III) a technical defect rate <10% (to ensure operational stability); and (IV) a clinical score >80% (to reflect high end-user acceptability) (39). These thresholds represent operational implementation criteria specific to our study and healthcare system, rather than universally mandated international cutoffs. This approach is consistent with current best practices, which recognize that acceptance thresholds for AI tools should be tailored to the intended clinical role, local workflow, and real-world performance requirements (48-50). This framework is further supported by our previous study and by local methodological recommendations adopted in Moscow, which similarly emphasize predefined performance criteria for the clinical introduction of AI-based diagnostic systems (39). Post-implementation, AI performance was monitored monthly by comparing BI-RADS categories assigned by the AI system and end-user radiologists (see “Statistical analysis” section for details).

Outcomes

Primary outcomes included diagnostic accuracy metrics (accuracy, sensitivity, specificity, AUC) measured across seven calibration tests. Secondary outcomes included technical defect rates, clinical score dynamics, radiologist-AI agreement rates, including critical discrepancy rates.

Safety and errors

Patient safety was ensured throughout all study stages because: (I) the observational design involved no invasive procedures; (II) functional and calibration testing used retrospective data; and (III) during prospective monitoring, final clinical reads were performed exclusively by radiologists, with AI outputs clearly labeled “For research use only”. All stages systematically identified and mitigated potential AI errors.

Human factors

Human factors may have influenced clinical monitoring results, as a single expert radiologist’s assessment served as ground truth. No specific human factors logging was performed, as it was outside study scope.

Statistical analysis

The statistical analysis plan was tailored to the specific objectives of each implementation stage. For calibration testing (Stage 2), diagnostic performance metrics—accuracy, sensitivity, and specificity—were calculated as follows:

Accuracy=TP+TNTP+FP+TN+FN[1]

Sensitivity=TPTP+FN[2]

Specificity=TNTN+FP[3]

where TP, true positives; TN, true negatives; FP, false positives; FN, false negatives.

Receiver operating characteristic curves were generated by plotting sensitivity against the false positive rate (1 − specificity) across all probability thresholds; the optimal activation threshold was determined using Youden’s index (51). For prospective monitoring (stage 3), trends in technical defects (complete or partial analysis failures) were analyzed using linear regression to identify performance degradation over time. Agreement between the AI system and an independent expert radiologist was quantitatively assessed on a sampled subset of cases. This assessment involved scoring localization and interpretation accuracy. Mean localization and interpretation scores were calculated using the arithmetic mean. The overall clinical assessment score was derived from these means using the formula:

Clinicalassessmentscore=L+12×100%[4]

where L, mean localization score; I, mean interpretation score per study.

Post-implementation, AI performance was assessed monthly by comparing BI-RADS categories assigned by the AI system and end-user radiologists; each examination was classified into one of the following BI-RADS categories: BI-RADS 0 (incomplete assessment), BI-RADS 1 (negative), BI-RADS 2 (benign), BI-RADS 3 (probably benign), BI-RADS 4 (suspicious abnormality), BI-RADS 5 (highly suggestive of malignancy), or BI-RADS 6 (known biopsy-proven malignancy). The analysis was automated using data from all AI-processed mammograms for that month, and all critical discrepancies underwent expert radiologist review. This evaluation method is a limitation, as the gold standard for breast cancer diagnosis is histology. The agreement between the AI system and both end-user radiologists and the expert radiologist was quantitatively assessed by calculating Cohen’s kappa coefficient for each monitoring period.

All descriptive statistics were calculated in Microsoft Excel, and no specialized statistical software was required for the analyses presented in this study.

Results

Participants

The functional testing dataset included four women (mean age 61 years; range, 56–68 years). The calibration testing dataset comprised 100 women (mean age 63 years; range, 48–71 years), with a 50:50 class balance (“target pathology” vs. “no target pathology”). The technical monitoring dataset included 429,027 mammograms. Demographic and clinical characteristics of the technical monitoring dataset are shown in Table 1. The clinical monitoring dataset, a subset of the technical monitoring data, comprised 480 mammograms (80 per each of six monitoring months), with a 75:25 class balance. The CMI program monitoring dataset included 547,909 mammograms. All studies across datasets were acquired using mammography equipment from three manufacturers: General Electric, Fujifilm, and Medical Technologies Ltd.

A total of 336 radiologists participated: two expert radiologists annotated the functional and calibration datasets; one conducted functional and calibration testing; one performed clinical monitoring. Meanwhile, 334 end-user radiologists performed second reads during prospective monitoring.

Implementation

Stage 1: functional testing

Functional testing identified one critical noncompliance: one out of four mammography studies was processed incompletely. The protocol with findings was sent to developers. Post-refinement, no critical discrepancies remained.

Stage 2: calibration testing

The first two calibration tests failed to meet the predefined performance thresholds (0.80). The third test successfully satisfied these criteria (Figure 2, Table 2).

Figure 2 Receiver operating characteristic curves from the first (A), second (B), and third (C) calibration tests. AUC, area under the curve; CI, confidence interval.

Stage 3: technical and clinical monitoring

Monitoring lasted 2 years and 1 month. Technical defect rates exceeded 10% in only 1 of 25 months (group B defects: 11.10%; Figure 3A). Mean clinical score was 72.46; dynamics shown in Figure 3B.

Figure 3 Performance assessment of the artificial intelligence system during technical and clinical monitoring. (A) Dynamics of technical defects during technical monitoring. (B) Dynamics of clinical scores during clinical monitoring. AI, artificial intelligence.

Stage 4: AI system updates

During the monitoring period, six model updates were released (Table S5). Architectural updates (1st, 2nd, 5th, 6th) underwent repeat calibration testing (Figure 4, Tables 3,4).

Figure 4 Receiver operating characteristic curves from the fourth (A), fifth (B), sixth (C), and seventh (D) calibration tests. AUC, area under the curve; CI, confidence interval.

Stage 5: AI implementation in regional CMI program

In June 2023 (6 months post-testing initiation), AI mammography analysis was first integrated into Moscow’s territorial CMI program. CMI monitoring lasted 9 months. Mean critical BI-RADS discrepancies between AI and radiologists was 0.08% (Figure 5A). Expert radiologist agreement with AI averaged 32.2% (range, 11.1–42.9%; Figure 5B); with end-users, 49.9% (27.7–68.2%; Figure 5C). An assessment using Cohen’s kappa coefficient for the agreement between the end-user radiologist and the expert radiologist (based on cases of significant discrepancies) revealed, on average, a lack of agreement (κ=0.08). Concordance varied substantially across individual periods, with kappa values ranging from −0.45 to 1.00 (Table S6).

Figure 5 Comparison of the artificial intelligence system with expert and end-user radiologists during compulsory medical insurance program implementation. (A) Dynamics of critical discrepancies between the artificial intelligence-based system and end-user radiologists during Compulsory Medical Insurance program implementation. (B) Dynamics of expert radiologist agreement with the artificial intelligence-based system. (C) Dynamics of expert radiologist agreement with end-user radiologists. AI, artificial intelligence.

Discussion

This study evaluates a continuous quality control methodology for a mammography AI system during its implementation into the CMI program. Results demonstrate that structured, ongoing testing—coupled with close developer-radiologist collaboration—plays a pivotal role in enhancing mammography AI performance. Applying this lifecycle-based testing and monitoring framework enabled successful integration into Moscow’s territorial CMI program, marking a critical step toward large-scale AI adoption in public healthcare system.

Initial functional testing identified a critical discrepancy in processed mammography views, subsequently resolved by developers. This underscores the value of rigorous pre-clinical functional verification. Subsequent calibration tests revealed initially subthreshold performance (AUC 0.83, accuracy 0.77, sensitivity 0.84, specificity 0.70), consistent with similar studies where early AI assessments uncover refinement needs (52,53). The highest achieved AUC of 0.92 falls within the upper range of literature values (0.71–0.96) reported in external validation studies of mammography AI, confirming its potential for high diagnostic accuracy after rigorous calibration and validation (10,12,19).

The implemented continuous quality control methodology demonstrably enhanced the performance of both functional AI-CADe and AI-CADx modules. For the lesion detection task (AI-CADe), the system’s sensitivity was optimized to a peak of 0.88 (range, 0.78–0.98), indicating a high capability for identifying suspicious lesions. Concurrently, for the diagnostic classification task (AI-CADx), the software yielded a maximum accuracy of 0.90 (range, 0.83–0.96) and specificity of 0.91 (range, 0.83–0.98), confirming its efficacy in minimizing false-positive findings and accurately characterizing detected abnormalities.

The 24-month technical and clinical monitoring demonstrated stable AI system performance, with technical error rates remaining below threshold (<10%) in nearly all observation periods. Peaks in technical defects likely stemmed from factors such as suboptimal mammography equipment settings, Picture Archiving and Communication System (PACS) failures, and vendor server issues. The observed upward trend in clinical scores aligns with prospective mammography AI studies showing positive clinical impact (49,54). Gradual improvements in calibration testing post-AI updates highlight the importance of iterative optimization and continuous monitoring for mammography AI.

Our findings align with prior research on iterative testing and systematic updates improving mammography AI performance. McKinney et al. (55) reported an AI system achieving AUC 0.889 upon external validation, underscoring rigorous evaluation’s importance. Badal et al. (56) emphasized that AI algorithms require ongoing assessment, adaptation, and iterative refinement to meet evolving clinical needs while incorporating new scientific insights. Mökander et al. (57) noted that post-registration surveillance minimizes unforeseen AI errors, ensuring mammography screening safety and efficacy. Several authors advocate a systematic AI implementation approach: initial retrospective evaluation on large representative datasets followed by prospective real-world monitoring (58-60). However, some studies report no performance gains from AI updates, contrasting our results (61).

Another promising direction, paralleling software development lifecycle approaches, is continual learning for AI systems. Wu et al. (62) reviewed continual learning approaches, highlighting class-, task-, and domain-based strategies. Li et al. (63) showed continual learning improved radiomic AI performance for breast cancer diagnosis on magnetic resonance imaging. Finally, Högberg et al. (64) reported that continual learning was the primary factor supporting radiologists’ trust in mammography AI.

Support for intended use

Our findings demonstrate the significant clinical potential of the AI system, exhibiting a robust discriminative ability with an AUC ranging from 0.78 to 0.92. This performance is comparable to the diagnostic accuracy reported for human radiologists in similar tasks (reported AUC range, 0.71–0.95) (65,66). Following the Moscow CMI implementation, the mean rate of critical discrepancies between AI and radiologist assessments of BI-RADS categories was 0.08% (range, 0.01–0.18%). Notably, inter-reader agreement between the expert and end-user radiologists reached only 49.9%, which aligns with established levels of reader variability and highlights the critical need for clinical decision support systems. Further evidence from our previous works reinforces these conclusions. The average accuracy of three AI systems across 81,895 real-world mammograms was 76.9% (range, 62.8–95.8%) (67). At a fixed sensitivity threshold of 100%, the AI system demonstrated a clinically significant error rate of 0.02%, representing a statistically significant reduction compared to the mean error rate of participating radiologists (68). Finally, the application of the AI algorithm allowed for a 66.4% reduction in the duration of double reading (P<0.0001) (69). These findings align with the current literature, suggesting that AI performance is statistically non-inferior to that of human experts.

Safety and errors

Our analysis revealed various AI system errors related to technical and clinical performance, as well as data pipeline stability within the PACS environment. At each study phase, identified errors were communicated to developers via a structured reporting system, prompting the release of corrective updates. No adverse clinical outcomes associated with the AI system were recorded, further confirming its safety profile. Nevertheless, ongoing monitoring remains essential upon deployment of medical AI systems due to their dynamic nature and potential for updates (70).

Strengths and limitations

A key strength of this study is the comprehensive AI evaluation, encompassing functional and calibration testing alongside 24-month of prospective technical and clinical monitoring. The conclusive results of this multi-stage, long-term evaluation significantly strengthen the evidence supporting the mammography AI system’s reliability. Iterative algorithm updates, validated through repeat calibration testing on histologically confirmed data, demonstrate progressive diagnostic performance improvements. Additional advantages include involvement of both expert and end-user radiologists, ensuring multifaceted assessment of clinical applicability and usability.

This study has several key limitations. First, a primary limitation of this study is the relatively small calibration dataset (n=100), which may limit the generalizability of the findings. The in vivo calibration tests, while informative, may underestimate AI performance in real-world clinical scenarios, such as rare or complex cases. Second, although data were aggregated through the Moscow Reference Center for Diagnostic Imaging—which oversees 238 facilities and ensures population diversity—the study’s single-center design remains a critical limitation. Consequently, the findings may not be fully generalizable to other healthcare systems and geographic regions. Third, the limited pool of expert radiologists precluded a robust assessment of interobserver variability and introduced potential observer bias. Furthermore, the study design precludes the ability to isolate the effect of specific model updates from concurrent changes in data distribution and workflow. While calibration testing followed architectural updates, the holistic nature of the methodology means that overall performance gains cannot be exclusively attributed to those updates. Finally, the study lacked a formal assessment of the AI system’s impact on patient-centered clinical outcomes, organizational workflows, or economic efficiency. While successful integration into the CMI program was achieved, these aspects warrant separate investigation.

Future directions and global applicability

The underlying operational principles of this lifecycle-based testing and monitoring framework may be transferable to other healthcare systems. While acceptance criteria must align with local regulations, continuous monitoring is paramount for risk mitigation. Specifically, it addresses performance drift following model updates and ensures the sustained clinical utility of AI within routine breast cancer screening. Future studies should further assess the reproducibility of this framework in other settings and clarify how monitoring results can guide model recalibration, update management, and clinical deployment decisions. Finally, future research must prioritize the organizational and clinical implications of AI integration by assessing patient-centered outcomes, workflow efficiency, and resource allocation—areas that exceeded the scope of this work.

Conclusions

This study demonstrates substantial improvements in mammography AI performance achieved through close collaboration between developers and radiologists. This multidisciplinary approach enabled successful integration of the system into Moscow’s regional population-based screening program. The proposed lifecycle-based methodology proved effective for continuous model monitoring and validation within routine clinical practice, encompassing rigorously defined diagnostic requirements, systematic error analysis, and iterative refinement based on prospective feedback. Large-scale deployment across the regional network validates the clinical utility of modern algorithms for high-volume interpretation while alleviating radiologist workload without compromising diagnostic accuracy. Although this work confirms the technical efficacy of mammography AI, methodological constraints regarding long-term patient outcomes, economic viability, and workflow integration necessitate further investigation prior to widespread adoption. Ultimately, sustained dialogue among developers, clinicians, and regulators will be critical to translating these gains into measurable improvements in patient care and establishing new standards for diagnostic imaging.

Acknowledgments

None.

Footnote

Reporting Checklist: The authors have completed the DECIDE-AI reporting checklist. Available at https://qims.amegroups.com/article/view/10.21037/qims-2026-0864/rc

Funding: This work was supported by the Moscow Health Care Department, as a part of the research and development effort titled “Artificial intelligence for synchronous evaluation of images and EHR (multimodal AI)”, in accordance with the Order No. 1213 dated November 27, 2025 “On approval of state assignments funded by means of allocations from the budget of the city of Moscow to the state budgetary (autonomous) institutions subordinate to the Moscow Health Care Department, for 2026 and the planned period of 2027 and 2028”.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://qims.amegroups.com/article/view/10.21037/qims-2026-0864/coif). All authors report that this work was supported by the Moscow Health Care Department. I.N., Arina Varlamova, V.B., and E.S. are employees of the Third Opinion Platform and developers of the AI system evaluated in this study. They contributed to providing information about the AI system but were not involved in conducting the study, data analysis, or interpretation of results. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Alowais SA, Alghamdi SS, Alsuhebany N, Alqahtani T, Alshaya AI, Almohareb SN, Aldairem A, Alrashed M, Bin Saleh K, Badreldin HA, Al Yami MS, Al Harbi S, Albekairy AM. Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Med Educ 2023;23:689.
2. Siegel RL, Giaquinto AN, Jemal A. Cancer statistics, 2024. CA Cancer J Clin 2024;74:12-49.
3. Smetherman D, Golding L, Moy L, Rubin E. The Economic Impact of AI on Breast Imaging. J Breast Imaging 2022;4:302-8.
4. Marmot MG, Altman DG, Cameron DA, Dewar JA, Thompson SG, Wilcox M. The benefits and harms of breast cancer screening: an independent review. Br J Cancer 2013;108:2205-40.
5. Parikh JR, Sun J, Mainiero MB. Prevalence of Burnout in Breast Imaging Radiologists. J Breast Imaging 2020;2:112-8.
6. Shamir SB, Sasson AL, Margolies LR, Mendelson DS. New Frontiers in Breast Cancer Imaging: The Rise of AI. Bioengineering (Basel) 2024;11:451.
7. Park GE, Kang BJ, Kim SH, Mun HS. Efficacy of Mammographic Artificial Intelligence-Based Computer-Aided Detection in Predicting Pathologic Complete Response to Neoadjuvant Chemotherapy. Life (Basel) 2024;14:1449.
8. Anyfantis D, Koutras A, Apostolopoulos G, Christoyianni I. Revealing occult malignancies in mammograms through GAN-driven breast density transformation. Electronics 2024;13:4826.
9. Salehi AW, Khan S, Gupta G, Alabduallah BI, Almjally A, Alsolai H, Siddiqui T, Mellit A. A study of CNN and transfer learning in medical imaging: advantages, challenges, future scope. Sustainability 2023;15:5930.
10. Yoon JH, Strand F, Baltzer PAT, Conant EF, Gilbert FJ, Lehman CD, Morris EA, Mullen LA, Nishikawa RM, Sharma N, Vejborg I, Moy L, Mann RM. Standalone AI for Breast Cancer Detection at Screening Digital Mammography and Digital Breast Tomosynthesis: A Systematic Review and Meta-Analysis. Radiology 2023;307:e222639.
11. Freeman K, Geppert J, Stinton C, Todkill D, Johnson S, Clarke A, Taylor-Phillips S. Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. BMJ 2021;374:n1872.
12. Liu J, Lei J, Ou Y, Zhao Y, Tuo X, Zhang B, Shen M. Mammography diagnosis of breast cancer screening through machine learning: a systematic review and meta-analysis. Clin Exp Med 2023;23:2341-56.
13. Hickman SE, Woitek R, Le EPV, Im YR. Mouritsen Luxhøj C, Aviles-Rivero AI, Baxter GC, MacKay JW, Gilbert FJ. Machine Learning for Workflow Applications in Screening Mammography: Systematic Review and Meta-Analysis. Radiology 2022;302:88-104.
14. Goisauf M, Cano Abadía M. Ethics of AI in Radiology: A Review of Ethical and Societal Implications. Front Big Data 2022;5:850383.
15. Najjar R. Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging. Diagnostics (Basel) 2023;13:2760.
16. Zeng A, Houssami N, Noguchi N, Nickel B, Marinovich ML. Frequency and characteristics of errors by artificial intelligence (AI) in reading screening mammography: a systematic review. Breast Cancer Res Treat 2024;207:1-13.
17. International Organization for Standardization (ISO). Quality management systems — requirements (ISO 9001:2015). Geneva: ISO; 2015. Available online: https://www.iso.org/standard/62085.html. Accessed 1 Mar 2026.
18. Jayakumar S, Sounderajah V, Normahani P, Harling L, Markar SR, Ashrafian H, Darzi A. Quality assessment standards in artificial intelligence diagnostic accuracy systematic reviews: a meta-research study. NPJ Digit Med 2022;5:11.
19. Branco PESC, Franco AHS, de Oliveira AP, Carneiro IMC, de Carvalho LMC, de Souza JIN, Leandro DR, Cândido EB. Artificial intelligence in mammography: a systematic review of the external validation. Rev Bras Ginecol Obstet 2024;46:e-rbgo71.
20. Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers. Korean J Radiol 2019;20:405-10.
21. Yu AC, Mohajer B, Eng J. External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review. Radiol Artif Intell 2022;4:e210064.
22. van Leeuwen KG, Schalekamp S, Rutten MJCM, van Ginneken B, de Rooij M. Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol 2021;31:3797-804.
23. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med 2015;162:55-63.
24. Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AKSPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med 2020;26:1364-74.
25. Vasey B, Novak A, Ather S, Ibrahim M, McCulloch P. DECIDE-AI: a new reporting guideline and its relevance to artificial intelligence studies in radiology. Clin Radiol 2023;78:130-6.
26. Cerdá-Alberich L, Solana J, Mallol P, Ribas G, García-Junco M, Alberich-Bayarri A, Marti-Bonmati L. MAIC-10 brief quality checklist for publications using artificial intelligence and medical images. Insights Imaging 2023;14:11.
27. Tejani AS, Klontzas ME, Gatti AA, Mongan JT, Moy L, Park SH, Kahn CE JrCLAIM 2024 Update Panel. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiol Artif Intell 2024;6:e240300.
28. Filice RW, Mongan J, Kohli MD. Evaluating Artificial Intelligence Systems to Guide Purchasing Decisions. J Am Coll Radiol 2020;17:1405-9.
29. Tadavarthi Y, Vey B, Krupinski E, Prater A, Gichoya J, Safdar N, Trivedi H. The State of Radiology AI: Considerations for Purchase Decisions and Current Market Offerings. Radiol Artif Intell 2020;2:e200004.
30. Tanguay W, Acar P, Fine B, Abdolell M, Gong B, Cadrin-Chênevert A, Chartrand-Lefebvre C, Chalaoui J, Gorgos A, Chin AS, Prénovault J, Guilbert F, Létourneau-Guillon L, Chong J, Tang A. Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework. Can Assoc Radiol J 2023;74:326-33.
31. Boverhof BJ, Redekop WK, Bos D, Starmans MPA, Birch J, Rockall A, Visser JJ, Radiology AI. Deployment and Assessment Rubric (RADAR) to bring value-based AI into radiological practice. Insights Imaging 2024;15:34.
32. Pianykh OS, Langs G, Dewey M, Enzmann DR, Herold CJ, Schoenberg SO, Brink JA. Continuous Learning AI in Radiology: Implementation Principles and Early Applications. Radiology 2020;297:6-14.
33. Al Mohammad B, Aldaradkeh A, Gharaibeh M, Reed W. Assessing radiologists' and radiographers' perceptions on artificial intelligence integration: opportunities and challenges. Br J Radiol 2024;97:763-9.
34. Vasilev YA, Vladzymyrskyy AV, Alymova YA, Akhmedzyanova DA, Blokhin IA, Romanenko MO, Seradzhi SR, Suchilova MM, Shumskaya YF, Reshetnikov RV. Development and Validation of a Questionnaire to Assess the Radiologists' Views on the Implementation of Artificial Intelligence in Radiology (ATRAI-14). Healthcare (Basel) 2024;12:2011.
35. Flory MN, Napel S, Tsai EB. Artificial Intelligence in Radiology: Opportunities and Challenges. Seminars in Ultrasound CT and MRI 2024;45:152-60.
36. Debnath J. Radiology in the era of artificial intelligence (AI): Opportunities and challenges. Med J Armed Forces India 2023;79:369-72.
37. Vasilev Y, Vladzymyrskyy A, Omelyanskaya O, Blokhin I, Kirpichev Y, Arzamasov K. AI-Based CXR First Reading: Current Limitations to Ensure Practical Value. Diagnostics (Basel) 2023;13:1430.
38. Goh SSN, Ng QX, Chan FJH, Goh RSJ, Jagmohan P, Ali SH, Koh GCH. Radiologists' Perspectives on AI Integration in Mammographic Breast Cancer Screening: A Mixed Methods Study. Cancers (Basel) 2025;17:3491.
39. Vasiliev YA, Vlazimirsky AV, Omelyanskaya OV, Arzamasov KM, Chetverikov SF, Rumyantsev DA, Zelenova MA. Methodology for testing and monitoring artificial intelligence-based software for medical diagnostics. Digit Diagn 2023;4:252-67.
40. Experiment on the introduction of artificial intelligence technologies. Available online: https://telemedai.ru/en/proekty/eksperiment-po-vnedreniyu-tehnologij-iskusstvennogo-intellekta. Accessed 1 Mar 2026.
41. Baseline functional and diagnostic requirements for AI services. Available online: https://mosmed.ai/en/ai/docs/. Accessed 1 Mar 2026.
42. Baseline diagnostic requirements AI service performance. Available online: https://mosmed.ai/en/ai/docs/. Accessed 1 Mar 2026.
43. DICOM Standards Committee. Application level confidentiality profile attributes. Available online: https://dicom.nema.org/medical/dicom/current/output/html/part15.html#table_E.1-1. Accessed 1 Mar 2026.
44. Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J. UNet++: a nested U-Net architecture for medical image segmentation. Deep Learn Med Image Anal Multimodal Learn Clin Decis Support 2018;11045:3-11.
45. Tan M, Le QV. EfficientNet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning; 2019 Jun 9-15; Long Beach, CA. PMLR; 2019:6105-14.
46. Cheng B, Misra I, Schwing AG, Kirillov A, Girdhar R. Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2022:1290-9.
47. Bobrovskaya TM, Vasiliev YA, Nikitin NY, Vladzymyrskyy AV, Omelyanskaya OV, Chetverikov SF, Arzamasov KM. Sample size for assessing a diagnostic accuracy of AI-based software in radiology. Siberian Journal of Clinical and Experimental Medicine 2024;39:188-98.
48. Brady AP, Allen B, Chong J, Kotter E, Kottler N, Mongan J, Oakden-Rayner L, Dos Santos DP, Tang A, Wald C, Slavotinek J. Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA. Insights Imaging 2024;15:16.
49. Chang YW, Ryu JK, An JK, Choi N, Park YM, Ko KH, Han K. Artificial intelligence for breast cancer screening in mammography (AI-STREAM): preliminary analysis of a prospective multicenter cohort study. Nat Commun 2025;16:2248.
50. Schiaffino S, Bernardi D, Healy N, Marino MA, Romeo V, Sechopoulos I, Mann RM, Pinker K. ESR Essentials: artificial intelligence in breast imaging-practice recommendations by the European Society of Breast Imaging. Eur Radiol 2026;36:1909-18.
51. Ruopp MD, Perkins NJ, Whitcomb BW, Schisterman EF. Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection. Biom J 2008;50:419-30.
52. Meijerink LM, Dunias ZS, Leeuwenberg AM, de Hond AAH, Jenkins DA, Martin GP, Sperrin M, Peek N, Spijker R, Hooft L, Moons KGM, van Smeden M, Schuit E. Updating methods for artificial intelligence-based clinical prediction models: a scoping review. J Clin Epidemiol 2025;178:111636.
53. Feng J, Phillips RV, Malenica I, Bishara A, Hubbard AE, Celi LA, Pirracchio R. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. NPJ Digit Med 2022;5:66.
54. Dembrower K, Crippa A, Colón E, Eklund M, Strand F. ScreenTrustCAD Trial Consortium. Artificial intelligence for breast cancer detection in screening mammography in Sweden: a prospective, population-based, paired-reader, non-inferiority study. Lancet Digit Health 2023;5:e703-11.
55. McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, et al. International evaluation of an AI system for breast cancer screening. Nature 2020;577:89-94.
56. Badal K, Lee CM, Esserman LJ. Guiding principles for the responsible development of artificial intelligence tools for healthcare. Commun Med (Lond) 2023;3:47.
57. Mökander J, Axente M, Casolari F, Floridi L. Conformity Assessments and Post-market Monitoring: A Guide to the Role of Auditing in the Proposed European AI Regulation. Minds Mach (Dordr) 2022;32:241-68.
58. Goh S, Goh RSJ, Chong B, Ng QX, Koh GCH, Ngiam KY, Hartman M. Challenges in Implementing Artificial Intelligence in Breast Cancer Screening Programs: Systematic Review and Framework for Safe Adoption. J Med Internet Res 2025;27:e62941.
59. Le EPV, Wang Y, Huang Y, Hickman S, Gilbert FJ. Artificial intelligence in breast imaging. Clin Radiol 2019;74:357-66.
60. Taylor-Phillips S, Seedat F, Kijauskaite G, Marshall J, Halligan S, Hyde C, Given-Wilson R, Wilkinson L, Denniston AK, Glocker B, Garrett P, Mackie A, Steele RJ. UK National Screening Committee's approach to reviewing evidence on artificial intelligence in breast cancer screening. Lancet Digit Health 2022;4:e558-65.
61. Taib AG, James JJ, Partridge GJW, Chen Y. Keeping AI on Track: Regular monitoring of algorithmic updates in mammography. Eur J Radiol 2025;187:112100.
62. Wu X, Xu Z, Tong RK. Continual learning in medical image analysis: A survey. Comput Biol Med 2024;182:109206.
63. Li H, Whitney HM, Ji Y, Edwards A, Papaioannou J, Liu P, Giger ML. Impact of continuous learning on diagnostic breast MRI AI: evaluation on an independent clinical dataset. J Med Imaging (Bellingham) 2022;9:034502.
64. Högberg C, Larsson S, Lång K. Engaging with artificial intelligence in mammography screening: Swedish breast radiologists' views on trust, information and expertise. Digit Health 2024;10:20552076241287958.
65. Lee SE, Hong H, Kim EK. Diagnostic performance with and without artificial intelligence assistance in real-world screening mammography. Eur J Radiol Open 2024;12:100545.
66. Yilmaz E, Seker ME, Guldogan N, Turk EB, Erdemli S, Koyluoglu YO, Sancak SN, Aribal E. Clinical Application of AI in Mammography: Insights from a Prospective Study. Acad Radiol 2025;32:5016-27.
67. Vasilev Y, Mayorova A, Rumyantsev D, Semenov S, Bobrovskaya T, Pestrenin L, Erizhokov R, Vladzymyrskyy A, Omelyanskaya O, Arzamasov K. Comparison of artificial intelligence (AI) services for Breast Imaging-Reporting and Data System (BI-RADS) classification on mammograms. Quant Imaging Med Surg 2026;16:311.
68. Vasilev YuA, Arzamasov KM, Vladzymyrskyy AV, et al. Autonomous artificial intelligence for sorting the results of preventive radiological studies on the example of mammography. Profilakticheskaya Meditsina 2024;27:23-9.
69. Kudryavtsev ND, Kozhikhina DD, Goncharova IV, Shulkin IM, Sharova D, Arzamasov KM, Vladzymirskyy AV. The impact of artificial intelligence on double reading of mammograms. Rus J Prev Med 2024;27:32-7.
70. Vasilev YA, Vladzymyrskyy AV, editors. Artificial intelligence in radiology: per aspera ad astra. Moscow: Izdatelskie Resheniya; 2025:493.