EXTERNAL TESTING OF A FROZEN PUBLIC VISION TRANSFORMER FOR BREAST ULTRASOUND CLASSIFICATION ON BUS-UCLM: MALIGNANT SENSITIVITY, CALIBRATION, AND CONFIDENCE-BASED ABSTENTION

Publicly released breast ultrasound artificial intelligence models may not retain their development performance when transferred unchanged to independently acquired data. This study externally tested the frozen three-class (benign, malignant, normal) Vision Transformer Parveshiiii/breast-cancer-detector, reported as trained on BUSI-derived data, on BUS-UCLM without retraining, fine-tuning, recalibration, or threshold optimization. The study followed a prespecified protocol frozen before full external inference. The analysis included 683 ultrasound images from 38 patients, including 90 malignant images; malignant sensitivity under the model's frozen decision rule was the primary outcome. Sensitivity was 43.3% (95% CI 24.2–61.4%) and specificity was 92.2% (87.9–95.8%), while malignant-score ROC-AUC was 0.785 (0.706–0.865), indicating moderate discrimination despite poor sensitivity at the frozen operating rule. Fifty-one malignant images were missed across 14 patients; 48/51 (94.1%) were classified as benign, and 27/51 (52.9%) had maximum-softmax confidence ≥80%. Overall three-class accuracy was 45.8%, whereas mean maximum-softmax confidence was 79.6%; Expected Calibration Error was 0.339 and multiclass Brier score was 0.808, consistent with marked overconfidence. At a 0.90 confidence-acceptance threshold, only 232/683 images (34.0%) were retained, accepted accuracy was 61.2%, and 16 malignant false negatives remained accepted. The model therefore showed limited external performance on BUS-UCLM, with low malignant sensitivity, miscalibration, and persistent high-confidence malignant misses. These findings highlight the value of reporting operating-point performance, calibration, patient-clustered confidence intervals, and selective-classification trade-offs alongside discrimination in external medical-imaging AI evaluation.