High-Confidence Discordance Among Large Language Models in Acute Myeloid Leukemia Gray-Zone Decision-Making
Materials and Methods/Case Presentation/Objective: Twenty single-step, multiple-choice AML vignettes were evaluated by ChatGPT, Claude, Gemini, and OpenEvidence. Each model completed three independent runs per vignette, generating 240 responses. Repeated runs were aggregated using prespecified rules: three identical responses were classified as stable, while a two-of-three valid majority determined the final answer. The primary endpoint was inter-model agreement measured using Gwet's AC1. Secondary endpoints included Fleiss' κ, pairwise raw agreement, complete consensus, response-pattern distribution, within-model repeated-run reliability, and model-reported confidence. Wilson 95% confidence intervals were used for proportions. Agreement coefficients and confidence summaries were estimated using 10,000 whole-vignette clustered bootstrap resamples. A sensitivity analysis recalculated confidence using only runs contributing to the final aggregated answer.
Results/Description/Main Outcome Measure(s):
All 240 expected responses and confidence values were complete, and all 80 model-vignette combinations yielded valid final responses. Across the 20 vignettes, inter-model agreement was substantial by Gwet's AC1 (0.611; 95% bootstrap CI, 0.425-0.793) and moderate by Fleiss' κ (0.506; 95% bootstrap CI, 0.270-0.708). Mean pairwise agreement was 69.2%, ranging from 60.0% to 75.0%. Complete four-model consensus occurred in only 10 of 20 vignettes (50.0%; 95% CI, 29.9-70.1). Among the remaining vignettes, 6 had a 3-1 split, 1 had a 2-2 split, and 3 produced a 2-1-1 pattern. High-confidence disagreement was present in 6 of 10 discordant vignettes (60.0%; 95% CI, 31.3-83.2) and was unchanged in the confidence sensitivity analysis. Within-model reliability varied substantially, with Gwet's AC1 ranging from 0.679 to 1.000. Mean model-reported confidence also varied widely, from 69.6% to 93.7%.
Conclusion(s).: Across AML gray-zone scenarios, contemporary LLMs demonstrated only moderate-to-substantial concordance and reached complete consensus in half of cases. Most importantly, discordant recommendations frequently persisted despite high model-reported confidence, indicating that apparent certainty may not reliably identify when an LLM's recommendation requires additional scrutiny. These findings support disease-specific benchmarking that evaluates reproducibility, cross-model disagreement, and uncertainty recognition rather than factual accuracy alone. Because no physician reference standard was included, this study measures model concordance rather than clinical correctness; blinded leukemia-expert adjudication is required to determine whether disagreement reflects legitimate clinical equipoise or potentially unsafe reasoning.
Trademarked Items:
References:
