Abstract:Biomedical retrieval-augmented large language models (LLMs) often face evidence that is incomplete, misleading, or internally contradictory, yet evaluation usually emphasizes answer accuracy under helpful context rather than reliability under conflict. Using HealthContradict, we evaluate six open-weight LLMs under five controlled evidence conditions: no retrieved context, correct-only context, incorrect-only context, and two mixed conditions containing both correct and contradictory documents in opposite orders. In this conflicting-evidence order contrast, where the same two documents are both present and only their order is reversed, accuracy drops for every model and 11.4%--25.2% of predictions flip. To support abstention in these difficult cases, we also evaluate a conflict-aware abstention score that combines model confidence with a detector of evidence conflict. In the two hardest conditions, this score improves selective accuracy over confidence-only, with mean gains of 7.2--33.4 points in incorrect-only (`IC') and 3.6--14.4 points in incorrect-first conflicting (`ICC') conditions across 75%, 50%, and 25% coverage. These results show that conflicting biomedical evidence is both an uncertainty and robustness problem and motivate evaluation and abstention methods that explicitly account for evidence disagreement.
| Comments: | Accepted by BioNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.14115 [cs.CL] |
| (or arXiv:2605.14115v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.14115 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yikun Han [view email]
[v1]
Wed, 13 May 2026 21:02:24 UTC (1,593 KB)