LLM Judges Focus on Presence, Not Absence, Citing Omission Blindness in AI Clinical Notes
| Source: HN | Original article
Large language models reviewing AI-generated clinical notes often confirm present information while failing to detect missing data, a bias known as omission blindness.
A new pre‑print by Sebastian Fox and three co‑authors uncovers a systematic blind spot in the way large language models (LLMs) are used as judges for AI‑generated clinical notes. The paper, titled **“LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It,”** introduces OmissionBench, a benchmark that tests whether an LLM judge can spot information that a note fails to include.
The authors find that judges are highly reliable at confirming that a note contains a claimed piece of information, but they perform at near‑chance levels when asked to verify that something is **absent**. Across eight different judge designs evaluated on a single clinical note, none retained useful signal for omission detection. The study also identifies a set of techniques that can restore the ability to catch missing content, and the authors have released the supporting code on GitHub.
Why it matters: Clinical documentation is a safety‑critical domain where omitted findings can lead to misdiagnosis or treatment errors. Many AI development pipelines now rely on LLM‑as‑a‑judge to automate quality control, assuming the model can flag both false inclusions and dangerous gaps. If judges routinely miss omissions, flawed notes may be approved without human review, amplifying the risk of downstream clinical harm. This echoes concerns raised in our earlier coverage of benchmark reliability (BenchMIRT, 2026‑09‑02), highlighting that not all evaluation metrics capture the nuances needed for high‑stakes applications.
What to watch next: The authors’ recovery methods—ranging from prompting strategies to hybrid deterministic checks—will likely be tested in real‑world health AI deployments. Follow‑up work may explore integrating OmissionBench into existing evaluation pipelines, extending the analysis to other medical writing tasks, and assessing whether the blind spot persists in newer, larger models. Industry groups and regulators could also adopt the benchmark as a sanity check before authorising AI‑generated clinical documentation.
Sources
Back to AIPULSEN