MedExpert
A clinician-annotated benchmark for evaluating the factuality and completeness of medical chatbots.
MedExpert is an expert-annotated dataset for evaluating medical chatbots on two safety-critical axes: factuality and completeness (omission detection).
- 540+ clinician-annotated question–response pairs from high-risk specialties (prenatal care and young-adult mental health), plus 32 dual-annotated pairs.
- 100+ unique questions authored by clinicians, each answered by 5 open-source LLMs (Llama-2 7B, Llama-3.3 70B, OLMo-2 13B, Gemma-2 27B, OpenBioLLM-70B).
- Annotated by 8 practicing clinicians (MDs, residents, LCSW), with severity ratings for every factuality error and omission.
References
2025
- MedExpert: An Expert-Annotated Dataset for Medical Chatbot EvaluationIn Machine Learning for Health 2025, 2025