Data Annotation: Rubrics, Interface & Processing
End-to-end human-data infrastructure for expert-grade evaluation.
High-quality evaluation data is only as good as the pipeline that produces it. I own the full expert annotation stack, the human-data infrastructure that frontier labs and AI teams increasingly depend on:
- Rubrics. Designed and iterated evaluation rubrics with domain experts across 5 versions, leading 15+ calibration sessions that raised raw inter-annotator agreement ~20%.
- Annotation interface. Built the clinician annotation tool (John Snow Labs + AWS) used by 8 practicing clinicians (MDs, residents, LCSW).
- Eval interfaces (HTML). Built interfaces that let the engineering team triage and clean system detections (TP/FP/FN/TN, with filtering and anonymization) before clinician review, cutting clinician annotation load ~30%.
- Processing. Engineered the pipeline that turns raw expert annotations into clean, analysis-ready evaluation data.