Director of AI Evaluation

Responsibilities

• Define and enforce quality standards for all high-value AI initiatives, including internally built and vendor-provided systems
• Manage and develop a multidisciplinary team of data scientists and senior analysts
• Own pre-production validation and live production monitoring methodologies
• Provide technical guidance on validation studies, equity audits, and monitoring plans
• Track AI system performance, user adoption, and clinical outcomes against baselines
• Monitor equity and disparate impact across subgroups

Requirements

• Minimum of 8 years of related work experience
• Minimum of 3 years of managerial or supervisory experience
• Strong foundation in experimental design and causal inference
• Hands-on experience with model evaluation in production settings
• Experience evaluating LLMs, generative AI, or complex ML systems
• Fluency in Python and SQL
• Experience evaluating fairness and equity in ML systems
• Bachelor's degree in a related field

Preferred Qualifications

• Experience in healthcare, clinical, or regulated industries

Benefits

• Healthcare benefits including vision, dental, and domestic partner coverage from day one

About the Company

Geisinger is a healthcare system serving more than 1 million people, comprising nine hospital campuses, a health plan, two research centers, and the Geisinger Commonwealth School of Medicine.