Tech Lead Data Scientist, AI Evaluation & Monitoring

Responsibilities

• Set the technical direction for AI evaluation across a large portfolio of production systems
• Design and operate the evaluation toolkit, including LLM-as-Judge frameworks, golden sets, and simulation harnesses
• Provide hands-on guidance to program teams on validation studies, equity audits, and monitoring plans
• Translate ambiguous failure modes into concrete, measurable production monitoring metrics
• Provide technical leadership, design review, and mentorship to a team of data analysts

Requirements

• 6+ years of experience in data science, statistics, ML engineering, or applied quantitative research
• Strong foundation in experimental design and causal inference
• Hands-on experience designing and running model evaluation studies in production settings
• Experience evaluating LLM, generative AI, or complex ML systems
• Proficiency in Python and SQL and comfort with cloud-native data environments
• Experience with fairness and equity evaluation for ML systems
• Ability to provide technical leadership without formal people-management authority

Preferred Qualifications

• MS or PhD in a quantitative field
• Experience in healthcare, clinical, or regulated industries

Benefits

• Healthcare benefits including vision and dental from day one
• Domestic partner benefits

About the Company

Geisinger is a healthcare system serving more than 1 million people, comprising nine hospital campuses, a health plan, two research centers, and the Geisinger Commonwealth School of Medicine.