Research Scientist

Responsibilities

• Conduct evaluation research to turn eval targets into original benchmark designs
• Own measurement questions including construct validity, item discrimination, and contamination
• Build evaluation packages with expert-verified ground truth and rigorous QC
• Recruit, calibrate, and review a pool of subject-matter experts across coding, agentic, and STEM fields
• Act as a technical point of contact for AI labs to translate measurement needs into evaluation designs
• Turn research findings into public benchmarks and papers for venues like NeurIPS, ICLR, and ACL

Requirements

• Research background in ML evaluation or benchmarking
• Deep expertise in LLM/frontier-model benchmarking, specifically code-model and agentic evaluation
• Strong understanding of psychometrics, rubrics, and the measurement problem
• Interest in AI safety, capability elicitation, and robustness
• Ability to manage an expert pool and maintain rigorous standards
• Proficiency in the full research loop from framing to publication
• Fluency in English; Spanish is a plus