Expert Talent Pool, Evals and Benchmarks (Associate)

Responsibilities

• Write questions and tasks that isolate and test specific model capabilities
• Develop rubrics defining strong, partial, and poor answers
• Create reference answers and edge cases to challenge AI models
• Grade model outputs against rubrics and explain scoring decisions
• Analyze failed answers, group error patterns, and document findings
• Compare multiple models using identical prompts
• Validate benchmarks and flag compromised or ineffective questions
• Participate in small user studies evaluating model outputs

Requirements

• Zero to five years of professional experience
• QA, test engineering, assessment design, or ML evaluation experience
• Experience writing test cases, exam questions, or rubrics
• Comfort using spreadsheets or Python to analyze results
• Clear written communication

Preferred Qualifications

• Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo

Benefits

• Compensation of $30 to $60 per hour, depending on project, field, and experience
• Payment for hours worked
• Fully remote; projects typically last one week to several months