Assessment Design Lead

About the Company

Speak is building a human-level, AI-powered tutor that provides a conversation-first experience to help learners progress from beginner to confident speaker. With over $150m raised from investors like OpenAI and Accel, Speak is a leading AI company serving learners in 15+ languages across the globe.

Responsibilities

• Own Assessment Design by defining what Speak measures across Curriculum Mastery, Proficiency, and Placement tests.
• Define constructs and build the rubric/blueprint layer to translate qualitative goals into scoreable technical specs.
• Own validity and the quality bar, signing off on content validity and auditing items for bias.
• Design and run validity evidence plans, including benchmarking against external proficiency measures like CEFR.
• Partner closely with Product Managers and ML Engineers on automated scoring, calibration, and feedback generation.

Requirements

• 4+ years of experience designing rubrics, blueprints, and item specs for shipped language assessment products.
• Deep familiarity with language proficiency frameworks such as CEFR, ACTFL, or IELTS/TOEFL.
• Ability to identify differential item functioning to ensure fairness across different L1 backgrounds and accents.
• Quantitative rigor in interpreting statistics like inter-rater reliability, classical test theory, and basic IRT concepts.
• Proficiency in using AI tools within a professional workflow to draft items or test rubric language.
• Ability to operate effectively in a 0-to-1, ambiguous startup environment.

Preferred Qualifications

• Background in speech or pronunciation science.
• Familiarity with adaptive testing or IRT-adjacent concepts.
• Experience in a large-scale language testing organization or EdTech startup.
• Advanced degree (Master's or PhD) in psychometrics, measurement, applied linguistics, or SLA.