Expert Talent Pool, Evals and Benchmarks
Clera
WorldwideFully remoteContractSeniorInformation Technology
USD 156,000–364,000 a yearPosted 2 hours ago2h agoApply link checked 1 hour ago
Why we think you can apply
- Hiring model
- Open worldwide · the employer states no country restriction
- Work from
- Anywhere, the UAE included
Posted 2 hours ago · apply link checked 1 hour ago
About the role
Responsibilities
- Design benchmarks covering scope, question types, difficulty and scoring
- Write and review rubrics and gold-standard answers
- Lead grading rounds, calibrate graders and resolve disputes
- Analyze model failures and recommend improvements
- Run model comparison studies and validate benchmarks
- Design user studies on how people rate model outputs
Requirements
- 5+ years in evaluation, testing or assessment, or domain expertise with evaluation work
- Experience designing a benchmark, test plan, exam or assessment
- Experience writing or reviewing rubrics and gold-standard answers
- Proficiency in Python and evaluation tools
- Clear written English communication
Preferred Qualifications
- Background in ML evaluation, QA, psychometrics, UX research or data science
- Experience validating benchmarks or measuring grader agreement
About the Company
The network supports AI research projects with rigorous model evaluation.
What we look for
- 5+ years in evaluation, testing or assessment, or domain expertise with evaluation work
- Experience designing a benchmark, test plan, exam or assessment
- Experience writing or reviewing rubrics and gold-standard answers
- Proficiency in Python and evaluation tools
- Clear written English communication
- Background in ML evaluation, QA, psychometrics, UX research or data science
- Experience validating benchmarks or measuring grader agreement
PythonMachine LearningData ScienceUX researchQuality AssuranceHelmBenchmarkingPsychometricsInspectlm-eval-harnesspromptfoo
SponsorWake up to a shortlist, not a search results page.
NeverApply scores every new listing against your CV, salary floor and visa. A handful of real matches by morning.
Get your daily matchesSimilar jobsbased on title, category and scope
SponsorWake up to a shortlist, not a search results page.
NeverApply scores every new listing against your CV, salary floor and visa. A handful of real matches by morning.
Get your daily matchesExpert Talent Pool, Evals and BenchmarksClera · free account, 30 seconds