
Expert Talent Pool, Evals and Benchmarks
Clera
WorldwideFully remoteContractSeniorInformation Technology
USD 75–175 an hourPosted 4 hours ago4h agoApply link checked 4 hours ago
Why we think you can apply
- Hiring model
- Open worldwide · the employer states no country restriction
- Work from
- Anywhere, the UAE included
Posted 4 hours ago · apply link checked 4 hours ago
About the role
Responsibilities
- Design benchmarks defining scope, question types, difficulty and scoring
- Write and review rubrics for consistent grading and grader agreement
- Develop gold-standard answers and test sets
- Lead grading rounds, calibrate graders and resolve disputes
- Analyze model failures and recommend improvements
- Design and conduct model comparison studies and user studies
- Validate benchmarks for contamination, ambiguity and saturation
Requirements
- At least 5 years in evaluation, testing or assessment
- Experience designing a benchmark, test plan, exam or assessment
- Experience writing or reviewing rubrics and gold-standard answers
- Proficiency with Python and evaluation tools such as Inspect, lm-eval-harness, HELM or promptfoo
- Experience analyzing model or test failures
- Clear written English communication
Preferred Qualifications
- Background in ML evaluation, QA, psychometrics, UX research or data science
- Experience validating benchmarks or measuring grader agreement
Benefits
- Pay of $75 to $175 per hour based on hours worked, project and experience
- Fully remote, projects lasting one week to several months
What we look for
- At least 5 years in evaluation, testing or assessment
- Experience designing a benchmark, test plan, exam or assessment
- Experience writing or reviewing rubrics and gold-standard answers
- Proficiency with Python and evaluation tools such as Inspect, lm-eval-harness, HELM or promptfoo
- Experience analyzing model or test failures
- Clear written English communication
- Background in ML evaluation, QA, psychometrics, UX research or data science
- Experience validating benchmarks or measuring grader agreement
PythonMachine LearningData ScienceUX researchQuality AssuranceBenchmarkingtest engineering
SponsorWake up to a shortlist, not a search results page.
NeverApply scores every new listing against your CV, salary floor and visa. A handful of real matches by morning.
Get your daily matchesSimilar jobsbased on title, category and scope
SponsorWake up to a shortlist, not a search results page.
NeverApply scores every new listing against your CV, salary floor and visa. A handful of real matches by morning.
Get your daily matchesExpert Talent Pool, Evals and BenchmarksClera · free account, 30 seconds