Expert Talent Pool, Evals and Benchmarks

Clera

WorldwideFully remoteContractSeniorInformation Technology
USD 75–175 an hourPosted 2h ago

Why we think you can apply

Hiring model
Open worldwide · the employer states no country restriction
Work from
Anywhere, the UAE included

Posted 2 hours ago · apply link checked 2 hours ago

About the role

Responsibilities

  • Design benchmarks defining scope, question types, difficulty and scoring
  • Write and review rubrics for consistent grading and grader agreement
  • Develop gold-standard answers and test sets
  • Lead grading rounds, calibrate graders and resolve disputes
  • Analyze model failures and recommend improvements
  • Design and conduct model comparison studies and user studies
  • Validate benchmarks for contamination, ambiguity and saturation

Requirements

  • At least 5 years in evaluation, testing or assessment
  • Experience designing a benchmark, test plan, exam or assessment
  • Experience writing or reviewing rubrics and gold-standard answers
  • Proficiency with Python and evaluation tools such as Inspect, lm-eval-harness, HELM or promptfoo
  • Experience analyzing model or test failures
  • Clear written English communication

Preferred Qualifications

  • Background in ML evaluation, QA, psychometrics, UX research or data science
  • Experience validating benchmarks or measuring grader agreement

Benefits

  • Pay of $75 to $175 per hour based on hours worked, project and experience
  • Fully remote, projects lasting one week to several months

What we look for

  • At least 5 years in evaluation, testing or assessment
  • Experience designing a benchmark, test plan, exam or assessment
  • Experience writing or reviewing rubrics and gold-standard answers
  • Proficiency with Python and evaluation tools such as Inspect, lm-eval-harness, HELM or promptfoo
  • Experience analyzing model or test failures
  • Clear written English communication
  • Background in ML evaluation, QA, psychometrics, UX research or data science
  • Experience validating benchmarks or measuring grader agreement
PythonMachine LearningData ScienceUX researchQuality AssuranceBenchmarkingtest engineering
NeverApplySponsor

Wake up to a shortlist, not a search results page.

NeverApply scores every new listing against your CV, salary floor and visa. A handful of real matches by morning.

Get your daily matches
Similar jobsbased on title, category and scope
Expert Talent Pool, Evals and Benchmarksfree account, 30 seconds

Every new role, on Telegram.

Members see each new UAE role first. A day later it lands in the channel, where 6,800 people follow.

FreeNo spamUpdated hourly
Or show me jobs that hire in the UAE

Remote Jobs in Dubai UAE

6,800+ followers

Get alerts on Telegram