Software Engineer - Benchmarking

About the Company

Office Hours is an on-demand expert network that connects leading organizations with trusted experts across various knowledge domains. Our vision is to make human knowledge easily accessible and infinitely scalable by building tools for the new age knowledge economy.

Responsibilities

• Prepare and maintain benchmark datasets including cleaning, preparation, and conversion into runnable formats
• Build and maintain evaluation pipelines to run evaluations consistently across model APIs and terminal agents
• Create lightweight, containerized environments for tasking and evaluating model performance on tool use
• Support model experiments including fine-tuning small open-source LLMs
• Develop scoreboards and leaderboards for published results across various performance levels
• Build analysis tools to identify failure modes and track capability improvements or regressions
• Build tooling for data creation and review, such as HTML viewers and internal tools for task authoring

Requirements

• 4+ years of professional experience building complex systems
• Strong proficiency in Python
• Experience preparing, cleaning, and maintaining datasets with high data rigor
• Experience with Docker and building reproducible execution environments
• Ability to collaborate closely with researchers and scientists to translate methodology into systems

Preferred Qualifications

• Hands-on experience running AI evaluations or using frameworks like Harbor, Terminal-Bench, or Inspect
• Experience fine-tuning or post-training open-source LLMs
• Experience with agentic, multi-turn, long-context, or tool-use evaluation
• Experience validating LLM-as-judge or rubric-based grading setups
• Experience building data-heavy dashboards or visualizations