Senior Software Engineer, AI Evaluation Infrastructure

Responsibilities

• Design and ship features end to end: data model, API, background workers, and UI when needed
• Build and improve how agents and models are evaluated in sandboxed environments
• Improve benchmark task quality checks: verifiers, rubric-based and LLM-as-a-judge grading
• Integrate new models, agents, and benchmarks in a generalizable way
• Keep the system fast and reliable as usage grows
• Review teammates' pull requests against engineering guidelines

Requirements

• 5+ years building and operating production backend systems
• Strong Python with Django or similar, PostgreSQL, and a task queue such as Celery
• Hands-on LLM evaluation experience beyond calling a model API
• Judgment about benchmark task quality and verifier weaknesses
• Ownership from fuzzy request through tests, migrations and edge cases
• Comfort with TypeScript and React when changes are visible in the frontend
• Clear written communication in pull requests and design notes

Preferred Qualifications

• Running agents in sandboxed or containerized environments (Docker, Kubernetes, Modal, Daytona)
• LLM-as-a-judge or rubric-based grading experience
• Familiarity with coding agents such as Claude Code, Codex, or OpenHands, and MCP
• Distributed job scheduling, concurrency limits, or database locking at scale
• Open-source evaluation or infrastructure contributions

About the Company

Turing works with frontier AI labs to generate datasets, RL environments and benchmarks, and with Fortune 500 enterprises to deploy agentic AI systems.