Senior Software Engineer, AI Benchmarking

Responsibilities

• Perform pre-release and post-release assessments of frontier and open-source AI models using SecureBio's suite of biosecurity evaluations
• Build adapters for models, identify and troubleshoot unexpected agent behaviors, and ensure methodological rigor
• Build, scale, maintain, and continuously improve SecureBio's suite of biosecurity evaluations
• Improve evaluations to use state-of-the-art agent frameworks, jailbreaking methods, and elicitation strategies
• Develop and improve internal tools that enable researchers to run capability assessments at scale
• Contribute to and maintain cloud infrastructure to run evaluations and analyses
• Contribute analyses to the public dashboard of trends in AI biology capabilities

Requirements

• 5+ years of experience in software engineering (industry or open-source)
• Fluency in Python
• Proficiency with Docker and AWS (particularly ECS)
• Experience building software tools that use or evaluate LLMs or AI agents
• Self-directed and comfortable working on small, fast-moving teams
• Commitment to good coding practices and code quality
• Alignment with SecureBio's mission of preventing catastrophic pandemics

Preferred Qualifications

• Experience with AI evaluations (agent systems, scorer/grader design, trajectory analysis, jailbreaking, or red-teaming)
• Experience with Terraform, Tofu, or AWS CDK
• Experience with modern, component-based web frontends (React with TypeScript)
• Experience with Inspect AI, OpenAI-compatible APIs, Anthropic, or Together
• Experience with agent frameworks like Codex or Claude Code

About the Company

SecureBio is a mission-driven team dedicated to protecting the world from biological threats. Through research and development, SecureBio builds the tools and technologies needed to secure the world against future biotechnology risks, including pandemic detection and reducing biological risks from advanced artificial intelligence.