Software Engineer - AI Coding Agent Evaluation

Responsibilities

• Invent realistic developer scenarios such as real bugs, broken ETLs, or missing features
• Build reproducible Docker environments with pinned dependencies
• Write deterministic pytest suites to verify outcomes
• Create instruction documentation formatted like Jira tickets
• Write reference solution scripts to prove task solvability
• Calibrate task difficulty for state-of-the-art AI agents
• Iterate on tasks based on expert QA feedback

Requirements

• 3+ years of production software development in Python, Go, Node.js, Java, or Rust
• Fluency in Python and pytest (fixtures, parametrize, monkeypatch)
• Proficiency in Docker authoring and multi-stage builds
• Strong Linux and Bash debugging skills
• Experience using AI coding agents like Claude Code or Cursor on non-trivial work
• B2+ level written English proficiency

Preferred Qualifications

• Domain expertise in Security, System Administration, Scientific Computing, DevOps, or Git internals
• Experience with modern Python tooling like uv or poetry
• Knowledge of coverage tooling or fuzzing/property-based testing
• Prior contribution to agent-evaluation benchmarks

About the Company

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.