Software Engineering & DevOps AI Rater/Evaluator

Responsibilities

• Evaluate AI outputs related to software engineering, DevOps, and infrastructure topics
• Perform structured scoring, comparison, classification, and judgment tasks
• Assess technical correctness, completeness, security implications, and best-practice alignment
• Identify hallucinations, incorrect code, unsafe recommendations, or misleading system guidance
• Apply domain-specific engineering and DevOps guidelines consistently across tasks

Requirements

• Software engineers, site reliability engineers, DevOps engineers, or platform engineers
• Experience with production systems, CI/CD pipelines, cloud infrastructure, or distributed systems
• Strong attention to detail and comfort working with structured evaluation criteria

Preferred Qualifications

• Senior software engineers, DevOps leads, SREs, or technical architects
• Experience defining technical standards, reviewing complex edge cases, or advising on system design and reliability
• Ability to clearly explain nuanced technical reasoning and tradeoffs

Benefits

About the Company

LILT is building a global network of domain experts to support high-quality AI evaluation across training, benchmarking, red-teaming, and ongoing model monitoring. We are seeking software engineering and DevOps professionals to contribute expert judgment to human-in-the-loop AI evaluation workflows used by leading enterprises and hyperscalers.

This role is designed for professionals who understand how software systems, infrastructure, and development practices work in real production environments and who can apply that expertise to evaluate, assess, and improve multilingual AI systems.

Your contribution of expertise will directly influence multilingual AI model quality, safety, and deployment readiness.

This role includes two distinct expert tracks, based on experience level and scope of responsibility.