AI Product Lead, AI Experience & Quality

Responsibilities

• Manage member-facing AI behavior across prompt-to-create, agentic editing, and future jobs like segmentation and analytics
• Create, test, version, and document prompts and agent instructions
• Define context, member data, and tool requirements for AI performance
• Own the AI evaluation practice, including datasets, scoring rubrics, and human review
• Run recurring evaluation and quality-monitoring processes independently
• Review traces and diagnose failures in prompts, context, or models
• Turn research and feedback into priorities and experiments
• Partner with product, design, and engineering to encode a coherent Flodesk point of view

Requirements

• Proven experience shaping or improving real LLM or agentic products
• Practical knowledge of prompts, context, tools, and agent behavior
• Hands-on experience with prompt iteration, evaluation tools, and trace review
• Proficiency with JSON, schemas, and structured outputs
• Strong product judgment and ability to turn ambiguous needs into testable hypotheses
• Experimental rigor in designing test sets and failure taxonomies
• Ability to communicate clearly across technical and non-technical disciplines

Preferred Qualifications

• Experience owning AI product behavior, conversation design, or prompt systems
• Experience building content-generation or marketing tools
• Experience with Langfuse, Braintrust, or comparable observability platforms
• Experience working on products for small businesses, creators, or marketers

Benefits

• $150,000 - $200,000 base salary depending on location
• Fully paid health insurance for individual coverage
• 16 weeks paid parental leave for non-birthing parents; 22 weeks paid maternity leave for birthing parents
• Unlimited flexible time off
• 401(k) match (US employees only)
• $1,000 annual stipend for learning and development