AI Operations Analyst at Calltree

About Calltree: We're a YC W25 company with multiple enterprise customers, including public retail brands you'd recognize. We're building AI that actually works for complex, real-world use cases. For example, our AI handles 30+ minute technical troubleshooting calls—the kind everyone else avoids. The Role: You'll be the human supervisor behind our AI agents, working with live customer interactions in real-time. You'll monitor how AI performs, ensure quality, and when needed, jump in to rescue conversations and ensure great customer outcomes. You'll also work directly with engineering to make the AI smarter every day. What You'll Do: Monitor live AI-customer conversations and ensure quality standards Make real-time judgment calls on when to intervene Step in as the escalation point when customers need a human, de-escalating frustrated customers and resolving complex issues Identify patterns in AI performance and document what's working and what's not QA new AI experiences before release—interact with them as customers would and flag issues Work with engineering to improve the AI based on what you observe Build the playbooks and processes for this function as we scale What Success Lo