Research Engineer, Evals at Variance

At Variance, we are teaching machines to make the hardest judgment calls at scale. That means building AI agents for the high-stakes gray area of risk investigations, fraud, and identity reviews. We’re a small, talent-dense team in San Francisco working on a problem at the edge of what AI systems can reliably do: making good decisions in messy, adversarial, real-world environments. We focus on building, high-consequence systems problems where the edge cases matter most. We’re looking for a Research Engineer to help define how we measure and improve model quality. You’ll build the benchmarks, datasets, tooling, and evaluation loops that tell us whether our systems are actually getting better on the tasks that matter. This role sits at the center of research, product, and engineering. It is about creating rigorous, domain-specific evaluations that reflect real customer workflows, expose meaningful failure modes, and drive the next generation of model and agent improvements. Care deeply about craftsmanship and have strong opinions about model quality, measurement, and experimental rigor Care deeply about craftsmanship and have strong opinions about model quality, measurement, and expe

Details

equity range: Full-time
job type: Full-time
job url: https://www.ycombinator.com/companies/variance/jobs/Eok2YSs-research-engineer-evals
location: Engineering, Full stack
role: At Variance, we are teaching machines to make the hardest judgment calls at scale. That means building AI agents for the high-stakes gray area of risk investigations, fraud, and identity reviews. We’re a small, talent-dense team in San Francisco working on a problem at the edge of what AI systems can reliably do: making good decisions in messy, adversarial, real-world environments. We focus on building, high-consequence systems problems where the edge cases matter most. We’re looking for a Research Engineer to help define how we measure and improve model quality. You’ll build the benchmarks, datasets, tooling, and evaluation loops that tell us whether our systems are actually getting better on the tasks that matter. This role sits at the center of research, product, and engineering. It is about creating rigorous, domain-specific evaluations that reflect real customer workflows, expose meaningful failure modes, and drive the next generation of model and agent improvements. You’re a fit if you: Care deeply about craftsmanship and have strong opinions about model quality, measurement, and experimental rigor Want to work on core model and agent behavior, not just surface-level product metrics Are excited by the challenge of defining what “good” looks like in messy, high-stakes environments Think in tight loops: hypothesis, benchmark design, evaluation, failure analysis, iteration Have strong engineering fundamentals and like building robust systems around ambiguous research problems Thrive in environments where success criteria are initially underspecified and need to be sharpened through work Are willing to do the work in the trenches: reviewing outputs, grading edge cases, curating datasets, and refining tasks until the evaluation actually measures what matters Care deeply about building systems that protect people from fraud, scams, and abuse What you’ll do Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk wor

company → has job1

Variance