We are looking for a Member of Technical Staff, AI Evaluation with 5+ years of experience, blending physical engineering (mechanical, electrical, aerospace, robotics) and AI evals, to own how we measure our AI engineer for the physical world. You will build the evaluation framework that shows how the system compares to human engineers and what it takes to grow it from junior to senior engineer, and eventually toward engineering superintelligence.
What you will be doing:
Own the evaluation framework from scratch, defining what good engineering design looks like when there is no single right answer. Work with engineering SMEs, customers, partners, and internal teams to learn how top engineers work and where the system succeeds or fails. Recreate real-world engineering tasks and failure modes in evaluation environments that development teams can rerun. Design and refine automated judges and human evaluation processes, and check that judges match expert human judgment. Turn eval results into clear priorities for improving the system's engineering reasoning.
This is a high-ownership role in a scrappy, early-stage environment building new evaluation capabilities from the ground up, ideal for someone strong across physical engineering and AI evaluation, or deep in one with working knowledge of the other.
- 5+ years of experience across physical engineering and/or AI work (strong in both preferred; deep strength in one with working knowledge of the other is fine)
- Hands-on AI evaluation experience: building eval tasks, designing judges, and running human evaluation for agentic systems
- Physical engineering background (mechanical, electrical, aerospace, robotics, or power systems) sufficient to recognize strong engineering design
- Extensive experience with AI evaluations, including creating evaluation tasks, designing judges, and comparing automated evaluations to human judgment
- Understanding of agentic AI systems
- Experience working directly with customers, subject matter experts, partners, and cross-functional technical teams
- Presence and confidence to lead work across SMEs, customers, and cross-functional teams in a high-ownership role
- Experience in scrappy, early-stage environments building new things from scratch (physical AI or robotics companies a plus)
- Experience with continual learning, fast adaptation, and self-improvement models
- Ability to own and drive a research agenda; comfortable in startup settings
- Strong academic background with practical application skills; PhD preferred with 3-7+ years in startups, labs, or big tech, ideally with applied research experience
- Background in academia and experience productizing research ideas is a plus
- Located in the United States or Canada (remote) or San Mateo, CA
- Ability to travel to the San Mateo, CA office about one week every 6 weeks
- 5-15 years of experience as a Member of Technical Staff, AI engineer, or AI research scientist