Design datasets and evaluation rubrics that shape how cutting-edge AI models learn and improve. Work closely with research teams at leading AI labs, running rapid experiments to diagnose failure modes and refine metrics for model progress. Your work directly impacts large-scale model training runs across diverse domains.
Key Responsibilities:
- Design data slices and explore data shapes that expose meaningful model failure modes across domains like finance, code, and enterprise workflows
- Build and refine evaluation rubrics and reward signals for RLHF and RLVR training pipelines
- Model annotator behavior and run experiments to improve different model capabilities
- Develop quantitative frameworks for measuring dataset quality, diversity, and downstream impact on model alignment and capability
- Create and manage both real-world and synthetic data pipelines
- Partner with lab research teams to translate their training objectives into concrete data and evaluation specifications
1-4 years of experience as a software engineer
Strong backend development focus with fast coding abilities
Experience creating benchmarks, with supervised fine-tuning (SFT) or reinforcement learning (RL)
CS or similar degree from a top university
Depth in Python, TypeScript, and other fullstack or backend languages
Bias for action and execution, willing to tackle difficult and tedious work
Preferred:
Experience at fast-growing startups or creating complex simulations
Background at top VC-backed startups, quant firms, hedge funds, or similar
Track record of side projects, published papers, or AI research/products
Previous experience as a founder or early-stage startup engineer
Explicit interest or experience in reinforcement learning