Hiring Experimentation Engineers: The Complete Guide

# Experimentation Engineer

**Location:** San Francisco, CA (Hybrid) · **Employment Type:** Full-time · **Level:** Mid-Senior

## About [Company]

[Company] is a data-driven company that runs 300+ experiments monthly across product, growth, and marketing. Our experimentation platform is how we make decisions—every major feature change goes through an A/B test.

Good experimentation infrastructure is competitive advantage. When experiments are easy to run and results are trustworthy, teams make better decisions faster. Our platform enables that at scale.

**Why join [Company]?**

- Work on experimentation at scale—300+ monthly experiments
- Enable data-driven decisions across the entire company
- Solve hard statistical and engineering problems
- Join a 5-person experimentation team within a 200-person engineering org
- Series C company with strong experimentation culture
- Competitive compensation with meaningful equity

## The Role

We're looking for an Experimentation Engineer to help us scale our experimentation capabilities. You'll work on experiment assignment, statistical analysis, results visualization, and the infrastructure that makes experiments reliable and trustworthy.

Experimentation engineering combines platform engineering with applied statistics. You need to build reliable infrastructure (assignment, exposure logging, metrics computation) while ensuring statistical validity (power analysis, multiple testing, variance reduction).

The ideal candidate has built experimentation systems and understands both sides. You can explain p-values to PMs and implement CUPED for variance reduction.

## What You'll Work On

### First 30 Days
- Onboard to our experimentation platform, assignment system, and analysis tools
- Understand how teams run experiments and common pain points
- Ship your first improvement (analysis tool, UI feature, or reliability fix)

### First 90 Days
- Own an experimentation system component (assignment, analysis, or metrics)
- Contribute to platform improvements
- Support teams running experiments
- Build relationships with data science and product teams

### First Year
- Lead experimentation initiatives that improve platform capabilities
- Architect improvements to our experimentation infrastructure
- Develop advanced capabilities (sequential testing, variance reduction)
- Establish experimentation best practices across the company

## Objectives of This Role

- Increase experiment velocity (more experiments, faster results)
- Improve statistical rigor (valid results, proper power)
- Reduce time-to-decision for experiment conclusions
- Build self-serve capabilities that reduce data science dependency
- Scale platform for 2x experiment volume growth

## Responsibilities

**Platform Development (50%)**
- Build and improve experiment assignment infrastructure
- Develop exposure logging and event tracking
- Create metrics computation pipelines
- Build experiment configuration and management tools
- Ensure platform reliability and consistency

**Statistical Analysis (30%)**
- Develop statistical analysis tools and automation
- Implement power analysis and sample size calculation
- Build significance testing and confidence intervals
- Create variance reduction techniques (CUPED, stratification)
- Design sequential testing capabilities

**User Experience & Support (20%)**
- Build experiment results dashboards
- Create self-serve experiment setup tools
- Support teams in designing and interpreting experiments
- Develop experimentation documentation and training
- Drive best practices adoption across teams

## Required Skills and Qualifications

**Technical Skills**
- 5+ years of software engineering experience
- 3+ years in experimentation, A/B testing, or data engineering
- Strong statistics fundamentals (hypothesis testing, confidence intervals, power)
- Platform engineering experience
- Data pipeline and metrics computation experience
- Proficiency in Python and SQL

**Statistical Knowledge**
- Understanding of experimental design
- Experience with common statistical tests
- Familiarity with online experimentation challenges

**Soft Skills**
- Clear communication of statistical concepts to non-statisticians
- Collaboration with product, engineering, and data science
- Judgment for experiment validity and interpretation

## Preferred Skills and Qualifications

- Advanced statistics (Bayesian methods, sequential testing)
- Variance reduction techniques (CUPED, stratification, regression adjustment)
- Experimentation platform experience (Optimizely, LaunchDarkly, Statsig)
- Multi-armed bandit experience
- Background in causal inference or econometrics
- Data science or statistics degree

## Tech Stack

**Backend:** Python, Go for services

**Data:** Snowflake, Spark, Airflow

**Statistics:** Python (scipy, statsmodels), custom analysis tools

**Platform:** Custom assignment service, feature flags integration

**Visualization:** Custom dashboards, Databricks notebooks

**Infrastructure:** AWS, Kubernetes

## Compensation and Benefits

**Salary:** $150,000 - $200,000 (based on experience)

**Equity:** 0.04% - 0.12% (4-year vest, 1-year cliff)

**Benefits:**

*Health & Wellness*
- Medical, dental, and vision insurance (100% covered)
- $100/month wellness stipend
- Mental health support through Lyra

*Time Off*
- Unlimited PTO with 15-day minimum encouraged
- 11 paid company holidays
- 16 weeks paid parental leave

*Professional Development*
- $2,500 annual learning budget
- Conference attendance (KDD, CIKM)
- Internal experimentation and statistics training

*Financial*
- 401(k) with 4% company match
- Equity refresh grants annually

*Workspace*
- $1,500 home office setup allowance
- Hybrid work (2-3 days in SF office)
- MacBook Pro and equipment of choice

## Engineering Culture

**How we work:**
- Experiment-driven culture—data informs decisions
- Statistical rigor—we care about validity
- Self-serve enablement—reduce bottlenecks
- Transparency in experiment results
- Blameless learning from experiment insights

**What we value:**
- Trustworthy results over fast results
- Enabling others to experiment effectively
- Clear communication about uncertainty
- Continuous improvement of the platform
- Work-life balance (experimentation is not urgent)

## Interview Process

Our interview process typically takes 2 weeks.

- **Step 1: Recruiter Screen** (30 min)
  Background discussion and role overview.

- **Step 2: Technical Screen** (60 min)
  Statistics and platform engineering fundamentals.

- **Step 3: Experimentation Deep Dive** (90 min)
  Technical discussion about experiment design, analysis, and your experience.

- **Step 4: System Design** (75 min)
  Design an experimentation platform component.

- **Step 5: Statistics Exercise** (60 min)
  Work through a statistical analysis scenario.

- **Step 6: Team Interviews** (2 x 30 min)
  Meet potential teammates from experimentation and product.

---

*[Company] is an equal opportunity employer.*