AI Experimentation and Feature Store Lead

CoAdvantage is an HCM company providing payroll, ASO, and PEO services to 16,000 clients. We deliver payroll, benefits, HR compliance, time/PTO, and risk management solutions, and we are building a governed AI platform that will become a primary source of differentiation versus AI-native competitors. The AI program runs three substrates (engineering knowledge graph, analytics feature store, customer knowledge store) and a multi-agent harness.

Position Summary:
CoAdvantage is establishing a structured experimentation function to evaluate, measure, and operationalize AI investments under a productivity-unit (PU) framework. The AI Experimentation and Feature Store Lead is responsible for designing pilots, measuring outcomes against pre-declared thresholds, and producing the evidence that determines whether a tooling investment scales, iterates, or is retired.

The role is quantitative and operational. The person in this role works directly with function leaders (implementations, payroll operations, benefits, contact center, underwriting) to instrument their workflows, run controlled pilots, and report realized τ (the per-resource productivity multiplier) against forecast.

This is a hands-on role. The Experimentation Lead is also accountable for the analytics feature store as a production substrate and manages two Data Scientists who work across both substrates: the PU framework (baselines, causal estimators, τ measurement) and the feature store (feature definitions and production models).

Core responsibilities:
Measurement and baselines
The Experimentation Lead is responsible for defining the productivity unit (PU) for each in-scope function, establishing baseline per-resource capacity (c_r), and maintaining the canonical record of these baselines. This includes the data pipelines, the definitional documentation, and the variance bounds.

Pilot design and execution
For each candidate AI investment, the Experimentation Lead designs the pilot: scope, sample, control, duration, success thresholds, and quality gates. Pilots are pre-registered with the Head of AI before launch. The Experimentation Lead runs the pilot through execution, including the operational coordination with function leaders.

Outcome measurement
At pilot conclusion, the Experimentation Lead produces a written readout containing realized τ, realized c_r, quality impact, and a recommendation against the pre-declared thresholds. Readouts are reproducible from the underlying data.

Stage-gate and portfolio reporting
The Experimentation Lead maintains the quarterly PU dashboard reported to the executive team and board: realized τ by function, N forecast versus actual, cumulative cost impact, and pilot pipeline status.

Cross-functional coordination
The Experimentation Lead works with finance to translate operational measurements into the cost case, with function leaders to access data and operations, and with vendors when pilots involve third-party tooling.

Analytics feature store ownership
The Experimentation Lead owns the analytics feature store as a production substrate: feature definitions, lineage, freshness, access patterns, and the contract with downstream model consumers. The Experimentation Lead works directly with the data team to validate feasibility of proposed features (source availability, refresh cadence, governance) and is accountable for the operational health of the feature store alongside the Staff MLOps Engineer who runs its infrastructure.

Management of the Data Science team
The Experimentation Lead manages two Data Scientists, both of whom work across the PU framework and the feature store. Day-to-day responsibilities include allocating workstream leadership across the two-person bench (Aetna pricing lead, contact-center baseline lead, etc.), prioritizing the combined PU and model backlog, reviewing methodological choices (identification strategy, validation, robustness), unblocking access to data, and signing off on both PU baselines and model promotion to production. The Experimentation Lead is the methodological reviewer of record for every PU baseline published and every model that lands in the feature store.

AI-assisted code and hands-on practice
The Experimentation Lead is hands-on with code. The role is expected to use AI-assisted coding tools (Claude Code, Copilot, or equivalent) as a default development surface for analysis pipelines, feature definitions, and pilot instrumentation. The role is not deck-only: working code is the deliverable that backs every readout.

Required Qualifications:

• Seven or more years of experience in a quantitative role: data science, operations research, business analytics, or industrial engineering. At least two of those years in a player-coach or team-lead capacity.
• Demonstrated experience designing and running controlled experiments or A/B tests on operational processes, not only on digital products. Candidates who have only run web experimentation should be able to articulate how the methodology transfers to back-office workflows.
• Working fluency with Python and SQL. Comfortable authoring production-grade analysis code, feature definitions, and pipelines without an engineering intermediary.
• Hands-on experience with AI-assisted coding tools (Claude Code, Copilot, Cursor, or equivalent) as a daily driver, with code commits or repositories to demonstrate the practice.
• Direct exposure to a feature store (Feast, Databricks Feature Store, Tecton, Vertex Feature Store, or an internal equivalent) — both as a consumer and as an owner of feature definitions.
• Direct experience translating operational metrics into financial impact: cost per unit, fully-loaded labor cost, payback period, NPV. Comfort sitting in a finance review meeting.
• Written communication skills sufficient to produce executive-grade readouts.

Preferred Qualifications:

• Prior experience in a PEO, HR outsourcing, BPO, or other labor-intensive services organization.
• Direct exposure to AI or automation deployments and to the gap between vendor claims and realized outcomes.
• Familiarity with causal inference techniques (difference-in-differences, synthetic control) for situations where randomized pilots are not feasible.
• Background in process improvement methodologies (Lean, Six Sigma) as a supporting framework, not as the primary lens.

What success looks like at 12 months:
By the end of year one, the Experimentation Lead is expected to have:

• Established PU definitions and validated c_r baselines for at least four CoAdvantage functions.
• Run at least six pilots end-to-end, with written readouts and clear scale-or-retire recommendations.
• Stood up the quarterly PU dashboard with executive sign-off on the methodology.
• Built a documented playbook for pilot design that another analyst can execute against.
• Identified at least two scaled deployments where realized τ tracked within twenty percent of forecast.
• Brought the analytics feature store to a defined v1 (feature catalog, freshness SLAs, lineage, access controls) with at least three production models served from it.
• Onboarded and developed the two Data Scientists into independent operators across both substrates — each capable of leading a workstream on either the PU framework side or the feature-store side without methodological hand-holding.

EEO

CoAdvantage is committed to providing equal employment opportunities to all employees and applicants without regard to race, color, religion, national origin, ancestry, citizenship status, age, sex (including pregnancy, childbirth, breast feeding and pregnancy-related medical conditions), gender, gender identity or expression, sexual orientation, marital status, uniform service member and veteran status, disability, genetic information, or any other characteristic protected by applicable federal, state, or local laws and ordinances.

#LI-remote