← Selected work · All case studies · How I work
Turning a recommendation model into a testable program
We broke a broad personalization question into seven planned comparisons, with placements, audiences, tracking, and analysis handoffs made explicit.
My role: Led the Digital Optimization Team. This case focuses on the shared evaluation plan, with product and Strategic Analytics partners.
The team: Digital Optimization Team (DOT), product, and Strategic Analytics, coordinating comparisons, tracking, and analysis handoffs.
The test brief
The problem: A recommendation model could behave differently across shopping contexts. One overall result would not explain where it helped.
The proposed solution (solution hypothesis): Plan separate comparisons across placements, filters, and control experiences so each test answers a specific question.
The primary success metric: Revenue per visitor, evaluated separately for each placement and audience.
The secondary success metric: Add-to-bag rate from recommendations, to understand whether the model helped guests find an additional product.
The full story
Break the broad question apart
A recommendation model could appear on category pages, brand pages, the homepage, in the bag, and after an add-to-bag action. Its usefulness could depend on the shopping context. A single aggregate question about whether personalization worked would leave those differences unexplored.
Make the comparison concrete
The Quazi 2.0 requirements mapped seven tests across placements, filters, and control experiences. The plan specified audiences and allocations, making clear which model or curated experience each test would compare and where guests would encounter it.
Plan the data conversation before the readout
The requirements documented tracking needs, reporting views, dates for data delivery, and dates for analysis to return to DOT. Open questions remained visible, including design and comparison choices. This gave product, testing, and Strategic Analytics partners a more specific basis for coordinating the work.
What the plan made possible
The plan made comparisons and dependencies explicit. Forecasts were estimates, and production dates were still to be agreed; the work shown here is the evaluation plan.