← Selected work · All case studies · How I work
Finding the right moment for a recommendation
Ulta Beauty’s published case study reports a 9% revenue increase from recommendations shown after an add-to-bag action. Timing made the idea useful.
My role: Led the Digital Optimization Team’s experimentation program, recommended follow-up tests of the overlay’s individual components, and developed monthly patch audits with Product to move winning experiences into production.
Team: Product, UX, Design, Development, Analytics, and our platform partner.
The test brief
The problem: A guest has added an item to the bag and needs a useful next step. The team explored whether relevant recommendations could support further discovery.
The proposed solution (solution hypothesis): An overlay with recommendations and clear routes to the bag or back to shopping could help guests discover and add other useful products.
The primary success metric: Add-to-bag rate on the recommendation overlay.
The secondary success metric: Conversion rate.
Tertiary metrics: Revenue, items per order, bag views, and average order value.
Control: the product page. Test: recommendations with routes to keep shopping or view the bag. Select the comparison above to enlarge it.
Start with the moment
The add-to-bag action is a transition. A guest may be ready to check out, continue browsing, or discover something that complements what they have chosen. The team explored whether an overlay could support those next steps with relevant recommendations.
I led the experimentation program, working with product, design, engineering, analytics, and our platform partner. The design and delivery were collaborative.
Make the hypothesis observable
The overlay paired product recommendations with routes to the bag and back to shopping. The hypothesis connected that interaction to additional product discovery and bag additions. It gave the team a specific behavior to look for, rather than a general ambition to personalize the experience.
Measure what happens after the click
Recommendation engagement was part of the picture. Bag views, items per order, conversion, and revenue helped evaluate whether the experience supported the wider purchase journey. A module can attract attention without helping someone complete what they came to do; the measurement needed to follow beyond its own boundary.
Read the result at the right scale
The public vendor case study reports a 9% revenue increase, a 15.1% increase in add-to-bag clicks, and a 2.3% conversion-rate uplift for the experiment. These are the publisher’s reported team results, scoped to the test.
Published Ulta Beauty case study
Results from the team readout
The team’s readout reported these relative changes during the test:
Add-to-bag clicks: +15.1%
Items per order: +4.7%
Bag views: +11.5%
Conversion: +2.3%
Average order value: +1.4%
Revenue: +9.0%
What happened next
The overlay was a complex build, closer to a pilot of a new experience than a small incremental A/B test. The result showed the value of the experience as a whole, but it could not isolate which individual components contributed most. I recommended following production implementation with smaller A/B tests of those components to understand how to improve the overlay further.
The cost of keeping a winner patched
We kept the winning overlay live as a patch through AB Tasty until the Product and Development teams built it into production. Maintaining that patch took more work than we wanted. API changes could arrive without our team being notified and break the experience. Sometimes a reported bug was how we learned something had changed.
My small team then had to repair the patch, taking time away from new experiments. A successful test had created an ongoing maintenance responsibility.
Give temporary experiences a path into production
That experience led me to develop a more rigorous patch audit process with Product. We held monthly check-ins to review patched experiences and keep their production handoffs on the roadmap. The goal was to move proven experiences off the testing tool and into production, freeing the team to focus on experimentation.
Across the program, we had managed about 35 patches simultaneously at our peak. By the time I left Ulta Beauty, no patches remained.