← Selected work · All case studies · How I work
Making the assumptions behind value visible
A common calculator helped distinguish test-period value from annual projections. The useful part was making the assumptions visible.
My role: Led the Digital Optimization Team. The framework below reflects the team’s shared work on measurement and value reporting.
The team: Amy Priola, the analyst on my team, built the ROI calculator. Our Digital Optimization Team (DOT) used it to connect experiment results with business questions about value.
The full story
The reporting problem
Experiment value can refer to revenue observed during a test, a projection over a longer period, or the value of avoiding an underperforming change. Those figures serve different purposes. Treating them as interchangeable makes a confident presentation difficult to interpret.
Build a common calculation path
Amy Priola built a self-service ROI calculator that extrapolated our test results using conversion rates, weighted average order value, and traffic. It separated measured test-period changes and secured gain from conservative and optimistic annual projections. The inputs and outputs gave the team a consistent structure for discussing what a result might be worth.
Make persistence an explicit assumption
The framework used a linear decay model to reduce the projected feature benefit over twelve months. That made effect persistence part of the model. It provided a way to discuss how a short observation period might translate into a longer forecast, rather than leaving that assumption implicit.
Make the next step clear
The framework was designed for self-service use. Migrating it into Airtable was a proposed next step.
What goes into the calculation
Inputs: conversion rates, weighted average order value, and traffic.
Outputs: orders added; test-period uplift or secured gain; conservative annual ROI; and optimistic annual ROI.
Value attributed to a test period and projected annual value answer different questions. A projection is not realized revenue. The $68.9 million figure reported for 2024 was an extrapolated revenue estimate based on our test results using this calculator, rather than revenue directly observed during the tests.