How I write up an A/B test

A test nobody can act on is just activity. I write every A/B test up on one report, built so a busy leader can read the top and know what to do. Here is how the report works, and a blank copy you can use.

What goes in the report

• Objective. The business problem, in one sentence.
• Hypothesis. If we change X, then Y will go up or down, because of Z.
• Target audience. Who sees the test, such as US mobile traffic or new visitors only.
• Measurement. The one main metric, defined before the test starts.
• Experience breakdown. What differs between the control (A) and the variant (B), and how traffic is split.

Findings

Both experiences side by side on the main metric, with the difference, shown as a table or a dashboard screenshot. Then a lift evaluation: if the winning experience went to every visitor, how much revenue could it influence for the rest of the year? I model that against the swings in monthly sales and call it potential, not earned.

Recommendations

I write these as if I owned the area that was tested. Pick a side, say why, and say what happens next.

• Run the winner, and name the metric behind the decision.
• Don’t throw out the control. It may work well in a different part of the customer journey.
• Re-test at a different time of year, when shopper mindset shifts: back to school, holiday, spring.

Supporting detail

Close with the screen grabs or graphs that back up the findings and the recommendation.

Before you run the test

The rules I run tests by are in the digital merchandising operating playbook.

Download the template

A/B test summary report template (Word)

The numbers in the sample table are made up for illustration. Replace the red text and delete the instructions before you share your own.