A/B Testing

Checkout Promotions offers the ability to A/B test promotions at the offer level. What this means is that you can show customers a version a or a version b of an offer to determine which offer converts better. 

Important Considerations

A/B Testing is only reliable when you run the experiment over a large enough sample size to ensure a statistically valid result. Small sample sizes can often lead to inaccurate results, this means if you don't have a high order volume, you may have to run the A/B test over a longer period of time. We are working on introducing automated logic to pick a statistically valid winning version.

Setting Up A/B Testing

Select a promotion and on the promotion's detail page, click on "Edit offer" or "Edit content" depending on offer type (Commerce vs content):

Then click on "Enable A/B Testing" within the offer editor sidebar on the left side:

This will create a second version. Make any changes to the two versions and save. Then click "Exit editor" to go back to the promotion page. You'll now see two offer versions shown with analytics specific to each offer. Promotion level analytics will not be broken down by offer version:

If the offer already has views, clicks, and other analytic data, you can optionally clear the offer level analytics (it will still keep the promotion level analytics). Next to "Edit offer" or "Edit content", click on the down arrow and select, "Reset analytics":

Picking A Winning Version

After the A/B test has been running and you are ready to select a winning version, just click on "Edit offer" or "Edit content" to navigate to the offer editor. And then select the offer version that you want to remove within the offer editor:

Then scroll down until you see an option to delete the version and click " Delete version b". This will disable A/B testing and keep only version a, you're selected winning version. In the future, we'll automate these steps for you: