A/B tests
An A/B test shows half of a collection's shoppers a different order, then tells you which half clicked, added to cart and ordered more. Open A/B tests under Merchandising and click New test, or click A/B test on a collection.
Version A and version B
A is the collection as it's set up now. B is the same collection with its own ranking, its own diversity layout, or both. Pins, boosts, hide and bury rules and filters apply to both versions, so the test compares only what you changed.
A collection can run one test at a time. You can test several collections at once.
Who sees which
- Each shopper gets A or B at random, like a coin flip, and keeps it on every visit. That keeps the two groups alike, so a difference between them comes from the order they saw.
- Only shoppers who allow analytics are in a test. Everyone else sees A and isn't counted.
- A shopper's very first page is always A: their coin flip is made as that page loads.
Reading the results
The test page compares the groups on:
- Shoppers who clicked a product in the collection.
- Shoppers who added to cart and who ordered a product they clicked in the collection.
- Revenue and revenue per shopper from those orders.
Confidence says how sure Relvello is that a difference isn't chance. At 95% or more it calls a winner, looking at orders first, then add-to-carts, then clicks. Below that, or until each group has 100 shoppers, it keeps collecting. Orders and revenue need a tracked checkout.
The page also shows the first products each group sees right now, side by side.
Ending a test
Give a test at least a week: shoppers behave differently on weekdays and weekends. Then end it one of two ways:
- End and keep A: every shopper sees the collection as it was.
- End and apply B: B's ranking and layout become the collection's own settings.
The results stay on the test after it ends.