Dynamic A/B Testing: Why You Should Do It? (With Proof)

Reading Time: 3 minutes

Do you A/B test your marketing campaigns? Well, you should. But do you Dynamic A/B test your marketing campaigns? The answer is “Hell Yes! You should.” Recently, we launched our AI-based content optimization tool – Sherpa that helps marketers optimize their A/B testing campaigns to generate higher engagement.

In a Dynamic A/B testing campaign, Sherpa learns the performance of each variant on the fly by registering user interactions, then automatically drives traffic towards top-performing content. Sherpa eliminates the need for the marketing team to analyze and choose manually.

As the saying goes, the proof of the pudding is in eating it. So, today let’s look at Dynamic A/B testing in action and how it has helped some of our customers get more results for their campaign efforts.

Proof #1

Chillr is a multi-bank payments app that allows bank account holders to do banking and financial transactions. In this example, Chillr ran a Dynamic A/B testing campaign with two variants: One with an image and the other without an image. As MoEngage Sherpa learned of a better performing variant among the two (Variation 2) it automatically diverted more traffic towards that (better performing) variant resultantly achieving a higher CTR for the overall campaign.

As seen above, based on initial learnings, Sherpa diverts more traffic to Variation 2 (higher performing variant). Variation 2 receives 82% vis-a-vis Variation 1 which receives only 18% of the share of traffic to maximize campaign engagement.

Proof #2

Matahari mall is the largest e-commerce store in Indonesia offering a wide range of products. In a bid to optimize their campaign performance, Matahari mall ran Dynamic A/B testing campaigns. Working with a promotional campaign offering free vouchers to shoppers, Matahari mall saw the following results for the campaign. As MoEngage Sherpa learned of a better performing variant among the two (Variation 1) it automatically diverted more traffic towards that (better performing) variant resultantly achieving higher CTR for the overall campaign.

Based on initial learnings, Sherpa diverts more traffic to Variation 1 (higher performing variant). Variation 1 receives 96% vis-a-vis Variation 2 receives only 4% of the share of traffic to maximize campaign engagement.

Proof #3

In this example, a leading digital, print, and e-commerce media company focused on millennials. The publication used web push notifications to drive users to interesting content such as puzzles, crosswords, interesting facts and such. The publication used Dynamic A/B Testing to drive engagement for their web push notification campaigns and witnessed the following results. As MoEngage Sherpa learned of a better performing variant among the two (Variation 2) it automatically diverted more traffic towards that (better performing) variant resultantly achieving a higher CTR for the overall campaign.

Based on initial learnings, Sherpa diverts more traffic to Variation 2 (higher performing variant). Variation 2 receives 90% vis-a-vis Variation 1 which receives only 10% of the share of traffic to maximize campaign engagement.

To learn more about how MoEngage Sherpa can help optimize your marketing campaigns, reach out to us below.