Sample A/B experiments for Push

Experimentation is the key to marketing. Marketers use A/B or Multivariate tests to test how small changes in campaigns can impact results in a big way. One can then plough back the learnings from one experiment into the next campaign and build on it. A/B experiments are conducted to test one variable. Multivariate experiments can be done across multiple ‘variables’ and one can compare how one variation performs better for a given conversion goal. As customers will be served either of the variations, their engagement with each experience is measured and collected to be shown as analytics. If possible, you should use Control Group to create a baseline. You can go as low as 2-5% for your control group allocation. You can start experimenting with your messages:

​

Emojis

Sample Hypothesis: Having an Emoji in a message gives better CTRs Experiment: Use Emoji in Variation 1 and send a plain message in Variation 2 Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better then there are high chances you have a valid hypothesis.

​

Personalization

Sample Hypothesis: Using the customer’s First Name or Full Name results in better CTRs Experiment: Use First Name attribute-based personalization in Variation 1 and create a plain message in Variation 2 Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better then there are high chances you have a valid hypothesis.

​

Action Buttons

Sample Hypothesis: Using action buttons allows results in higher click-throughs Experiment: Use notification action in Variation 1 while using only the default notification click in Variation 2. Set the conversion goal as per what conversion event you want to track. Measure: Compare CTR for Variation 1 & Variation 2 . If the CTR of Variation 1 is better than Variation 2, there are high chances you have a valid hypothesis.

​

Images

Sample Hypothesis: Using images in Android push notifications results in higher click-throughs Experiment: Use an image in Variation 1 while do not use any image in Variation 2 Measure: Compare CTRs of Variation 1 & Variation 2 . If CTRs of Variation 1 are better than that of Variation 2 then there are high chances you have a valid hypothesis.

​

Coupon Code

Sample Hypothesis: Sending Coupon Codes with push notifications result in higher conversions Experiment: Create an experiment where Variation 1 is using the Coupon Code while Variation 2 is not using it.Set the conversion goal as per what conversion event you want to track. Measure: Compare conversions of Variation 1 & Variation 2. If conversions of Variation 1 are better than 2, there are high chances you have a valid hypothesis.

​

Landing pages

Sample Hypothesis: Taking new customers to any deal page results in a higher conversion rate Experiment: Create an experiment where Variation 1 is using the discount code of 10-15% OFF and taking the user to the Home page while Variation 2 does not serve any discount code but takes users to your deals page. Measure: Compare conversions of Variation 1 & Variation 2. If conversions of Variation 2 are better than 1, there are high chances you have a valid hypothesis. You can similarly experiment:

• if using a custom sound gets you more user attention resulting in higher CTRs
• if using a carousel of images gets more clicks in comparison to a single image

Do let us know what else are you experimenting with.