Data Story: OYO Steadily Increases Quarter-on-Quarter Push Notification CTR by 15% Using AI Optimization

Reading Time: 7 minutes

OYO is a global platform that aims to empower entrepreneurs and small businesses with hotels and homes by providing full-stack technology products and services that aims to increase revenue and ease operations; bringing easy-to-book, affordable, and trusted accommodation to customers around the world. OYO offers 40+ integrated products and solutions to patrons who operate over 157,000 hotel and home storefronts in more than 35 countries including in India, Europe, and Southeast Asia, as of March 31, 2021. For more information, visit www.oyorooms.com

With a unicorn valuation and a pan-India presence, OYO is one of the most well-known and popular hospitality brands in the country. Known for its technology-driven process and a deep emphasis on customer experience, OYO naturally prioritizes customer engagement above all else. As such the customer interactions via channels like mobile push and email among others are significantly on the higher end.

Like any other goal-oriented team, the objective of marketing at OYO was to maximize the impact of their customer interactions across various campaigns. The team runs different campaigns with triggered journeys based on customer action on their app while customizing or personalizing down to the letter. In order to run effective customer-centric campaigns that engage, it is imperative for the team to understand which message variant is performing the best and send the same communication to a larger chunk of the customers.

The results of effective engagement campaigns would reflect in the incremental CTR (click-through rate) improvement, which ultimately adds to the bottom line in terms of new and repeat hotel bookings. This data story is all about the Quarter-on-Quarter CTR increment of 15% observed by OYO and how MoEngage helped them achieve it.

Check out how OYO optimizes customer journeys and launches industry-first initiatives with MoEngage:

Manual Configuration of Best-performing Campaign Variant and Loss of CTRs During Experimentation

In the past, teams at OYO would perform static A/B testing, stick around for the campaign to run its course, and then arrive at results.

This process required a lot of manual intervention to configure the better performing variation and maximize its use. This sort of manual intervention and timely inference becomes even more difficult with automatic trigger and period campaigns that run for longer durations. On top of that, it also resulted in the loss of CTR during the initial stage of experimentation.

It is important to point out that while static A/B testing focuses on statistical significance and hypothesis proving, what the team at OYO needed was continuous optimization where the focus would be on maintaining higher average conversions and click rates.

As you can see below, apart from manually deciding the percentage of customers to send each message variant, the percentage distribution would also remain unchanged throughout the campaign’s run:

Dynamic Testing and Real-time Optimization Saves Time and Manual Effort

The team at OYO needed to optimize campaigns in real-time, by analyzing the performance of message variants and maximizing percentage distribution for those variants. This was possible through the implementation Sherpa, MoEngage’s proprietary machine learning engine, which allowed the team to complete the testing and optimization quicker and in real-time.

Sherpa automatically maximizes campaign performance by predicting the right message. Sherpa automatically optimizes the percentage distribution in the variations to maximize the CTR from the campaign. With this machine learning-based feature, the message variation with chances of highest interaction is intelligently predicted on the fly and sent to customers to maximize engagement.

Post the implementation, Sherpa automatically assigned message variations to the customer segments who qualified for a campaign and optimized the distribution across the variations based on their performance, for the duration of the campaign.

How Oyo’s Team Identified Segments and Campaigns to Optimize Using Sherpa

The team at OYO predominantly used Sherpa’s content optimization capabilities for triggered campaigns.

They ran these engagement campaigns for two segments majorly among other segments:

• customers visiting the hotel detail page (penultimate step in the conversion journey) but not booking
• customers searching for properties in a particular city, viewing search listings page but not booking

For these segments, the team rolled out different content variants:

In one content variant, they focused on the discount amount, along with the hotel name and other such details

In the second variation, they decided not to disclose the discount amount and instead keep it a mystery

In the third variation, they specified the rating of a particular hotel, alongside the number of customers who’ve rated said hotel

With Sherpa’s content optimization capabilities, the team was able to directly set the variation and Sherpa automatically maximized the distribution of customers on the variation that performed better.

Prior to Sherpa implementation, whenever the teams at OYO set up an experiment, they had to manually set up the variations and divide them into equal percentages and after 3-4 days, they’d check the CTRs of those variants to understand which one’s is performing better and optimize accordingly. With Sherpa, they are able to set it from the initial stages and Sherpa would optimize the distribution of customers by measuring CTR performance of the aforementioned three variants.

Steady Uplift in Clicks Quarter-on-quarter and Identification of Best-performing Campaign Variants

As a result of the Sherpa optimization, OYO has observed a steady increase in the quarter-on-quarter clicks with an absolute improvement being 15% in Q3, 2021, a massive incremental CTR growth!

One of the other aspects which have come in handy for OYO is the identification of best-performing campaigns. In this case, OYO observed the campaign variants with mystery elements (message variants not disclosing the discount amount) performing much better than the other campaigns (alongside price drops, urgency setting, and exclusive content). This understanding of what works well and what doesn’t will go a long way in creating effective, engagement campaigns for customers.

Inside Look into the Working of Sherpa-enabled Content Optimization

Oftentimes when marketers create campaigns, there are multiple variations of a particular campaign (the message) that they want to test and understand what works best for them. Typically they do this via A/B testing. Say, there are two variations to test, the marketer would split the entire customer group into two halves, sending one variation to 50% of customers and another to the other half. Post the end of the campaign, the marketer can pull the data to verify which variation is doing well and use that variation in subsequent communication.

Ensure your best-performing campaign reaches maximum customers

Sherpa content optimization dynamically modifies the allocation of customers to each campaign variation in real-time. Through this approach, marketers and brands can ensure their best-performing campaign gets delivered to the maximum number of customers automatically so that the CTRs and overall conversions are improved. While in the traditional scenario, there’s a lot of manual intervention in terms of deciding what percentage of customers to be allocated to each variant (say 50%-50%), with Sherpa, the marketer gets to decide on the KPI on which to optimize (open rate or click rate or both) and define the campaign duration. Sherpa then distributes the customer base across this duration into equal chunks.

Predict the message variant your customers will love

Using a Multi-arm Bandit algorithm, Sherpa splits percentages for different variations. The algorithm uses a method called ‘explore and exploit’ which looks at past variations of the message sent and the conversion/CTR numbers and then predicts what is the likelihood that a particular variation will do well for the next set of customers. Based on that, Sherpa will then optimize the percentage of customers who need to be sent a particular variation of the message.

Say, for example, you have 3 variations – A, B, and C, in the first go the message is sent to an equal split of customers, i.e. 33%.

Now, as the campaign progresses, Sherpa will observe impressions and clicks on each of the variations and recalculate the chance of beating all.

At this point, Sherpa continues the campaign with new percentages arrived at. C here looks like a winner and gets 78% allocation accordingly. This is exploitation, while the rest 22% going to variation A & B, is exploration. In this way, Sherpa keeps recomputing the ‘chance of beating all’ and keeps changing the allocation continuously.

For campaigns created using Dynamic A/B Testing powered by Sherpa, marketers can calculate the following metrics:

Insights-led Approach Is the Way Ahead!

Marketers need to move away from the ‘plan > run > analyze > course-correct’ process, and instead focus on ‘analyze > plan > execute > measure > course-correct’. Consumers today have a plethora of offerings to choose from and consumer brands need to realize that and provide delightful, seamless, consistent, and relevant omnichannel experiences. This is where insights-led engagement comes into the picture.

Insights-led engagement allows marketers and brands to understand customer behavior and actions and lead with these insights to personalize engagement in a relevant manner for each customer (or cohort). And the results are clear for everyone to see: OYO was able to steadily increase their quarter-on-quarter clicks by 15% using dynamic optimization (offered by Sherpa) which helped the team identify the best-performing message variant and create effective, engagement campaigns for the customers accordingly.

Start your journey to insights-led engagement and drive growth for your business today:

GET A DEMO OF SHERPA

Further Reading:

• Find out in detail about how Sherpa helps you optimize your customer engagement
• Find out how OYO boosts push notification delivery by 44% using MoEngage
• Discover how OYO increased engagement by 8X with a seamless omnichannel approach
• Find out how Media & Entertainment brands can engage and retain millions of users efficiently using Insight-Led Engagement