A Year of Building: MoEngage Engineering Review 2022

Reading Time: 7 minutes

2022 has been a year of quiet but effective work. There were no big bang projects from engineering, but we slowly and steadily knocked off some of the essential tech debts. In this article, I will discuss a few milestones for Engineering at MoEngage.

Reliability at Scale

Reliability metrics indicate how satisfied our customers are. These SLAs are measured for every product and reviewed monthly to ensure we deliver on our customer commitments. In 2021, we found SLA breaches in a few areas and worked to improve them in 2022. Here are the improvements we’ve achieved, along with the results:

1. Frequency capping helps our customers create large campaigns without worrying about sending too many messages to their end customers. (An example of a frequency cap setting would be – Sending only 5 communication in 10 days per user.) Earlier, our code would not scale according to our requirements, which meant we were throwing more redis at the problem. Today we are not only able to scale effectively but also optimize costs by up to 70%. This project saved us a lot of dollars and helped deliver on customer SLAs.
2. Our Engineering Manager for QA ensures the team is constantly upskilling and prepped for global best practices. Our QA team gained ISTQB certifications and aims to add more certifications in the coming years. These initiatives help our teams improve their skills and implement global best practices that help us deliver products with minimal bugs and benefit our customers.
3. Last year, we formed a team to focus on core FE improvements and prioritized pending projects. One such project was the lazy loading of a few libraries to improve our dashboard loading times. We have improved build times by 32% and upgraded our node version. We introduced caching in a few areas to improve our dashboard lighthouse score from 62 to 70.
4. Users in the segment (UIS) as a service is a base for our customers’ personalization services. We faced some scale problems and set out to solve them last year. We have changed the stack and achieved 10x write performance and 3x latency reduction at the exact cost.
5. Flows is the most popular product at MoEngage – almost all our customers use it. We constantly improve this product to help our customers achieve more. Last year, the team significantly reduced the on-call, measured, and improved SLAs to meet these near real-time expectations. Why weren’t we able to do this earlier? We had to balance between the costs and SLAs here – to meet the SLAs, we had to throw a lot of infrastructure. Though the team hasn’t changed the stack, they achieved SLA and cost optimization with a good separation of customers and code optimizations.

We believe in challenging the status quo and working toward our customer’s needs. The desire to deliver the right technology for use cases keeps our team motivated and challenged.

Efficiency to Drive Sustainable Growth

Every company needs to find a way to operate efficiently. At MoEngage, we have always followed the principles of sustainable growth and keep a close eye on efficiency as an organizational metric. Last year was an excellent start in this direction, and I am confident that MoEngage will see a considerable improvement in this area going forward. Here are a few areas we improved in efficiency and results:

1. We identified on-call load as a massive area of improvement. Last year, cutting the on-call pagers was identified as a priority for our teams. While some teams could reduce the on-call toil by 50%, others had zero on-call pagers. There is less context switching, or pagers, more reliability to our customers, and happy coding for our teams.
2. As a SAAS company, our developers sometimes need to get involved in the support chain, another major item on our on-call schedule. We have consciously decided to train our support teams and develop enough documentation to eliminate any support escalations that can come to the engineering team. I am excited to share that the escalations have been reduced by more than 50%. As a result, we served more customers with less bandwidth.
3. One of our most prominent data centers has gotten big; any slight glitch would affect many customers. We had 3 DCs and launched our fourth DC in the US region to fulfill the data residency requirements and support better SLAs. While launching the DCs, we had our share of learnings, but eventually, we went live. This also means the devs need not worry about scale problems for some time.

Cost Optimization

Serving customers at the right cost is critical for any business. Our pricing is simple, and we bill our customers based on events and MAUs. But, only some of our products are directly proportionally used on the events or MAUs, making it challenging for the engineering team to predict the costs and keep them in check.

1. We tried to solve this problem by developing FUPs and pricing models for new products that don’t scale with MAUs. We built an internal service that can tell us how much we spend on a customer to serve a product. Last year, we rolled out this internal service to most of our services, and we can predict our serving price for all products with 90% accuracy. This project helps us in 2 ways – get the pricing models right for new sales deals and identify customers who are overusing our products based on their pricing.

Data Privacy and Security Are Priorities

You can never over-emphasize data security and user privacy. Yet, there is also a limit on how much you can achieve at a given time. We understand our customers’ concerns about consumer data safeguards, and we are constantly striving to reassure our customers. We made significant investments in security in 2021, and the trend continued during the last year.

1. We gained the SOC2 Type2 certification and CSA star Level 2, which goes a long way to assure our customers about our security practices. I believe certifications are a consequence of strengthening our practices and shouldn’t be our primary goal.
2. We have integrated most of our internal products with Okta to ease onboarding and offboarding practices. Thanks to our security team, we have revamped access control and defined how our teams access internal tools.
3. Our Infosec team has also launched whitepapers to tackle the ever-growing data residency requirements across the countries. We made tweaks to our products, such as launching the PII tokenized sending feature that allows our customers to store the data in their region and work with MoEngage solution that exists in other regions.

Stronger Team and Culture

While we worry about developing industry-leading products and technology, the team’s health and satisfaction are also my focus. Our organization’s culture encourages the team to speak up and act on feedback as much as possible. Here are some milestones we achieved in 2022:

1. We re-opened our office in August last year – we followed a hybrid work model, and in-office days were optional. However, teams could experience the difference between WFH and in-office work days. In-person meetings helped with faster decision-making, and knowledge transfer was quicker. The social aspect is the added icing on the cake; teams felt they were not isolated and were part of something bigger.
2. We have also successfully transferred some team members from the Business departments to the Engineering departments. Engineering at MoEngage is always open to the right talent, no matter their background.
3. By November, we had completed our leadership hunt and had a solid team with rich and diverse experience. I am hopeful that we will solve a lot of structural and process gaps with the experienced and talented leadership we have assembled now.

Other Highlights

1. We envision MoEngage as a multi-cloud platform. We did a few POCs on Azure last year and changed our building principles to not look at AWS services as the only way to build the software for our customers. Any decision we take regarding the stack needs to be multi-cloud compatible. We will continue to invest in multi-cloud systems this year.
2. We completed our first open-source project to help those trying to discover if they have alarms on their AWS infrastructure. The link to the project is given here.
3. We had our first hybrid hackathon, with a few teams in the office and others online. This time, we took our hackathon projects to our product roadmap and are close to making them live for our customers. We, as a culture, give space to out-of-the-box ideas and pursue them.

Opportunities 2023

We started last year with plenty to accomplish. We were able to accommodate some ad hoc tasks, which means we missed out on a few planned projects.

1. We aimed to roll out a new stack for data ingestion to all our customers, but we could do it only for 5% of them, which is way behind our targets. A significant learning for us has been how to make the right choices for the tech stack when we alter the fundamental functionality of the company and how to roll it out into production. I made many mistakes, but I learned a lot here, and I am sure our team and individuals in this project have their share of learnings.
2. We could not prioritize the tech stack revamp for Dynamic Product Messaging. The current stack doesn’t scale as efficiently with our ever-growing customer requirements. We need to ensure maintenance twice a month to upkeep the existing stack. We hope to provide a better experience to our customers and team members working on this product in 2023.
3. Our new DC(data center) launch took longer than planned. We identified the problem and launched one more DC in a record time. MoEngage has 5 DCs now, and our SRE team has done a fantastic job of making this happen. They had a lot of patience with these unexpected DC launches, and we have made our DC launches faster.
4. We had to rework our K8s stack to fix its fundamentals and still needed to meet our K8s targets last year. Something we had set as the target again this year, but with some rework already done, we hope we can do better in this area.

At MoEngage, we continue learning, do not shy away from acknowledging failures, and fix the tech as needed. Customer obsession, innovation, ownership, and humility are a few cultural values that define the MoEngage Engineering team.

As I look towards 2023, I want to thank the team that’s been part of all the hits, and I hope we’ll achieve many more milestones together this year.