Snowflake Exports

This feature is part of the Streams paid add-on. To check what’s available for your plan, refer to Plans and Pricing. To enable it for your account, contact your MoEngage Customer Success Manager (CSM) or the Support team.

Overview

Snowflake is a data platform that allows organizations to store, analyze, and process large volumes of structured and semi-structured data in a highly scalable and efficient manner. With its unique architecture, Snowflake allows organizations to consolidate their data, perform quick analytics, and gain valuable data-driven insights accessible to all users.

How Does MoEngage Work with Snowflake?

The MoEngage and Snowflake integration exports Campaign Interaction Events data directly to your Snowflake tables.

Integration

Prerequisites

Ensure you have a Snowflake account with relevant permissions to create/edit Snowflake instances, roles, and policies. It is optional to create a new user; you must have access to create a new user in your Snowflake database.
If your system requires IP whitelisting, you must whitelist MoEngage IPs. For more information, refer here.
Ensure you have a Snowflake Connection on the App marketplace.
The Add Integration option for Snowflake in the App Marketplace appears only after the Streams add-on is enabled for your workspace (see note above).

Set Up Your Export

For the full list of events and attributes, refer to the Data Export Glossary.
The event_time of the exported events is always in UTC.

Step 1: Grant the Required Permissions

MoEngage will require WRITE access to your database so that we can write data into your Snowflake database. You can either grant these permissions to an existing database user or choose to create a new dedicated database user for MoEngage:

-- Create a role for the MoEngage user
CREATE ROLE MOENGAGE_ROLE;
-- Allow the user to run queries in the warehouse
GRANT USAGE ON WAREHOUSE <YOUR_WAREHOUSE_NAME> TO ROLE MOENGAGE_ROLE;
-- Let the user see this database
GRANT USAGE ON DATABASE <YOUR_DATABASE_NAME> TO ROLE MOENGAGE_ROLE;
-- Let the user see all schemas in this database
GRANT USAGE ON SCHEMA <YOUR_DATABASE_NAME>.<YOUR_SCHEMA_NAME> TO ROLE MOENGAGE_ROLE;
-- Let the user create a table in your schema
GRANT CREATE TABLE ON SCHEMA <YOUR_DATABASE_NAME>.<YOUR_SCHEMA_NAME> TO ROLE MOENGAGE_ROLE;
-- Let the user insert data in your tables
GRANT INSERT, SELECT ON ALL TABLES IN SCHEMA <YOUR_DATABASE_NAME>.<YOUR_SCHEMA_NAME> TO ROLE MOENGAGE_ROLE;

-- Create the MoEngage user (MOENGAGE_USER is username here)
CREATE USER MOENGAGE_USER WITH DEFAULT_ROLE = MOENGAGE_ROLE DEFAULT_WAREHOUSE = <YOUR_WAREHOUSE_NAME> PASSWORD = '<YOUR_PASSWORD>';
-- Grant the role to the user
GRANT ROLE MOENGAGE_ROLE TO USER MOENGAGE_USER;

Remember to change the following values:

<YOUR_WAREHOUSE_NAME> - The name of the warehouse you want MoEngage to use while executing queries.
<YOUR_DATABASE_NAME> - The name of the database you want to import data from.
<YOUR_PASSWORD> - Give a strong password to the newly created database user.

Step 2: Create a New Export in the App Marketplace

For the full list of events and attributes exported, refer to the Data Exports Glossary.
Please note that the event_time of the exported events will always be in UTC.

On your MoEngage dashboard, navigate to App Marketplace > Snowflake.

Click Integrate > Add Integration.
Select your Snowflake connection.
Select which schema MoEngage will use to create new tables and export your events into.
Select the events and additional user properties you want to export to Snowflake.
Click Save Export.

MoEngage will start dumping events to the newly created table every hour.

Export Process and Frequency

After you save the export, MoEngage creates a destination table in your schema. New event data is then appended to this same table at hourly intervals:

Initial action: MoEngage creates the table when the first export is triggered.
Recurring action: MoEngage adds (appends) data to the same table every hour.

MoEngage exports the data at hourly intervals.

Sample Table Structure

MoEngage creates a single table in your schema using the following format: moe_events_<table_creation_timestamp>

This table is created once per connection. All hourly exports and different event types configured under a single connection are consolidated and appended to this same table.

The overall structure of the table remains fixed for all types of events. Any new user property or event property that is exported will be included in the user_attributes or event_attributes column, respectively. All events are consolidated in a single table containing necessary information, such as event names and event times, for easy analysis:

Field	Type	NL
`app_name`	VARCHAR	NN
`event_name`	VARCHAR	NN
`event_code`	VARCHAR	NN
`event_uuid`	VARCHAR	NN
`event_time`	TIMESTAMP_NTZ	NN
`event_type`	VARCHAR	NN
`event_source`	VARCHAR	NN
`uid`	VARCHAR	NULL
`user_attributes_moengage_user_id`	VARCHAR	NULL
`device_attributes_moengage_device_id`	VARCHAR	NULL
`push_id`	VARCHAR	NULL
`email_address`	VARCHAR	NULL
`mobile_number`	VARCHAR	NULL
`event_attributes_app_version`	VARCHAR	NULL
`event_attributes_sdk_version`	VARCHAR	NULL
`event_attributes_platform`	VARCHAR	NULL
`event_attributes_campaign_id`	VARCHAR	NULL
`event_attributes_campaign_name`	VARCHAR	NULL
`event_attributes_campaign_type`	VARCHAR	NULL
`event_attributes_campaign_channel`	VARCHAR	NULL
`event_attributes_readable_campaign_id`	VARCHAR	NULL
`event_attributes_parent_campaign_id`	VARCHAR	NULL
`event_attributes_parent_flow_id`	VARCHAR	NULL
`event_attributes_parent_flow_name`	VARCHAR	NULL
`event_attributes_variation_id`	VARCHAR	NULL
`event_attributes_locale_id`	VARCHAR	NULL
`event_attributes_locale_name`	VARCHAR	NULL
`event_attributes_url`	VARCHAR	NULL
`event_attributes_timestamp`	TIMESTAMP_NTZ	NULL
`event_attributes_first_session`	BOOLEAN	NULL
`event_attributes_logged_in_status`	VARCHAR	NULL
`event_attributes`	VARIANT	NULL
`user_attributes`	VARIANT	NULL
`device_attributes`	VARIANT	NULL

Export Failure Policy

MoEngage pauses an export that keeps failing. While it is paused, MoEngage collects the events the export would have delivered, and you replay them after you fix the problem.

When MoEngage Pauses an Export

Exports to Snowflake run hourly. MoEngage will retry an export multiple times before marking it as failed. The export will be paused after 3 consecutive scheduled export failures. When MoEngage pauses an export, it alerts you and shows the export as Paused on the App Marketplace.

What Happens While an Export Is Paused

A paused Snowflake export on the Integrate tab, with Export status turned off and the Paused label beside it

MoEngage collects the events the export would have delivered and keeps only the most recent 7 days. If the export stays paused for longer than a week, MoEngage drops the oldest days as newer ones come in. If an export pauses more than once before you replay, MoEngage combines the data from all those pauses, provided it falls within the 7-day window.

Resume an Export

Find the Failure Reason

On the App Marketplace, click the Integrate tab, and then hover over the Paused label on the export. MoEngage names the reason for the failure, such as a missing write permission on the destination table, and tells you how to fix it.

The Paused label hover on the Integrate tab, showing the failure reason and what you need to do

Enable the Export

Fix the cause at your end, and then turn on Export status for the export. Do this as soon as you can, because the 7-day window keeps moving while the export stays paused. When you enable the export, MoEngage stops collecting and sets aside what it has: the last 7 days of paused data, or all of it if the pause was shorter. This is the data you can replay. It stays fixed from then on, and does not reduce further even if you wait to replay it.

Replay Paused Data

Request a Replay

MoEngage saves the data it collected while the export was paused. An export with data waiting shows a Replay available label on the Integrate tab. To replay, perform the following steps:

Hover over the Replay available label, or open the kebab menu on the export.
Click Replay paused data.
In the Replay paused data dialog box, click Confirm.

A replay sends everything MoEngage collected while the export was paused. MoEngage appends the data to the destination table. You can replay at any time until 30 days from the date the oldest paused data was collected. After that, the data is no longer available.

Replay before the export pauses again. If it pauses a second time before you replay, MoEngage adds the new paused data to the data still waiting from the first pause. The combined data is then trimmed back to the most recent 7 days, so if the two pauses are more than 7 days apart, the older data is lost.For example, your export pauses on day 1. You enable it but do not replay. It pauses again on day 20. Since that is 19 days later, the day 1 data falls outside the 7 day window and is gone, even though its 30 day replay window had not expired. If the second pause had happened within 7 days of the first, the day 1 data would still be included.

Replay Results

When the replay finishes, MoEngage emails the user who requested it with one of the following results:

Result	What it means
Successful	MoEngage delivered all the collected data. The replay option is no longer available.
Partially Successful	MoEngage delivered some of the collected data, and then hit an error. The replay stops and MoEngage pauses the export again. The email states why the replay did not finish. Fix the cause, enable the export, and then replay again. For the data you can replay, refer to What Happens While an Export Is Paused.
Failed	MoEngage delivered none of the collected data, because of an error. The replay stops and MoEngage pauses the export again. The email states why the replay did not finish. Fix the cause, enable the export, and then replay again. For the data you can replay, refer to What Happens While an Export Is Paused.

You can run only one replay per export at a time.

Avoid Duplicate Data

Every replay sends the whole collection again, because MoEngage does not deduplicate. If you replay more than once, deduplicate at your end on the event_uuid field, which is unique for each event. For the fields in the export, refer to the Data Export Glossary.

Export Paused Alerts

MoEngage alerts you when it pauses an export, so you can fix the issue and replay the data collected while the export was paused. MoEngage subscribes workspace admins to the Data Exports system alert by default, and adds the user who creates or updates an export. To subscribe or unsubscribe anyone else, go to the Alert Management page. Subscribing or unsubscribing changes notifications only for that person, and does not affect anyone else’s subscription.

Frequently Asked Questions

What does the Replay available label mean?

MoEngage is holding data it collected while the export was paused. Hover over the label or open the kebab menu, and then click Replay paused data.

My export was paused for three weeks. Why can I replay only part of it?

MoEngage keeps only the most recent 7 days of paused data, so a three-week pause leaves only the last 7 days to replay.

Why did my replay come back as Partially Successful?

MoEngage delivered some of the data and not the rest, usually because the destination rejected part of it. Fix the cause and replay again. The next replay sends the whole collection, not only the part that failed.

Can I run two replays at the same time?

No. You can run only one replay per export at a time. Wait for the result email before you request another.

I replayed twice and I see duplicate rows. Why?

Every replay sends the whole collection again, because MoEngage does not deduplicate. Deduplicate at your end on the event_uuid field, which is unique for each event.

How soon will I start getting the data?

It typically takes up to 30 minutes for the automated data exports to be set up for your account and for you to start seeing data there.

Can I export additional events at a later time?

Yes, if you need to export new events at a later time, you can always edit your configuration on the App Marketplace and get the new events added to your exports.

Can I export historical data using Automated Data Exports?

As of now, you cannot export historical data using Automated Data Exports. Once configured, you will start seeing data for each event from the time you enabled your exports.

Is there any pricing involved for Automated Data Exports?

If your billing plan does not cover this already and you need to get this enabled at a later time, you would need to work with your Customer Success Manager to get this included in your billing plan. Based on the amount of data that MoEngage will place in your Data warehouse/Partner/API on a monthly basis, this could lead to additional charges.

Does Data Exports guarantee the latest user properties?

No, Data Exports is primarily built to export your Events in near real-time and user properties in MoEngage are updated asynchronously so it is currently not possible to guarantee the latest values of user properties in the exports.

​Overview

​How Does MoEngage Work with Snowflake?

​Integration

​Prerequisites

​Set Up Your Export

​Step 1: Grant the Required Permissions

​Step 2: Create a New Export in the App Marketplace

​Export Process and Frequency

​Sample Table Structure

​Export Failure Policy

​When MoEngage Pauses an Export

​What Happens While an Export Is Paused

​Resume an Export

​Find the Failure Reason

​Enable the Export

​Replay Paused Data

​Request a Replay

​Replay Results

​Avoid Duplicate Data

​Export Paused Alerts

​Frequently Asked Questions

Overview

How Does MoEngage Work with Snowflake?

Integration

Prerequisites

Set Up Your Export

Step 1: Grant the Required Permissions

Step 2: Create a New Export in the App Marketplace

Export Process and Frequency

Sample Table Structure

Export Failure Policy

When MoEngage Pauses an Export

What Happens While an Export Is Paused

Resume an Export

Find the Failure Reason

Enable the Export

Replay Paused Data

Request a Replay

Replay Results

Avoid Duplicate Data

Export Paused Alerts

Frequently Asked Questions