Google Cloud Storage (GCS) Exports

This feature is part of the Streams paid add-on. To check what’s available for your plan, refer to Plans and Pricing. To enable it for your account, contact your MoEngage Customer Success Manager (CSM) or the Support team.

Overview

Google Cloud Storage is an enterprise public cloud storage platform that can house large unstructured data sets.

How Does MoEngage Work with GCS?

The MoEngage and GCS integration exports Campaign Interaction Events to your Google Cloud Storage buckets. You can then import these files into BigQuery for further processing and analysis.

Integration

Prerequisites

Ensure you have a Google Cloud Platform account with permission to create a new bucket and a service account.
If your system requires IP whitelisting, you must whitelist MoEngage IPs. For more information, refer here.
Ensure you have a GCS Connection on the App marketplace.

Set Up Your Export

For the full list of events and attributes, refer to the Data Export Glossary.
The event_time of the exported events is always in UTC.

Add a New GCS Export on the MoEngage App Marketplace

On your MoEngage dashboard, go to the App Marketplace.
Search for Google Cloud Storage.
Go to the Integrate tab, and click + Add Integration.
Give your connection an identifiable name.
Enter your bucket name.
Optionally, select your bucket region. You can find this on your Bucket Details page.
Upload the service account credential JSON file that you downloaded earlier. You can verify your Client ID and Client Email upon upload.
Select which events you want to export. By default, each event is exported with standard attributes.
Optionally, you can test the connection. MoEngage attempts to verify whether the details you entered are correct. If there is an issue, an error is displayed that you can rectify.

Verify You Are Receiving Data Correctly

MoEngage uploads files using the following path structure:

gs://<your-bucket-name>/event-exports/<your_app_name>/<connection_id>/export_day=YYYY-MM-DD/export_hour=HH/<file-name>.json.gz

Go to your bucket and verify that the files are present:

Verifying that exported files are present in the GCS bucket

Export Process and Frequency

MoEngage dumps the data at hourly intervals.

Sample File

The exported files are in the .json.gz format. Download a sample file to inspect the structure. The exact contents will differ based on the events and attributes you opt to export.

JSON

{
  "app_name": "App Name",
  "export_hour": "6",
  "db_name": "MoEngage App Name",
  "event": {
    "uid": "user id",
    "event_type": "CAMPAIGN_EVENT",
    "event_code": "MOE_EMAIL_SENT",
    "event_name": "Email Sent",
    "user_attributes": {
      "moengage_user_id": "userid1",
      "user_attr1": "value1",
      "user_attr2": "value2"
    },
    "event_source": "MOENGAGE",
    "email_id": "[email protected]",
    "event_attributes": {
      "campaign_name": "Campaign name",
      "moe_campaign_tags": [
        "Tag1",
        "Tag2"
      ],
      "campaign_type": "GENERAL",
      "moe_campaign_channel": "Email",
      "moe_delivery_type": "One Time",
      "campaign_id": "moengage campaign id",
      "campaign_channel": "EMAIL"
    },
    "event_uuid": "3b918064-08dc-47ae-b674-c23797475a8d",
    "event_time": 1653022310,
    "device_attributes": {
      "moengage_device_id": "device id if applicable"
    }
  },
  "export_day": "2022-05-20"
}

For a list of supported events and attributes, refer to our Data Export Glossary.

Export Failure Policy

MoEngage pauses an export that keeps failing. While it is paused, MoEngage collects the events the export would have delivered, and you replay them after you fix the problem.

When MoEngage Pauses an Export

Exports to Google Cloud Storage run hourly. MoEngage will retry an export multiple times before marking it as failed. The export will be paused after 3 consecutive scheduled export failures. When MoEngage pauses an export, it alerts you and shows the export as Paused on the App Marketplace.

What Happens While an Export Is Paused

A paused Google Cloud Storage export on the Integrate tab, with Export status turned off and the Paused label beside it

MoEngage collects the events the export would have delivered and keeps only the most recent 7 days. If the export stays paused for longer than a week, MoEngage drops the oldest days as newer ones come in. If an export pauses more than once before you replay, MoEngage combines the data from all those pauses, provided it falls within the 7-day window.

Resume an Export

Find the Failure Reason

On the App Marketplace, click the Integrate tab, and then hover over the Paused label on the export. MoEngage names the reason for the failure, such as a missing write permission on the destination bucket, and tells you how to fix it.

The Paused label hover on the Integrate tab, showing the failure reason and what you need to do

Enable the Export

Fix the cause at your end, and then turn on Export status for the export. Do this as soon as you can, because the 7-day window keeps moving while the export stays paused. When you enable the export, MoEngage stops collecting and sets aside what it has: the last 7 days of paused data, or all of it if the pause was shorter. This is the data you can replay. It stays fixed from then on, and does not reduce further even if you wait to replay it.

Replay Paused Data

Request a Replay

MoEngage saves the data it collected while the export was paused. An export with data waiting shows a Replay available label on the Integrate tab. To replay, perform the following steps:

Hover over the Replay available label, or open the kebab menu on the export.
Click Replay paused data.
In the Replay paused data dialog box, click Confirm.

A replay sends everything MoEngage collected while the export was paused. MoEngage backfills the data into the date folders it would have written to at the time, rather than writing it to a new location. You can replay at any time until 30 days from the date the oldest paused data was collected. After that, the data is no longer available.

Replay before the export pauses again. If it pauses a second time before you replay, MoEngage adds the new paused data to the data still waiting from the first pause. The combined data is then trimmed back to the most recent 7 days, so if the two pauses are more than 7 days apart, the older data is lost.For example, your export pauses on day 1. You enable it but do not replay. It pauses again on day 20. Since that is 19 days later, the day 1 data falls outside the 7 day window and is gone, even though its 30 day replay window had not expired. If the second pause had happened within 7 days of the first, the day 1 data would still be included.

Replay Results

When the replay finishes, MoEngage emails the user who requested it with one of the following results:

Result	What it means
Successful	MoEngage delivered all the collected data. The replay option is no longer available.
Partially Successful	MoEngage delivered some of the collected data, and then hit an error. The replay stops and MoEngage pauses the export again. The email states why the replay did not finish. Fix the cause, enable the export, and then replay again. For the data you can replay, refer to What Happens While an Export Is Paused.
Failed	MoEngage delivered none of the collected data, because of an error. The replay stops and MoEngage pauses the export again. The email states why the replay did not finish. Fix the cause, enable the export, and then replay again. For the data you can replay, refer to What Happens While an Export Is Paused.

You can run only one replay per export at a time.

Avoid Duplicate Data

Every replay sends the whole collection again, because MoEngage does not deduplicate. If you replay more than once, deduplicate at your end on the event_uuid field, which is unique for each event. For the fields in the export, refer to the Data Export Glossary.

Export Paused Alerts

MoEngage alerts you when it pauses an export, so you can fix the issue and replay the data collected while the export was paused. MoEngage subscribes workspace admins to the Data Exports system alert by default, and adds the user who creates or updates an export. To subscribe or unsubscribe anyone else, go to the Alert Management page. Subscribing or unsubscribing changes notifications only for that person, and does not affect anyone else’s subscription.

Frequently Asked Questions

What does the Replay available label mean?

MoEngage is holding data it collected while the export was paused. Hover over the label or open the kebab menu, and then click Replay paused data.

My export was paused for three weeks. Why can I replay only part of it?

MoEngage keeps only the most recent 7 days of paused data, so a three-week pause leaves only the last 7 days to replay.

Why did my replay come back as Partially Successful?

MoEngage delivered some of the data and not the rest, usually because the destination rejected part of it. Fix the cause and replay again. The next replay sends the whole collection, not only the part that failed.

Can I run two replays at the same time?

No. You can run only one replay per export at a time. Wait for the result email before you request another.

I replayed twice and I see duplicate rows. Why?

Every replay sends the whole collection again, because MoEngage does not deduplicate. Deduplicate at your end on the event_uuid field, which is unique for each event.

How soon will I start getting the data?

It typically takes up to 30 minutes for the automated data exports to be set up for your account and for you to start seeing data there.

Can I export additional events at a later time?

Yes, if you need to export new events at a later time, you can always edit your configuration on the App Marketplace and get the new events added to your exports.

Can I export historical data using Automated Data Exports?

As of now, you cannot export historical data using Automated Data Exports. Once configured, you will start seeing data for each event from the time you enabled your exports.

Is there any pricing involved for Automated Data Exports?

If your billing plan does not cover this already and you need to get this enabled at a later time, you would need to work with your Customer Success Manager to get this included in your billing plan. Based on the amount of data that MoEngage will place in your Data warehouse/Partner/API on a monthly basis, this could lead to additional charges.

Does Data Exports guarantee the latest user attributes?

No, Data Exports is primarily built to export your Events in near real-time and user attributes in MoEngage are updated asynchronously so it is currently not possible to guarantee the latest values of user attributes in the exports.

​Overview

​How Does MoEngage Work with GCS?

​Integration

​Prerequisites

​Set Up Your Export

​Add a New GCS Export on the MoEngage App Marketplace

​Verify You Are Receiving Data Correctly

​Export Process and Frequency

​Sample File

​Export Failure Policy

​When MoEngage Pauses an Export

​What Happens While an Export Is Paused

​Resume an Export

​Find the Failure Reason

​Enable the Export

​Replay Paused Data

​Request a Replay

​Replay Results

​Avoid Duplicate Data

​Export Paused Alerts

​Frequently Asked Questions

Overview

How Does MoEngage Work with GCS?

Integration

Prerequisites

Set Up Your Export

Add a New GCS Export on the MoEngage App Marketplace

Verify You Are Receiving Data Correctly

Export Process and Frequency

Sample File

Export Failure Policy

When MoEngage Pauses an Export

What Happens While an Export Is Paused

Resume an Export

Find the Failure Reason

Enable the Export

Replay Paused Data

Request a Replay

Replay Results

Avoid Duplicate Data

Export Paused Alerts

Frequently Asked Questions