Set up alerts and notifications for Dataplex Universal Catalog tasks

This page describes how to set up alerts for failed job executions of a Dataplex Universal Catalog task.

To set up alerts, you can use the Logs Explorer in the Google Cloud console or the Monitoring API.

Terminology

  • Incident autoclose duration: You can close an incident automatically if there are no matching log entries within this duration.
  • Time between notifications: You can lower the frequency of repeated notifications by setting a time between the same notification. Log entries that match a filter are skipped during this time. The minimum time you can set is 5 minutes.

Set up notifications using Logs Explorer

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs explorer

  2. Click the Show query toggle.

    1. To set up an alert for a failed job execution of any Dataplex Universal Catalog task in a project, type the following query:

      severity=ERROR
resource.type = "dataplex.googleapis.com/Task"
logName = "projects/PROJECT_NAME/logs/dataplex.googleapis.com%2Fprocess"
jsonPayload.state = "FAILED"

    2. To set up an alert for a failed job execution of a specific Dataplex Universal Catalog task in a project, type the following query:

      severity=ERROR
resource.type = "dataplex.googleapis.com/Task"
resource.labels.task_id = TASK_ID
logName = "projects/PROJECT_NAME/logs/dataplex.googleapis.com%2Fprocess"
jsonPayload.state = "FAILED"

  3. Click Create alert.

  4. In the Alert Policy Name field, enter a name for the new alert.

  5. In the Documentation field, enter the description you want to include in the notification. For more information about the content you can include in the documentation field of an alerting policy, see Using Markdown and variables in documentation templates.

  6. Click Next.

  7. Choose the logs you want to include in the alert, and click Next.

  8. Set the Time between notifications and the Incident autoclose duration, and click Next.

  9. Optional: Select a notification channel. For more information about configuring notification channels using the Google Cloud console, see Manage notification channels.

  10. Click Save.

Set up notifications using Monitoring API

  1. Optional: Enable the Monitoring API and authorize users to access it. For more information, see Enable the Monitoring API.
  2. Optional: Set the roles or permissions needed to create an alerting policy. For more information, see Authorization required to create alerting policies.
  3. Create a notification channel by using either Google Cloud console or Google Cloud CLI.

  4. Get the channel_id:

    gcloud beta monitoring channels list

  5. Create a policy.json file with the following content for log-based-alerting-policy:

    1. Set up an alert for a failed job execution of any Dataplex Universal Catalog task in a project:

      {
"displayName": "Alert Using Monitoring API",
"documentation": {
  "content": "Log-based alert in project PROJECT.",
  "mimeType": "text/markdown"
},

"conditions": [
  {
    "displayName": "Log match condition: Failed Job Execution",
    "conditionMatchedLog": {
      "filter": "severity=\"ERROR\" resource.type = \"dataplex.googleapis.com/Task\" logName = \"projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess\" jsonPayload.state = \"FAILED\"",
    },
  }
],
"combiner": "OR",

"alertStrategy": {
  "notificationRateLimit": {
    "period": "300s"
  },
  "autoClose": "604800s",
},

"notificationChannels": [
  "projects/PROJECT_ID/notificationChannels/CHANNEL_ID"
]
}

    2. Set up an alert for a failed job execution for a specific Dataplex Universal Catalog task in a project:

      {
"displayName": "Alert Using Monitoring API",
"documentation": {
  "content": "Log-based alert in project PROJECT.",
  "mimeType": "text/markdown"
},

"conditions": [
  {
    "displayName": "Log match condition: Failed Job Execution",
    "conditionMatchedLog": {
      "filter": "severity=\"ERROR\" resource.type = \"dataplex.googleapis.com/Task\" logName = \"projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess\" jsonPayload.state = \"FAILED\" resource.labels.task_id = \"TASK_ID\"",
    },
  }
],
"combiner": "OR",

"alertStrategy": {
  "notificationRateLimit": {
    "period": "300s"
  },
  "autoClose": "604800s",
},

"notificationChannels": [
  "projects/PROJECT_ID/notificationChannels/CHANNEL_ID"
]
}

  6. Pass the policy.json file to the gcloud CLI:

    gcloud alpha monitoring policies create --policy-from-file="policy.json"

    If successful, the command returns the resource name of the new policy. For example:

    Created alert policy [projects/PROJECT_ID/alertPolicies/*POLICY_ID*].

What's next