BigQuery Engine for Apache Flink logging

BigQuery Engine for Apache Flink provides logs so that you can use the Google Cloud console to monitor your deployments and jobs. You can monitor your log files either on the BigQuery Engine for Apache Flink monitoring interface or in Logs Explorer.

View deployment logs

To view deployment logs in the Google Cloud console, follow these steps.

  1. Sign in to the Google Cloud console.
  2. Select your Google Cloud project.
  3. Open the navigation menu and select BigQuery Engine for Apache Flink.
  4. Click Deployments.
  5. In the deployments list, click the name of the deployment.
  6. Click the Logs tab.

To view additional information about each log entry, click Expand this log entry.

To filter the messages that appear in the logs panel, in the Filter field, enter text, and then search the logs.

View job logs

To view job logs in the Google Cloud console, follow these steps. The job logs panel filters the log text by using the job ID. The job logs might not be complete.

  1. Sign in to the Google Cloud console.
  2. Select your Google Cloud project.
  3. Open the navigation menu and select BigQuery Engine for Apache Flink.
  4. Click Jobs.
  5. In the jobs list, click the name of the job.
  6. To expand the logs panel, in the Logs header, click Show.

To view additional information about each log entry, click Expand this log entry.

To filter the messages that appear in the logs panel, in the Filter field, enter text, and then search the logs.

The job logs for a specific BigQuery Engine for Apache Flink job only contain a subset of the log messages. If you need more detailed log messages from your job, including error details, look at the logs in Logs Explorer.

Use Logs Explorer

To view your BigQuery Engine for Apache Flink logs in Logs Explorer, in the Logs panel, click View in Logs Explorer.

  • In Logs Explorer, the Log fields toggle controls whether you see the panel with different log types.

  • On the Logs Explorer page, the query might filter the logs. To remove filters, make sure the Show query toggle is selected, and edit the query.

Deployment logs

To see all logs available for a deployment, follow these steps:

  1. In the Query field, enter the following query:

    resource.labels.deployment_id="DEPLOYMENT_ID"

    Replace DEPLOYMENT_ID with the ID of your deployment.

  2. Click Run query.

  3. If you use this query and don't see logs for your deployment, click Edit time.

  4. Adjust the start time and end time, and then click Apply.

Job logs

To see all logs available for a job, follow these steps:

  1. In the Query field, enter the following query:

    resource.labels.job_id="JOB_ID"

    Replace JOB_ID with the ID of your job.

  2. Click Run query.

  3. If you use this query and don't see logs for your job, click Edit time.

  4. Adjust the start time and end time, and then click Apply.

Log storage and retention

Operational logs are stored in the _Default log bucket. The logging API service name is managedflink.googleapis.com. For more information about the Google Cloud monitored resource types and services used in Cloud Logging, see Monitored resources and services.

For details about how long log entries are retained by Logging, see the retention information in Quotas and limits: Logs retention periods.