Dataflow estimated cost

The Estimated Cost page in the Google Cloud console shows the estimated cost of your current Dataflow job. Estimated costs are calculated by multiplying the resource usage metrics as shown in Cloud Monitoring by the price of those resources in the job region.

To view the estimated cost for a job, perform the following steps:

  1. In the Google Cloud console, go to the Dataflow > Jobs page.

    Go to Jobs

  2. Select a job.

  3. Click the Cost tab.

Use cost monitoring

Job cost estimates are available for both batch and streaming jobs. The Estimated Cost page in the Google Cloud console provides the following information:

  • Details about which resources contribute to the job cost and by how much. Resources include vCPUs, memory, Dataflow Shuffle data processed or Streaming Engine data processed, and SSD and HDD disk usage.
  • Costs over specific time windows, such as: time since the job started, the previous hour, the last 24 hours, the preceding seven days, and a user-specified time range.

You can use monitoring alerts to get notifications when your job costs cross a specified threshold. You can also use alerts to make changes to your jobs, such as stopping or canceling jobs, based on the thresholds that you set.

To create a Cloud Monitoring alert rule, click Create alert. For instructions about how to configure these alerts, see Use Cloud Monitoring for Dataflow pipelines.

Limitations

Dataflow cost monitoring does not support Dataflow Prime jobs and GPU metrics.