BigQuery Engine for Apache Flink autoscaling

Autoscaling enables BigQuery Engine for Apache Flink to choose the appropriate number of task slots for your job, adding and removing slots as needed. The autoscaler calculates the required number of task slots for each Flink job, with the goal of keeping up with the input load while using the minimum number of resources necessary.

Parallelism is an estimate of the number of task slots needed by a job to most efficiently process data at any given time. To determine the number of task slots needed for a job, BigQuery Engine for Apache Flink uses an algorithm that analyses the parallelism per task in a job.

The results are constrained by resource limitations per job, per deployment, and per project. You can configure some of the limits, such as quotas. Other limits are based on the project quota limits.

Jobs use autoscaling by default. You can set the minimum and maximum slots for a job, or you can disable autoscaling for a job and use a fixed number of slots.

Benefits

Autoscaling has the following potential benefits.

• Allows BigQuery Engine for Apache Flink jobs to process data more efficiently.
• Improves parallel processing by adjusting the number of slots available to run tasks in parallel.
• Promotes efficient resource usage, which might reduce your costs.

Support and limitations

• Supports streaming pipelines.
• Scaling up is constrained by your slot quota limit. Quotas are enforced at the deployment level, not the job level.
• You can't assign more task slots to your deployment than are available in the Google Cloud project.
• The BigQuery Engine for Apache Flink autoscaler works best with Apache Kafka sources. For other I/Os, implement the new Apache Flink source interface and the Apache Flink standardized connector metrics.

How it works

Autotuning dynamically scales the task slot count used by a job up or down depending on resource requirements. Increasing the number of slots allows more tasks to run in parallel. As tasks complete and the slots are no longer needed, the slot count scales down. An algorithm determines how many task slots each job needs.

To determine the appropriate slot count, BigQuery Engine for Apache Flink uses the following metrics:

• The average number of milliseconds in a second spent processing data, excluding backpressure
• The average number of records coming in per second
• The average number of records going out per second
• The size of the backlog, that is, the number of pending records
• The rate of growth of the backlog
• The total number of splits that the source can process in parallel

Configure autoscaling

To configure autoscaling, set the following parameters on your deployments and jobs.

Deployments

When you create a deployment, set the maximum number of slots. This value limits the total number of slots that are available for all jobs in the deployment.

gcloud alpha managed-flink deployments create ... \
  --max-slots=SLOT_NUMBER

Replace SLOT_NUMBER with the number of slots assigned to the deployment. If the value is greater than the number of task slots available in the Google Cloud project, the request is rejected.

Jobs

When you create a new job, autoscaling is enabled by default. Set the minimum and maximum number of slots for the job.

gcloud alpha managed-flink jobs create ... \
--min-parallelism=MINIMUM_SLOTS \
--max-parallelism=MAXIMUM_SLOTS

Replace the following variables:

• MINIMUM_SLOTS: the minimum number of task slots available to your job
• MAXIMUM_SLOTS: the maximum number of task slots available to your job

If you create a job in an existing deployment, these values cannot exceed the maximum number of slots assigned to the deployment.

Disable autoscaling

You can disable autoscaling when you create a job, and instead assign a fixed number of task slots to the job.

gcloud alpha managed-flink jobs create ... \
  --autotuning-mode=fixed \
  --parallelism=SLOTS \

Replace SLOTS with the number of task slots available to your job.

Update autoscaling

gcloud alpha managed-flink jobs update JOB_ID \
  --project=PROJECT_ID \
  --location=REGION \
  --min-parallelism=MINIMUM_SLOTS \
  --max-parallelism=MAXIMUM_SLOTS

gcloud alpha managed-flink jobs update JOB_ID \
  --project=PROJECT_ID \
  --location=REGION \
  --autotuning-mode=fixed \
  --parallelism=SLOTS

Replace the following variables:

• JOB_ID: the ID of your BigQuery Engine for Apache Flink job
• PROJECT_ID: your BigQuery Engine for Apache Flink project ID
• REGION: the region that your BigQuery Engine for Apache Flink job is in
• MINIMUM_SLOTS: the minimum number of task slots available to your job
• MAXIMUM_SLOTS: the maximum number of task slots available to your job
• SLOTS: the number of task slots available to your job

Verify that autoscaling is enabled

gcloud alpha managed-flink jobs describe \
  JOB_ID \
  --location=REGION

autotuningConfig:
  throughputBased:
    maxParallelism: NUMBER
    minParallelism: NUMBER
    parallelism: NUMBER

Monitor

Use the Google Cloud console to monitor autoscaling. You can view the autoscaling monitoring chart in the BigQuery Engine for Apache Flink monitoring interface in the Metrics tab on the Job details page. The charts display metrics over the duration of a job and includes the following information:

• The current parallelism
• The recommended parallelism
• The minimum and maximum allowed task slots

For more information, see Autoscaling metrics.

BigQuery Engine for Apache Flink job-level autoscaling metrics are also exported to Cloud Monitoring. Use Metrics Explorer to create charts.