FlexTemplateRuntimeEnvironment(
mapping=None, *, ignore_unknown_fields=False, **kwargs
)The environment values to be set at runtime for flex template.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
Attributes |
|
|---|---|
| Name | Description |
num_workers |
int
The initial number of Google Compute Engine instances for the job. |
max_workers |
int
The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000. |
zone |
str
The Compute Engine `availability zone |
service_account_email |
str
The email address of the service account to run the job as. |
temp_location |
str
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
|
machine_type |
str
The machine type to use for the job. Defaults to the value from the template if not specified. |
additional_experiments |
MutableSequence[str]
Additional experiment flags for the job. |
network |
str
Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default". |
subnetwork |
str
Subnetwork to which VMs will be assigned, if desired. You can specify a subnetwork using either a complete URL or an abbreviated path. Expected to be of the form "https://www.googleapis.com/compute/v1/projects/HOST_PROJECT_ID/regions/REGION/subnetworks/SUBNETWORK" or "regions/REGION/subnetworks/SUBNETWORK". If the subnetwork is located in a Shared VPC network, you must use the complete URL. |
additional_user_labels |
MutableMapping[str, str]
Additional user labels to be specified for the job. Keys and values must follow the restrictions specified in the `labeling restrictions |
kms_key_name |
str
Name for the Cloud KMS key for the job. Key format is: projects/ |
ip_configuration |
google.cloud.dataflow_v1beta3.types.WorkerIPAddressConfiguration
Configuration for VM IPs. |
worker_region |
str
The Compute Engine region (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1". Mutually exclusive with worker_zone. If neither worker_region nor worker_zone is specified, default to the control plane's region. |
worker_zone |
str
The Compute Engine zone (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1-a". Mutually exclusive with worker_region. If neither worker_region nor worker_zone is specified, a zone in the control plane's region is chosen based on available capacity. If both worker_zone and zone are set,
worker_zone takes precedence.
|
enable_streaming_engine |
bool
Whether to enable Streaming Engine for the job. |
flexrs_goal |
google.cloud.dataflow_v1beta3.types.FlexResourceSchedulingGoal
Set FlexRS goal for the job. https://cloud.google.com/dataflow/docs/guides/flexrs |
staging_location |
str
The Cloud Storage path for staging local files. Must be a valid Cloud Storage URL, beginning with gs://.
|
sdk_container_image |
str
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines. |
disk_size_gb |
int
Worker disk size, in gigabytes. |
autoscaling_algorithm |
google.cloud.dataflow_v1beta3.types.AutoscalingAlgorithm
The algorithm to use for autoscaling |
dump_heap_on_oom |
bool
If true, when processing time is spent almost entirely on garbage collection (GC), saves a heap dump before ending the thread or process. If false, ends the thread or process without saving a heap dump. Does not save a heap dump when the Java Virtual Machine (JVM) has an out of memory error during processing. The location of the heap file is either echoed back to the user, or the user is given the opportunity to download the heap file. |
save_heap_dumps_to_gcs_path |
str
Cloud Storage bucket (directory) to upload heap dumps to. Enabling this field implies that dump_heap_on_oom is set
to true.
|
launcher_machine_type |
str
The machine type to use for launching the job. The default is n1-standard-1. |
enable_launcher_vm_serial_port_logging |
bool
If true serial port logging will be enabled for the launcher VM. |
streaming_mode |
google.cloud.dataflow_v1beta3.types.StreamingMode
Optional. Specifies the Streaming Engine message processing guarantees. Reduces cost and latency but might result in duplicate messages committed to storage. Designed to run simple mapping streaming ETL jobs at the lowest cost. For example, Change Data Capture (CDC) to BigQuery is a canonical use case. For more information, see `Set the pipeline streaming mode |
Classes
AdditionalUserLabelsEntry
AdditionalUserLabelsEntry(mapping=None, *, ignore_unknown_fields=False, **kwargs)The abstract base class for a message.
| Parameters | |
|---|---|
| Name | Description |
kwargs |
dict
Keys and values corresponding to the fields of the message. |
mapping |
Union[dict,
A dictionary or message to be used to determine the values for this message. |
ignore_unknown_fields |
Optional(bool)
If True, do not raise errors for unknown fields. Only applied if |