Reference documentation and code samples for the Google Cloud Dataflow V1beta3 Client class Environment.
Describes the environment in which a Dataflow Job runs.
Generated from protobuf message google.dataflow.v1beta3.Environment
Namespace
Google \ Cloud \ Dataflow \ V1beta3Methods
__construct
Constructor.
| Parameters | |
|---|---|
| Name | Description |
data |
array
Optional. Data for populating the Message object. |
↳ temp_storage_prefix |
string
The prefix of the resources the system should use for temporary storage. The system will append the suffix "/temp-{JOBNAME} to this resource prefix, where {JOBNAME} is the value of the job_name field. The resulting bucket and object prefix is used as the prefix of the resources used to store temporary data needed during the job execution. NOTE: This will override the value in taskrunner_settings. The supported resource type is: Google Cloud Storage: storage.googleapis.com/{bucket}/{object} bucket.storage.googleapis.com/{object} |
↳ cluster_manager_api_service |
string
The type of cluster manager API to use. If unknown or unspecified, the service will attempt to choose a reasonable default. This should be in the form of the API service name, e.g. "compute.googleapis.com". |
↳ experiments |
array
The list of experiments to enable. This field should be used for SDK related experiments and not for service related experiments. The proper field for service related experiments is service_options. |
↳ service_options |
array
Optional. The list of service options to enable. This field should be used for service related experiments only. These experiments, when graduating to GA, should be replaced by dedicated fields or become default (i.e. always on). |
↳ service_kms_key_name |
string
Optional. If set, contains the Cloud KMS key identifier used to encrypt data at rest, AKA a Customer Managed Encryption Key (CMEK). Format: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY |
↳ worker_pools |
array<WorkerPool>
The worker pools. At least one "harness" worker pool must be specified in order for the job to have workers. |
↳ user_agent |
Google\Protobuf\Struct
A description of the process that generated the request. |
↳ version |
Google\Protobuf\Struct
A structure describing which components and their versions of the service are required in order to run the job. |
↳ dataset |
string
Optional. The dataset for the current project where various workflow related tables are stored. The supported resource type is: Google BigQuery: bigquery.googleapis.com/{dataset} |
↳ sdk_pipeline_options |
Google\Protobuf\Struct
The Cloud Dataflow SDK pipeline options specified by the user. These options are passed through the service and are used to recreate the SDK pipeline options on the worker in a language agnostic and platform independent way. |
↳ internal_experiments |
Google\Protobuf\Any
Experimental settings. |
↳ service_account_email |
string
Optional. Identity to run virtual machines as. Defaults to the default account. |
↳ flex_resource_scheduling_goal |
int
Optional. Which Flexible Resource Scheduling mode to run in. |
↳ worker_region |
string
Optional. The Compute Engine region (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1". Mutually exclusive with worker_zone. If neither worker_region nor worker_zone is specified, default to the control plane's region. |
↳ worker_zone |
string
Optional. The Compute Engine zone (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1-a". Mutually exclusive with worker_region. If neither worker_region nor worker_zone is specified, a zone in the control plane's region is chosen based on available capacity. |
↳ shuffle_mode |
int
Output only. The shuffle mode used for the job. |
↳ debug_options |
DebugOptions
Optional. Any debugging options to be supplied to the job. |
↳ use_streaming_engine_resource_based_billing |
bool
Output only. Whether the job uses the Streaming Engine resource-based billing model. |
↳ streaming_mode |
int
Optional. Specifies the Streaming Engine message processing guarantees. Reduces cost and latency but might result in duplicate messages committed to storage. Designed to run simple mapping streaming ETL jobs at the lowest cost. For example, Change Data Capture (CDC) to BigQuery is a canonical use case. For more information, see Set the pipeline streaming mode. |
getTempStoragePrefix
The prefix of the resources the system should use for temporary storage. The system will append the suffix "/temp-{JOBNAME} to this resource prefix, where {JOBNAME} is the value of the job_name field. The resulting bucket and object prefix is used as the prefix of the resources used to store temporary data needed during the job execution. NOTE: This will override the value in taskrunner_settings.
The supported resource type is: Google Cloud Storage: storage.googleapis.com/{bucket}/{object} bucket.storage.googleapis.com/{object}
| Returns | |
|---|---|
| Type | Description |
string |
|
setTempStoragePrefix
The prefix of the resources the system should use for temporary storage. The system will append the suffix "/temp-{JOBNAME} to this resource prefix, where {JOBNAME} is the value of the job_name field. The resulting bucket and object prefix is used as the prefix of the resources used to store temporary data needed during the job execution. NOTE: This will override the value in taskrunner_settings.
The supported resource type is: Google Cloud Storage: storage.googleapis.com/{bucket}/{object} bucket.storage.googleapis.com/{object}
| Parameter | |
|---|---|
| Name | Description |
var |
string
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getClusterManagerApiService
The type of cluster manager API to use. If unknown or unspecified, the service will attempt to choose a reasonable default. This should be in the form of the API service name, e.g. "compute.googleapis.com".
| Returns | |
|---|---|
| Type | Description |
string |
|
setClusterManagerApiService
The type of cluster manager API to use. If unknown or unspecified, the service will attempt to choose a reasonable default. This should be in the form of the API service name, e.g. "compute.googleapis.com".
| Parameter | |
|---|---|
| Name | Description |
var |
string
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getExperiments
The list of experiments to enable. This field should be used for SDK related experiments and not for service related experiments. The proper field for service related experiments is service_options.
| Returns | |
|---|---|
| Type | Description |
Google\Protobuf\Internal\RepeatedField |
|
setExperiments
The list of experiments to enable. This field should be used for SDK related experiments and not for service related experiments. The proper field for service related experiments is service_options.
| Parameter | |
|---|---|
| Name | Description |
var |
string[]
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getServiceOptions
Optional. The list of service options to enable. This field should be used for service related experiments only. These experiments, when graduating to GA, should be replaced by dedicated fields or become default (i.e. always on).
| Returns | |
|---|---|
| Type | Description |
Google\Protobuf\Internal\RepeatedField |
|
setServiceOptions
Optional. The list of service options to enable. This field should be used for service related experiments only. These experiments, when graduating to GA, should be replaced by dedicated fields or become default (i.e. always on).
| Parameter | |
|---|---|
| Name | Description |
var |
string[]
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getServiceKmsKeyName
Optional. If set, contains the Cloud KMS key identifier used to encrypt data at rest, AKA a Customer Managed Encryption Key (CMEK).
Format: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
| Returns | |
|---|---|
| Type | Description |
string |
|
setServiceKmsKeyName
Optional. If set, contains the Cloud KMS key identifier used to encrypt data at rest, AKA a Customer Managed Encryption Key (CMEK).
Format: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
| Parameter | |
|---|---|
| Name | Description |
var |
string
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getWorkerPools
The worker pools. At least one "harness" worker pool must be specified in order for the job to have workers.
| Returns | |
|---|---|
| Type | Description |
Google\Protobuf\Internal\RepeatedField |
|
setWorkerPools
The worker pools. At least one "harness" worker pool must be specified in order for the job to have workers.
| Parameter | |
|---|---|
| Name | Description |
var |
array<WorkerPool>
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getUserAgent
A description of the process that generated the request.
| Returns | |
|---|---|
| Type | Description |
Google\Protobuf\Struct|null |
|
hasUserAgent
clearUserAgent
setUserAgent
A description of the process that generated the request.
| Parameter | |
|---|---|
| Name | Description |
var |
Google\Protobuf\Struct
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getVersion
A structure describing which components and their versions of the service are required in order to run the job.
| Returns | |
|---|---|
| Type | Description |
Google\Protobuf\Struct|null |
|
hasVersion
clearVersion
setVersion
A structure describing which components and their versions of the service are required in order to run the job.
| Parameter | |
|---|---|
| Name | Description |
var |
Google\Protobuf\Struct
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getDataset
Optional. The dataset for the current project where various workflow related tables are stored.
The supported resource type is: Google BigQuery: bigquery.googleapis.com/{dataset}
| Returns | |
|---|---|
| Type | Description |
string |
|
setDataset
Optional. The dataset for the current project where various workflow related tables are stored.
The supported resource type is: Google BigQuery: bigquery.googleapis.com/{dataset}
| Parameter | |
|---|---|
| Name | Description |
var |
string
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getSdkPipelineOptions
The Cloud Dataflow SDK pipeline options specified by the user. These options are passed through the service and are used to recreate the SDK pipeline options on the worker in a language agnostic and platform independent way.
| Returns | |
|---|---|
| Type | Description |
Google\Protobuf\Struct|null |
|
hasSdkPipelineOptions
clearSdkPipelineOptions
setSdkPipelineOptions
The Cloud Dataflow SDK pipeline options specified by the user. These options are passed through the service and are used to recreate the SDK pipeline options on the worker in a language agnostic and platform independent way.
| Parameter | |
|---|---|
| Name | Description |
var |
Google\Protobuf\Struct
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getInternalExperiments
Experimental settings.
| Returns | |
|---|---|
| Type | Description |
Google\Protobuf\Any|null |
|
hasInternalExperiments
clearInternalExperiments
setInternalExperiments
Experimental settings.
| Parameter | |
|---|---|
| Name | Description |
var |
Google\Protobuf\Any
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getServiceAccountEmail
Optional. Identity to run virtual machines as. Defaults to the default account.
| Returns | |
|---|---|
| Type | Description |
string |
|
setServiceAccountEmail
Optional. Identity to run virtual machines as. Defaults to the default account.
| Parameter | |
|---|---|
| Name | Description |
var |
string
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getFlexResourceSchedulingGoal
Optional. Which Flexible Resource Scheduling mode to run in.
| Returns | |
|---|---|
| Type | Description |
int |
|
setFlexResourceSchedulingGoal
Optional. Which Flexible Resource Scheduling mode to run in.
| Parameter | |
|---|---|
| Name | Description |
var |
int
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getWorkerRegion
Optional. The Compute Engine region (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1". Mutually exclusive with worker_zone. If neither worker_region nor worker_zone is specified, default to the control plane's region.
| Returns | |
|---|---|
| Type | Description |
string |
|
setWorkerRegion
Optional. The Compute Engine region (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1". Mutually exclusive with worker_zone. If neither worker_region nor worker_zone is specified, default to the control plane's region.
| Parameter | |
|---|---|
| Name | Description |
var |
string
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getWorkerZone
Optional. The Compute Engine zone (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1-a". Mutually exclusive with worker_region. If neither worker_region nor worker_zone is specified, a zone in the control plane's region is chosen based on available capacity.
| Returns | |
|---|---|
| Type | Description |
string |
|
setWorkerZone
Optional. The Compute Engine zone (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1-a". Mutually exclusive with worker_region. If neither worker_region nor worker_zone is specified, a zone in the control plane's region is chosen based on available capacity.
| Parameter | |
|---|---|
| Name | Description |
var |
string
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getShuffleMode
Output only. The shuffle mode used for the job.
| Returns | |
|---|---|
| Type | Description |
int |
|
setShuffleMode
Output only. The shuffle mode used for the job.
| Parameter | |
|---|---|
| Name | Description |
var |
int
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getDebugOptions
Optional. Any debugging options to be supplied to the job.
| Returns | |
|---|---|
| Type | Description |
DebugOptions|null |
|
hasDebugOptions
clearDebugOptions
setDebugOptions
Optional. Any debugging options to be supplied to the job.
| Parameter | |
|---|---|
| Name | Description |
var |
DebugOptions
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getUseStreamingEngineResourceBasedBilling
Output only. Whether the job uses the Streaming Engine resource-based billing model.
| Returns | |
|---|---|
| Type | Description |
bool |
|
setUseStreamingEngineResourceBasedBilling
Output only. Whether the job uses the Streaming Engine resource-based billing model.
| Parameter | |
|---|---|
| Name | Description |
var |
bool
|
| Returns | |
|---|---|
| Type | Description |
$this |
|
getStreamingMode
Optional. Specifies the Streaming Engine message processing guarantees.
Reduces cost and latency but might result in duplicate messages committed to storage. Designed to run simple mapping streaming ETL jobs at the lowest cost. For example, Change Data Capture (CDC) to BigQuery is a canonical use case. For more information, see Set the pipeline streaming mode.
| Returns | |
|---|---|
| Type | Description |
int |
|
setStreamingMode
Optional. Specifies the Streaming Engine message processing guarantees.
Reduces cost and latency but might result in duplicate messages committed to storage. Designed to run simple mapping streaming ETL jobs at the lowest cost. For example, Change Data Capture (CDC) to BigQuery is a canonical use case. For more information, see Set the pipeline streaming mode.
| Parameter | |
|---|---|
| Name | Description |
var |
int
|
| Returns | |
|---|---|
| Type | Description |
$this |
|