Package Classes (0.9.0)

Stay organized with collections Save and categorize content based on your preferences.

Provides a service for Flex templates.

Provides a service for Flex templates.

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

A pager for iterating through aggregated_list_jobs requests.

This class thinly wraps an initial ListJobsResponse object, and provides an __aiter__ method to iterate through its jobs field.

If there are more pages, the __aiter__ method will make additional AggregatedListJobs requests and continue to iterate through the jobs field on the corresponding responses.

All the usual ListJobsResponse attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

A pager for iterating through aggregated_list_jobs requests.

This class thinly wraps an initial ListJobsResponse object, and provides an __iter__ method to iterate through its jobs field.

If there are more pages, the __iter__ method will make additional AggregatedListJobs requests and continue to iterate through the jobs field on the corresponding responses.

All the usual ListJobsResponse attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

A pager for iterating through list_jobs requests.

This class thinly wraps an initial ListJobsResponse object, and provides an __aiter__ method to iterate through its jobs field.

If there are more pages, the __aiter__ method will make additional ListJobs requests and continue to iterate through the jobs field on the corresponding responses.

All the usual ListJobsResponse attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

A pager for iterating through list_jobs requests.

This class thinly wraps an initial ListJobsResponse object, and provides an __iter__ method to iterate through its jobs field.

If there are more pages, the __iter__ method will make additional ListJobs requests and continue to iterate through the jobs field on the corresponding responses.

All the usual ListJobsResponse attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

A pager for iterating through list_job_messages requests.

This class thinly wraps an initial ListJobMessagesResponse object, and provides an __aiter__ method to iterate through its job_messages field.

If there are more pages, the __aiter__ method will make additional ListJobMessages requests and continue to iterate through the job_messages field on the corresponding responses.

All the usual ListJobMessagesResponse attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

A pager for iterating through list_job_messages requests.

This class thinly wraps an initial ListJobMessagesResponse object, and provides an __iter__ method to iterate through its job_messages field.

If there are more pages, the __iter__ method will make additional ListJobMessages requests and continue to iterate through the job_messages field on the corresponding responses.

All the usual ListJobMessagesResponse attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

A pager for iterating through get_job_execution_details requests.

This class thinly wraps an initial JobExecutionDetails object, and provides an __aiter__ method to iterate through its stages field.

If there are more pages, the __aiter__ method will make additional GetJobExecutionDetails requests and continue to iterate through the stages field on the corresponding responses.

All the usual JobExecutionDetails attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

A pager for iterating through get_job_execution_details requests.

This class thinly wraps an initial JobExecutionDetails object, and provides an __iter__ method to iterate through its stages field.

If there are more pages, the __iter__ method will make additional GetJobExecutionDetails requests and continue to iterate through the stages field on the corresponding responses.

All the usual JobExecutionDetails attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

A pager for iterating through get_stage_execution_details requests.

This class thinly wraps an initial StageExecutionDetails object, and provides an __aiter__ method to iterate through its workers field.

If there are more pages, the __aiter__ method will make additional GetStageExecutionDetails requests and continue to iterate through the workers field on the corresponding responses.

All the usual StageExecutionDetails attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

A pager for iterating through get_stage_execution_details requests.

This class thinly wraps an initial StageExecutionDetails object, and provides an __iter__ method to iterate through its workers field.

If there are more pages, the __iter__ method will make additional GetStageExecutionDetails requests and continue to iterate through the workers field on the corresponding responses.

All the usual StageExecutionDetails attributes are available on the pager. If multiple requests are made, only the most recent response is retained, and thus used for attribute lookup.

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

Provides a method to create Cloud Dataflow jobs from templates.

Provides a method to create Cloud Dataflow jobs from templates.

Specifies the algorithm used to determine the number of worker processes to run at any given point in time, based on the amount of data left to process, the number of workers, and how quickly existing workers are processing data.

A structured message reporting an autoscaling decision made by the Dataflow service.

Indicates the type of autoscaling event.

Settings for WorkerPool autoscaling.

Metadata for a BigQuery connector used by the job.

Metadata for a Cloud Bigtable connector used by the job.

Request to check is active jobs exists for a project

Response for CheckActiveJobsRequest.

All configuration data for a particular Computation.

Container Spec.

A request to create a Cloud Dataflow job from a template.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

The abstract base class for a message.

Request to create a Cloud Dataflow job.

Identifies the location of a custom souce.

Data disk assignment for a given VM instance.

Configuration options for sampling elements.

The following enum defines what to sample for a running job.

Metadata for a Datastore connector used by the job.

Describes any options that have an effect on the debugging of pipelines.

The default set of packages to be staged on a pool of workers.

Request to delete a snapshot.

Response from deleting a snapshot.

Describes the data disk used by a workflow job.

Data provided with a pipeline or transform to provide descriptive info.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

Parameters to pass when launching a dynamic template.

Describes the environment in which a Dataflow Job runs.

A message describing the state of a particular execution stage.

Description of the composing transforms, names/ids, and input/outputs of a stage of execution. Some composing transforms and sources may have been generated by the Dataflow service during execution planning.

Description of an interstitial value between transforms in an execution stage.

Description of a transform executed as part of an execution stage.

Description of an input or output of an execution stage.

The state of some component of job execution.

Indicates which regional endpoint failed to respond to a request for data.

Metadata for a File connector used by the job.

Specifies the resource to optimize for in Flexible Resource Scheduling.

The environment values to be set at runtime for flex template.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

The abstract base class for a message.

Request to get job execution details.

Request to get job metrics.

Request to get the state of a Cloud Dataflow job.

Request to get information about a snapshot

Request to get information about a particular execution stage of a job. Currently only tracked for Batch jobs.

A request to retrieve a Cloud Dataflow job template.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

The various views of a template that may be retrieved.

The response to a GetTemplate request.

Template Type.

Information useful for debugging a hot key detection.

The abstract base class for a message.

Information about a hot key.

Used in the error_details field of a google.rpc.Status message, this indicates problems with the template parameter.

A specific template-parameter violation.

Defines a job to be run by the Cloud Dataflow service. Do not enter confidential information when you supply string values using the API.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

The abstract base class for a message.

The abstract base class for a message.

Information about the execution of a job.

Additional information about how a Cloud Dataflow job will be executed that isn't contained in the submitted job.

The abstract base class for a message.

Contains information about how a particular google.dataflow.v1beta3.Step][google.dataflow.v1beta3.Step] will be executed.

A particular message pertaining to a Dataflow job.

Indicates the importance of the message.

Metadata available primarily for filtering jobs. Will be included in the ListJob response and Job SUMMARY view.

The abstract base class for a message.

JobMetrics contains a collection of metrics describing the detailed progress of a Dataflow job. Metrics correspond to user-defined and system-defined metrics in the job. For more information, see Dataflow job metrics.

This resource captures only the most recent values of each metric; time-series data can be queried for them (under the same metric names) from Cloud Monitoring.

Describes the overall state of a google.dataflow.v1beta3.Job][google.dataflow.v1beta3.Job].

Specifies the processing model used by a [google.dataflow.v1beta3.Job], which determines the way the Job is managed by the Cloud Dataflow service (how workers are scheduled, how inputs are sharded, etc).

Selector for how much information is returned in Job responses.

    Project ID, Job ID, job name, job type, job
    status, start/end time, and Cloud SDK version
    details.
JOB_VIEW_ALL (2):
    Request all information available for this job. When the job
    is in `JOB_STATE_PENDING`, the job has been created but is
    not yet running, and not all job information is available.
    For complete job information, wait until the job in is
    `JOB_STATE_RUNNING`. For more information, see
    `JobState <https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#jobstate>`__.
JOB_VIEW_DESCRIPTION (3):
    Request summary info and limited job
    description data for steps, labels and
    environment.

Data disk assignment information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

Location information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

Type of transform or stage operation.

Launch FlexTemplate Parameter.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

The abstract base class for a message.

The abstract base class for a message.

The abstract base class for a message.

A request to launch a Cloud Dataflow job from a FlexTemplate.

Response to the request to launch a job from Flex Template.

Parameters to provide to the template being launched. Note that the metadata in the pipeline code determines which runtime parameters are valid.

The abstract base class for a message.

The abstract base class for a message.

A request to launch a template.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

Response to the request to launch a template.

Request to list job messages. Up to max_results messages will be returned in the time range specified starting with the oldest messages first. If no time range is specified the results with start with the oldest message.

Response to a request to list job messages.

Request to list Cloud Dataflow jobs.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

This field filters out and returns jobs in the specified job state. The order of data returned is determined by the filter used, and is subject to change.

Response to a request to list Cloud Dataflow jobs in a project. This might be a partial response, depending on the page size in the ListJobsRequest. However, if the project does not have any jobs, an instance of ListJobsResponse is not returned and the requests's response body is empty {}.

Request to list snapshots.

List of snapshots.

Identifies a metric, by describing the source which generated the metric.

The abstract base class for a message.

Describes the state of a metric. Next ID: 14

Describes mounted data disk.

The packages that must be installed in order for a worker to run the steps of the Cloud Dataflow job that will be assigned to its worker pool.

This is the mechanism by which the Cloud Dataflow SDK causes code to be loaded onto the workers. For example, the Cloud Dataflow Java SDK might use this to install jars containing the user's code and all of the various dependencies (libraries, data files, etc.) required in order for that code to run.

Metadata for a specific parameter.

The abstract base class for a message.

ParameterMetadataEnumOption specifies the option shown in the enum form.

ParameterType specifies what kind of input we need for this parameter.

A descriptive representation of submitted pipeline as well as the executed form. This data is provided by the Dataflow service for ease of visualizing the pipeline and interpreting Dataflow provided metrics.

Information about the progress of some component of job execution.

A point in the timeseries.

Metadata for a Pub/Sub connector used by the job.

Identifies a pubsub location to use for transferring data into or out of a streaming Dataflow job.

Represents a Pubsub snapshot.

The environment values to set at runtime.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

The abstract base class for a message.

RuntimeMetadata describing a runtime environment.

Additional job parameters that can only be updated during runtime using the projects.jobs.update method. These fields have no effect when specified during job creation.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

SDK Information.

SDK Language.

A bug found in the Dataflow SDK.

Indicates the severity of the bug. Other severities may be added to this list in the future.

Nature of the issue, ordered from least severe to most. Other bug types may be added to this list in the future.

Defines an SDK harness container for executing Dataflow pipelines.

The version of the SDK used to run the job.

The support status of the SDK used to run the job.

Resources used by the Dataflow Service to run the job.

Specifies the shuffle mode used by a [google.dataflow.v1beta3.Job], which determines the approach data is shuffled during processing. More details in: https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-shuffle

Represents a snapshot of a job.

Request to create a snapshot of a job.

Snapshot state.

Metadata for a Spanner connector used by the job.

Information about the workers and work items within a stage.

Information about a particular execution stage of a job.

State family configuration.

Defines a particular step within a Cloud Dataflow job.

A job consists of multiple steps, each of which performs some specific operation as part of the overall job. Data is typically passed from one step to another as part of the job.

Note: The properties of this object are not stable and might change.

Here's an example of a sequence of steps which together implement a Map-Reduce job:

• Read a collection of data from some source, parsing the collection's elements.

• Validate the elements.

• Apply a user-defined function to map each element to some value and extract an element-specific key value.

• Group elements with the same key into a single element with that key, transforming a multiply-keyed collection into a uniquely-keyed collection.

• Write the elements out to some data sink.

Note that the Cloud Dataflow service may be used to run many different types of jobs, not just Map-Reduce.

Information for a straggler.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

Information useful for straggler identification and debugging.

The abstract base class for a message.

Information useful for debugging a straggler. Each type will provide specialized debugging information relevant for a particular cause. The StragglerDebuggingInfo will be 1:1 mapping to the StragglerCause enum.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

Summarized straggler identification details.

The abstract base class for a message.

Describes a stream of data, either as input to be processed or as output of a streaming Dataflow job.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

Streaming appliance snapshot configuration.

Describes full or partial data disk assignment information of the computation ranges.

Specifies the Streaming Engine message processing guarantees. Reduces cost and latency but might result in duplicate messages written to storage. Designed to run simple mapping streaming ETL jobs at the lowest cost. For example, Change Data Capture (CDC) to BigQuery is a canonical use case. For more information, see Set the pipeline streaming mode <https://cloud.google.com/dataflow/docs/guides/streaming-modes>__.

Identifies the location of a streaming side input.

Identifies the location of a streaming computation stage, for stage-to-stage communication.

Information useful for streaming straggler identification and debugging.

A rich message format, including a human readable string, a key for identifying the message, and structured data associated with the message for programmatic consumption.

Structured data associated with this message.

Taskrunner configuration settings.

Specifies what happens to a resource when a Cloud Dataflow google.dataflow.v1beta3.Job][google.dataflow.v1beta3.Job] has completed.

Metadata describing a template.

Global topology of the streaming Dataflow job, including all computations and their sharded locations.

The abstract base class for a message.

Description of the type, names/ids, and input/outputs for a transform.

Request to update a Cloud Dataflow job.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

Information about an individual work item execution.

Information about a worker

Specifies how to allocate IP addresses to worker machines. You can also use pipeline options <https://cloud.google.com/dataflow/docs/reference/pipeline-options#security_and_networking>__ to specify whether Dataflow workers use external IP addresses.

Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.

The abstract base class for a message.

Provides data to pass through to the worker harness.

API documentation for dataflow_v1beta3.services.jobs_v1_beta3.pagers module.

API documentation for dataflow_v1beta3.services.messages_v1_beta3.pagers module.

API documentation for dataflow_v1beta3.services.metrics_v1_beta3.pagers module.