Google Cloud Dataproc v1 API - Namespace Google.Cloud.Dataproc.V1 (5.10.0)

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.

Autoscaling Policy config associated with the cluster.

Describes an autoscaling policy for Dataproc cluster autoscaler.

Resource name for the AutoscalingPolicy resource.

The API interface for managing autoscaling policies in the Dataproc API.

Base class for server-side implementations of AutoscalingPolicyService

Client for AutoscalingPolicyService

AutoscalingPolicyService client wrapper, for convenient use.

Builder class for AutoscalingPolicyServiceClient to provide simple configuration of credentials, endpoint etc.

AutoscalingPolicyService client wrapper implementation, for convenient use.

Settings for AutoscalingPolicyServiceClient instances.

Node group identification and configuration information.

Auxiliary services configuration for a Cluster.

Basic algorithm for autoscaling.

Basic autoscaling configurations for YARN.

A representation of a batch workload in the service.

Container for nested types declared in the Batch message type.

Historical state information.

The BatchController provides methods to manage batch workloads.

Base class for server-side implementations of BatchController

Client for BatchController

BatchController client wrapper, for convenient use.

Builder class for BatchControllerClient to provide simple configuration of credentials, endpoint etc.

BatchController client wrapper implementation, for convenient use.

Settings for BatchControllerClient instances.

Resource name for the Batch resource.

Metadata describing the Batch operation.

Container for nested types declared in the BatchOperationMetadata message type.

A request to cancel a job.

Describes the identifying information, config, and status of a Dataproc cluster

The cluster config.

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

Base class for server-side implementations of ClusterController

Client for ClusterController

ClusterController client wrapper, for convenient use.

Builder class for ClusterControllerClient to provide simple configuration of credentials, endpoint etc.

ClusterController client wrapper implementation, for convenient use.

Settings for ClusterControllerClient instances.

Contains cluster daemon metrics, such as HDFS and YARN stats.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Resource name for the Cluster resource.

The cluster operation triggered by a workflow.

Metadata describing the operation.

The status of the operation.

Container for nested types declared in the ClusterOperationStatus message type.

Resource name for the ClusterRegion resource.

A selector that chooses target cluster for jobs based on metadata.

The status of a cluster and its instances.

Container for nested types declared in the ClusterStatus message type.

Confidential Instance Config for clusters using Confidential VMs

A request to create an autoscaling policy.

A request to create a batch workload.

A request to create a cluster.

A request to create a node group.

A request to create a session.

A request to create a session template.

A request to create a workflow template.

Dataproc metric config.

Container for nested types declared in the DataprocMetricConfig message type.

A Dataproc custom metric.

A request to delete an autoscaling policy.

Autoscaling policies in use by one or more clusters will not be deleted.

A request to delete a batch workload.

A request to delete a cluster.

A request to delete a job.

A request to delete a session.

A request to delete a session template.

A request to delete a workflow template.

Currently started workflows will remain running.

A request to collect cluster diagnostic information.

The location of diagnostic output.

Specifies the config of disk options for a group of VM instances.

Driver scheduling configuration.

Encryption settings for the cluster.

Endpoint config for this cluster

Environment configuration for a workload.

Execution configuration for a workload.

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

Container for nested types declared in the GceClusterConfig message type.

A request to fetch an autoscaling policy.

A request to get the resource representation for a batch workload.

Request to get the resource representation for a cluster in a project.

A request to get the resource representation for a job in a project.

A request to get a node group .

A request to get the resource representation for a session.

A request to get the resource representation for a session template.

A request to fetch a workflow template.

The cluster's GKE config.

The configuration of a GKE node pool used by a Dataproc-on-GKE cluster.

Container for nested types declared in the GkeNodePoolConfig message type.

Parameters that describe cluster nodes.

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

GKE node pools that Dataproc workloads run on.

Container for nested types declared in the GkeNodePoolTarget message type.

A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.

A Dataproc job for running Apache Hive queries on YARN.

Identity related configuration, including service account based secure multi-tenancy user mappings.

Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.

Container for nested types declared in the InstanceFlexibilityPolicy message type.

Defines machines types and a rank to which the machines types belong.

Defines a mapping from machine types to the number of VMs that are created with each machine type.

Configuration for the size bounds of an instance group, including its proportional size to other groups.

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

Container for nested types declared in the InstanceGroupConfig message type.

A reference to a Compute Engine instance.

A request to instantiate an inline workflow template.

A request to instantiate a workflow template.

A Dataproc job resource.

The JobController provides methods to manage jobs.

Base class for server-side implementations of JobController

Client for JobController

JobController client wrapper, for convenient use.

Builder class for JobControllerClient to provide simple configuration of credentials, endpoint etc.

JobController client wrapper implementation, for convenient use.

Settings for JobControllerClient instances.

Job Operation metadata.

Dataproc job config.

Encapsulates the full scoping used to reference a job.

Job scheduling options.

Dataproc job status.

Container for nested types declared in the JobStatus message type.

Jupyter configuration for an interactive session.

Container for nested types declared in the JupyterConfig message type.

Specifies Kerberos related configuration.

The configuration for running the Dataproc cluster on Kubernetes.

The software configuration for this Dataproc cluster running on Kubernetes.

Specifies the cluster auto-delete schedule configuration.

A request to list autoscaling policies in a project.

A response to a request to list autoscaling policies in a project.

A request to list batch workloads in a project.

A list of batch workloads.

A request to list the clusters in a project.

The list of all clusters in a project.

A request to list jobs in a project.

Container for nested types declared in the ListJobsRequest message type.

A list of jobs in a project.

A request to list session templates in a project.

A list of session templates.

A request to list sessions in a project.

A list of interactive sessions.

A request to list workflow templates in a project.

A response to a request to list workflow templates in a project.

The runtime logging config of the job.

Container for nested types declared in the LoggingConfig message type.

Cluster that is managed by the workflow.

Specifies the resources used to actively manage an instance group.

Specifies a Metastore configuration.

Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc [NodeGroupAffinity][google.cloud.dataproc.v1.NodeGroupAffinity] resource.

Container for nested types declared in the NodeGroup message type.

Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc [NodeGroup][google.cloud.dataproc.v1.NodeGroup] resource.

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

Base class for server-side implementations of NodeGroupController

Client for NodeGroupController

NodeGroupController client wrapper, for convenient use.

Builder class for NodeGroupControllerClient to provide simple configuration of credentials, endpoint etc.

NodeGroupController client wrapper implementation, for convenient use.

Settings for NodeGroupControllerClient instances.

Resource name for the NodeGroup resource.

Metadata describing the node group operation.

Container for nested types declared in the NodeGroupOperationMetadata message type.

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

A job executed by the workflow.

Configuration for parameter validation.

Auxiliary services configuration for a workload.

A Dataproc job for running Apache Pig queries on YARN.

A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.

Configuration for PyPi repository

A configuration for running an Apache PySpark batch workload.

A Dataproc job for running Apache PySpark applications on YARN.

A list of queries to run on a cluster.

Validation based on regular expressions.

Resource name for the Region resource.

Configuration for dependency repositories

Reservation Affinity for consuming Zonal reservation.

Container for nested types declared in the ReservationAffinity message type.

A request to resize a node group.

Runtime configuration for a workload.

Runtime information about workload execution.

Security related configuration, including encryption, Kerberos, etc.

Resource name for the Service resource.

A representation of a session.

Container for nested types declared in the Session message type.

Historical state information.

The SessionController provides methods to manage interactive sessions.

Base class for server-side implementations of SessionController

Client for SessionController

SessionController client wrapper, for convenient use.

Builder class for SessionControllerClient to provide simple configuration of credentials, endpoint etc.

SessionController client wrapper implementation, for convenient use.

Settings for SessionControllerClient instances.

Resource name for the Session resource.

Metadata describing the Session operation.

Container for nested types declared in the SessionOperationMetadata message type.

A representation of a session template.

The SessionTemplateController provides methods to manage session templates.

Base class for server-side implementations of SessionTemplateController

Client for SessionTemplateController

SessionTemplateController client wrapper, for convenient use.

Builder class for SessionTemplateControllerClient to provide simple configuration of credentials, endpoint etc.

SessionTemplateController client wrapper implementation, for convenient use.

Settings for SessionTemplateControllerClient instances.

Resource name for the SessionTemplate resource.

Shielded Instance Config for clusters using Compute Engine Shielded VMs.

Specifies the selection and config of software inside the cluster.

A configuration for running an Apache Spark batch workload.

Spark History Server configuration for the workload.

A Dataproc job for running Apache Spark applications on YARN.

A configuration for running an Apache SparkR batch workload.

A Dataproc job for running Apache SparkR applications on YARN.

A configuration for running Apache Spark SQL queries as a batch workload.

A Dataproc job for running Apache Spark SQL queries.

A request to start a cluster.

Configuration to handle the startup of instances during cluster create and update process.

A request to stop a cluster.

A request to submit a job.

A configurable parameter that replaces one or more fields in the template. Parameterizable fields:

• Labels
• File uris
• Job properties
• Job arguments
• Script variables
• Main class (in HadoopJob and SparkJob)
• Zone (in ClusterSelector)

A request to terminate an interactive session.

A Dataproc job for running Trino queries. IMPORTANT: The Dataproc Trino Optional Component must be enabled when the cluster is created to submit a Trino job to the cluster.

A request to update an autoscaling policy.

A request to update a cluster.

A request to update a job.

A request to update a session template.

A request to update a workflow template.

Usage metrics represent approximate total resources consumed by a workload.

The usage snapshot represents the resources consumed by a workload at a specified time.

Validation based on a list of allowed values.

The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.

The workflow graph.

A Dataproc workflow template resource.

Container for nested types declared in the WorkflowMetadata message type.

The workflow node.

Container for nested types declared in the WorkflowNode message type.

A Dataproc workflow template resource.

Resource name for the WorkflowTemplate resource.

Specifies workflow execution target.

Either managed_cluster or cluster_selector is required.

The API interface for managing Workflow Templates in the Dataproc API.

Base class for server-side implementations of WorkflowTemplateService

Client for WorkflowTemplateService

WorkflowTemplateService client wrapper, for convenient use.

Builder class for WorkflowTemplateServiceClient to provide simple configuration of credentials, endpoint etc.

WorkflowTemplateService client wrapper implementation, for convenient use.

Settings for WorkflowTemplateServiceClient instances.

A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Container for nested types declared in the YarnApplication message type.

Enum of possible cases for the "algorithm" oneof.

The possible contents of AutoscalingPolicyName.

Enum of possible cases for the "config" oneof.

Enum of possible cases for the "batch_config" oneof.

The batch state.

The possible contents of BatchName.

Operation type for Batch resources

The possible contents of ClusterName.

The operation state.

The possible contents of ClusterRegionName.

The cluster state.

The cluster substate.

Cluster components that can be activated.

A source for the collection of Dataproc custom metrics (see Custom metrics).

Enum of possible cases for the "network" oneof.

Actions in response to failure of a resource associated with a cluster.

PrivateIpv6GoogleAccess controls whether and how Dataproc cluster nodes can communicate with Google Services through gRPC over IPv6. These values are directly mapped to corresponding values in the Compute Engine Instance fields.

Role specifies the tasks that will run on the node pool. Roles can be specific to workloads. Exactly one [GkeNodePoolTarget][google.cloud.dataproc.v1.GkeNodePoolTarget] within the virtual cluster must have the DEFAULT role, which is used to run all workloads that are not associated with a node pool.

Enum of possible cases for the "driver" oneof.

Enum of possible cases for the "queries" oneof.

Controls the use of preemptible instances within the group.

Enum of possible cases for the "type_job" oneof.

The job state.

The job substate.

Jupyter kernel types.

Enum of possible cases for the "config" oneof.

Enum of possible cases for the "ttl" oneof.

A matcher that specifies categories of job states.

The Log4j level for job execution. When running an Apache Hive job, Cloud Dataproc configures the Hive client to an equivalent verbosity level.

Node pool roles.

The possible contents of NodeGroupName.

Operation type for node group resources.

Enum of possible cases for the "job_type" oneof.

Enum of possible cases for the "validation_type" oneof.

Enum of possible cases for the "queries" oneof.

Enum of possible cases for the "queries" oneof.

The possible contents of RegionName.

Indicates whether to consume capacity from an reservation or not.

The possible contents of ServiceName.

Enum of possible cases for the "session_config" oneof.

The session state.

The possible contents of SessionName.

Operation type for Session resources

Enum of possible cases for the "session_config" oneof.

The possible contents of SessionTemplateName.

Enum of possible cases for the "driver" oneof.

Enum of possible cases for the "driver" oneof.

Enum of possible cases for the "queries" oneof.

Enum of possible cases for the "queries" oneof.

Enum of possible cases for the "infrastructure_config" oneof.

The operation state.

The workflow node state.

The possible contents of WorkflowTemplateName.

Enum of possible cases for the "placement" oneof.

The application state, corresponding to <code>YarnProtos.YarnApplicationStateProto</code>.