Google Cloud Dataproc v1 API - Namespace Google.Cloud.Dataproc.V1 (5.6.0)

Classes

AcceleratorConfig

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.

AutoscalingConfig

Autoscaling Policy config associated with the cluster.

AutoscalingPolicy

Describes an autoscaling policy for Dataproc cluster autoscaler.

AutoscalingPolicyName

Resource name for the AutoscalingPolicy resource.

AutoscalingPolicyService

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyService.AutoscalingPolicyServiceBase

Base class for server-side implementations of AutoscalingPolicyService

AutoscalingPolicyService.AutoscalingPolicyServiceClient

Client for AutoscalingPolicyService

AutoscalingPolicyServiceClient

AutoscalingPolicyService client wrapper, for convenient use.

AutoscalingPolicyServiceClientBuilder

Builder class for AutoscalingPolicyServiceClient to provide simple configuration of credentials, endpoint etc.

AutoscalingPolicyServiceClientImpl

AutoscalingPolicyService client wrapper implementation, for convenient use.

AutoscalingPolicyServiceSettings

Settings for AutoscalingPolicyServiceClient instances.

AuxiliaryNodeGroup

Node group identification and configuration information.

AuxiliaryServicesConfig

Auxiliary services configuration for a Cluster.

BasicAutoscalingAlgorithm

Basic algorithm for autoscaling.

BasicYarnAutoscalingConfig

Basic autoscaling configurations for YARN.

Batch

A representation of a batch workload in the service.

Batch.Types

Container for nested types declared in the Batch message type.

Batch.Types.StateHistory

Historical state information.

BatchController

The BatchController provides methods to manage batch workloads.

BatchController.BatchControllerBase

Base class for server-side implementations of BatchController

BatchController.BatchControllerClient

Client for BatchController

BatchControllerClient

BatchController client wrapper, for convenient use.

BatchControllerClientBuilder

Builder class for BatchControllerClient to provide simple configuration of credentials, endpoint etc.

BatchControllerClientImpl

BatchController client wrapper implementation, for convenient use.

BatchControllerSettings

Settings for BatchControllerClient instances.

BatchName

Resource name for the Batch resource.

BatchOperationMetadata

Metadata describing the Batch operation.

BatchOperationMetadata.Types

Container for nested types declared in the BatchOperationMetadata message type.

CancelJobRequest

A request to cancel a job.

Cluster

Describes the identifying information, config, and status of a Dataproc cluster

ClusterConfig

The cluster config.

ClusterController

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterController.ClusterControllerBase

Base class for server-side implementations of ClusterController

ClusterController.ClusterControllerClient

Client for ClusterController

ClusterControllerClient

ClusterController client wrapper, for convenient use.

ClusterControllerClientBuilder

Builder class for ClusterControllerClient to provide simple configuration of credentials, endpoint etc.

ClusterControllerClientImpl

ClusterController client wrapper implementation, for convenient use.

ClusterControllerSettings

Settings for ClusterControllerClient instances.

ClusterMetrics

Contains cluster daemon metrics, such as HDFS and YARN stats.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

ClusterName

Resource name for the Cluster resource.

ClusterOperation

The cluster operation triggered by a workflow.

ClusterOperationMetadata

Metadata describing the operation.

ClusterOperationStatus

The status of the operation.

ClusterOperationStatus.Types

Container for nested types declared in the ClusterOperationStatus message type.

ClusterRegionName

Resource name for the ClusterRegion resource.

ClusterSelector

A selector that chooses target cluster for jobs based on metadata.

ClusterStatus

The status of a cluster and its instances.

ClusterStatus.Types

Container for nested types declared in the ClusterStatus message type.

ConfidentialInstanceConfig

Confidential Instance Config for clusters using Confidential VMs

CreateAutoscalingPolicyRequest

A request to create an autoscaling policy.

CreateBatchRequest

A request to create a batch workload.

CreateClusterRequest

A request to create a cluster.

CreateNodeGroupRequest

A request to create a node group.

CreateWorkflowTemplateRequest

A request to create a workflow template.

DataprocMetricConfig

Dataproc metric config.

DataprocMetricConfig.Types

Container for nested types declared in the DataprocMetricConfig message type.

DataprocMetricConfig.Types.Metric

A Dataproc custom metric.

DeleteAutoscalingPolicyRequest

A request to delete an autoscaling policy.

Autoscaling policies in use by one or more clusters will not be deleted.

DeleteBatchRequest

A request to delete a batch workload.

DeleteClusterRequest

A request to delete a cluster.

DeleteJobRequest

A request to delete a job.

DeleteWorkflowTemplateRequest

A request to delete a workflow template.

Currently started workflows will remain running.

DiagnoseClusterRequest

A request to collect cluster diagnostic information.

DiagnoseClusterResults

The location of diagnostic output.

DiskConfig

Specifies the config of disk options for a group of VM instances.

DriverSchedulingConfig

Driver scheduling configuration.

EncryptionConfig

Encryption settings for the cluster.

EndpointConfig

Endpoint config for this cluster

EnvironmentConfig

Environment configuration for a workload.

ExecutionConfig

Execution configuration for a workload.

GceClusterConfig

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

GceClusterConfig.Types

Container for nested types declared in the GceClusterConfig message type.

GetAutoscalingPolicyRequest

A request to fetch an autoscaling policy.

GetBatchRequest

A request to get the resource representation for a batch workload.

GetClusterRequest

Request to get the resource representation for a cluster in a project.

GetJobRequest

A request to get the resource representation for a job in a project.

GetNodeGroupRequest

A request to get a node group .

GetWorkflowTemplateRequest

A request to fetch a workflow template.

GkeClusterConfig

The cluster's GKE config.

GkeNodePoolConfig

The configuration of a GKE node pool used by a Dataproc-on-GKE cluster.

GkeNodePoolConfig.Types

Container for nested types declared in the GkeNodePoolConfig message type.

GkeNodePoolConfig.Types.GkeNodeConfig

Parameters that describe cluster nodes.

GkeNodePoolConfig.Types.GkeNodePoolAcceleratorConfig

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.

GkeNodePoolConfig.Types.GkeNodePoolAutoscalingConfig

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

GkeNodePoolTarget

GKE node pools that Dataproc workloads run on.

GkeNodePoolTarget.Types

Container for nested types declared in the GkeNodePoolTarget message type.

HadoopJob

A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.

HiveJob

A Dataproc job for running Apache Hive queries on YARN.

IdentityConfig

Identity related configuration, including service account based secure multi-tenancy user mappings.

InstanceFlexibilityPolicy

Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.

InstanceFlexibilityPolicy.Types

Container for nested types declared in the InstanceFlexibilityPolicy message type.

InstanceFlexibilityPolicy.Types.InstanceSelection

Defines machines types and a rank to which the machines types belong.

InstanceFlexibilityPolicy.Types.InstanceSelectionResult

Defines a mapping from machine types to the number of VMs that are created with each machine type.

InstanceGroupAutoscalingPolicyConfig

Configuration for the size bounds of an instance group, including its proportional size to other groups.

InstanceGroupConfig

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

InstanceGroupConfig.Types

Container for nested types declared in the InstanceGroupConfig message type.

InstanceReference

A reference to a Compute Engine instance.

InstantiateInlineWorkflowTemplateRequest

A request to instantiate an inline workflow template.

InstantiateWorkflowTemplateRequest

A request to instantiate a workflow template.

Job

A Dataproc job resource.

JobController

The JobController provides methods to manage jobs.

JobController.JobControllerBase

Base class for server-side implementations of JobController

JobController.JobControllerClient

Client for JobController

JobControllerClient

JobController client wrapper, for convenient use.

JobControllerClientBuilder

Builder class for JobControllerClient to provide simple configuration of credentials, endpoint etc.

JobControllerClientImpl

JobController client wrapper implementation, for convenient use.

JobControllerSettings

Settings for JobControllerClient instances.

JobMetadata

Job Operation metadata.

JobPlacement

Dataproc job config.

JobReference

Encapsulates the full scoping used to reference a job.

JobScheduling

Job scheduling options.

JobStatus

Dataproc job status.

JobStatus.Types

Container for nested types declared in the JobStatus message type.

KerberosConfig

Specifies Kerberos related configuration.

KubernetesClusterConfig

The configuration for running the Dataproc cluster on Kubernetes.

KubernetesSoftwareConfig

The software configuration for this Dataproc cluster running on Kubernetes.

LifecycleConfig

Specifies the cluster auto-delete schedule configuration.

ListAutoscalingPoliciesRequest

A request to list autoscaling policies in a project.

ListAutoscalingPoliciesResponse

A response to a request to list autoscaling policies in a project.

ListBatchesRequest

A request to list batch workloads in a project.

ListBatchesResponse

A list of batch workloads.

ListClustersRequest

A request to list the clusters in a project.

ListClustersResponse

The list of all clusters in a project.

ListJobsRequest

A request to list jobs in a project.

ListJobsRequest.Types

Container for nested types declared in the ListJobsRequest message type.

ListJobsResponse

A list of jobs in a project.

ListWorkflowTemplatesRequest

A request to list workflow templates in a project.

ListWorkflowTemplatesResponse

A response to a request to list workflow templates in a project.

LoggingConfig

The runtime logging config of the job.

LoggingConfig.Types

Container for nested types declared in the LoggingConfig message type.

ManagedCluster

Cluster that is managed by the workflow.

ManagedGroupConfig

Specifies the resources used to actively manage an instance group.

MetastoreConfig

Specifies a Metastore configuration.

NodeGroup

Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc [NodeGroupAffinity][google.cloud.dataproc.v1.NodeGroupAffinity] resource.

NodeGroup.Types

Container for nested types declared in the NodeGroup message type.

NodeGroupAffinity

Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc [NodeGroup][google.cloud.dataproc.v1.NodeGroup] resource.

NodeGroupController

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

NodeGroupController.NodeGroupControllerBase

Base class for server-side implementations of NodeGroupController

NodeGroupController.NodeGroupControllerClient

Client for NodeGroupController

NodeGroupControllerClient

NodeGroupController client wrapper, for convenient use.

NodeGroupControllerClientBuilder

Builder class for NodeGroupControllerClient to provide simple configuration of credentials, endpoint etc.

NodeGroupControllerClientImpl

NodeGroupController client wrapper implementation, for convenient use.

NodeGroupControllerSettings

Settings for NodeGroupControllerClient instances.

NodeGroupName

Resource name for the NodeGroup resource.

NodeGroupOperationMetadata

Metadata describing the node group operation.

NodeGroupOperationMetadata.Types

Container for nested types declared in the NodeGroupOperationMetadata message type.

NodeInitializationAction

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

OrderedJob

A job executed by the workflow.

ParameterValidation

Configuration for parameter validation.

PeripheralsConfig

Auxiliary services configuration for a workload.

PigJob

A Dataproc job for running Apache Pig queries on YARN.

PrestoJob

A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.

PySparkBatch

A configuration for running an Apache PySpark batch workload.

PySparkJob

A Dataproc job for running Apache PySpark applications on YARN.

QueryList

A list of queries to run on a cluster.

RegexValidation

Validation based on regular expressions.

RegionName

Resource name for the Region resource.

ReservationAffinity

Reservation Affinity for consuming Zonal reservation.

ReservationAffinity.Types

Container for nested types declared in the ReservationAffinity message type.

ResizeNodeGroupRequest

A request to resize a node group.

RuntimeConfig

Runtime configuration for a workload.

RuntimeInfo

Runtime information about workload execution.

SecurityConfig

Security related configuration, including encryption, Kerberos, etc.

ServiceName

Resource name for the Service resource.

ShieldedInstanceConfig

Shielded Instance Config for clusters using Compute Engine Shielded VMs.

SoftwareConfig

Specifies the selection and config of software inside the cluster.

SparkBatch

A configuration for running an Apache Spark batch workload.

SparkHistoryServerConfig

Spark History Server configuration for the workload.

SparkJob

A Dataproc job for running Apache Spark applications on YARN.

SparkRBatch

A configuration for running an Apache SparkR batch workload.

SparkRJob

A Dataproc job for running Apache SparkR applications on YARN.

SparkSqlBatch

A configuration for running Apache Spark SQL queries as a batch workload.

SparkSqlJob

A Dataproc job for running Apache Spark SQL queries.

StartClusterRequest

A request to start a cluster.

StopClusterRequest

A request to stop a cluster.

SubmitJobRequest

A request to submit a job.

TemplateParameter

A configurable parameter that replaces one or more fields in the template. Parameterizable fields:

  • Labels
  • File uris
  • Job properties
  • Job arguments
  • Script variables
  • Main class (in HadoopJob and SparkJob)
  • Zone (in ClusterSelector)

TrinoJob

A Dataproc job for running Trino queries. IMPORTANT: The Dataproc Trino Optional Component must be enabled when the cluster is created to submit a Trino job to the cluster.

UpdateAutoscalingPolicyRequest

A request to update an autoscaling policy.

UpdateClusterRequest

A request to update a cluster.

UpdateJobRequest

A request to update a job.

UpdateWorkflowTemplateRequest

A request to update a workflow template.

UsageMetrics

Usage metrics represent approximate total resources consumed by a workload.

UsageSnapshot

The usage snaphot represents the resources consumed by a workload at a specified time.

ValueValidation

Validation based on a list of allowed values.

VirtualClusterConfig

The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.

WorkflowGraph

The workflow graph.

WorkflowMetadata

A Dataproc workflow template resource.

WorkflowMetadata.Types

Container for nested types declared in the WorkflowMetadata message type.

WorkflowNode

The workflow node.

WorkflowNode.Types

Container for nested types declared in the WorkflowNode message type.

WorkflowTemplate

A Dataproc workflow template resource.

WorkflowTemplateName

Resource name for the WorkflowTemplate resource.

WorkflowTemplatePlacement

Specifies workflow execution target.

Either managed_cluster or cluster_selector is required.

WorkflowTemplateService

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateService.WorkflowTemplateServiceBase

Base class for server-side implementations of WorkflowTemplateService

WorkflowTemplateService.WorkflowTemplateServiceClient

Client for WorkflowTemplateService

WorkflowTemplateServiceClient

WorkflowTemplateService client wrapper, for convenient use.

WorkflowTemplateServiceClientBuilder

Builder class for WorkflowTemplateServiceClient to provide simple configuration of credentials, endpoint etc.

WorkflowTemplateServiceClientImpl

WorkflowTemplateService client wrapper implementation, for convenient use.

WorkflowTemplateServiceSettings

Settings for WorkflowTemplateServiceClient instances.

YarnApplication

A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

YarnApplication.Types

Container for nested types declared in the YarnApplication message type.

Enums

AutoscalingPolicy.AlgorithmOneofCase

Enum of possible cases for the "algorithm" oneof.

AutoscalingPolicyName.ResourceNameType

The possible contents of AutoscalingPolicyName.

BasicAutoscalingAlgorithm.ConfigOneofCase

Enum of possible cases for the "config" oneof.

Batch.BatchConfigOneofCase

Enum of possible cases for the "batch_config" oneof.

Batch.Types.State

The batch state.

BatchName.ResourceNameType

The possible contents of BatchName.

BatchOperationMetadata.Types.BatchOperationType

Operation type for Batch resources

ClusterName.ResourceNameType

The possible contents of ClusterName.

ClusterOperationStatus.Types.State

The operation state.

ClusterRegionName.ResourceNameType

The possible contents of ClusterRegionName.

ClusterStatus.Types.State

The cluster state.

ClusterStatus.Types.Substate

The cluster substate.

Component

Cluster components that can be activated.

DataprocMetricConfig.Types.MetricSource

A source for the collection of Dataproc custom metrics (see Custom metrics).

ExecutionConfig.NetworkOneofCase

Enum of possible cases for the "network" oneof.

FailureAction

Actions in response to failure of a resource associated with a cluster.

GceClusterConfig.Types.PrivateIpv6GoogleAccess

PrivateIpv6GoogleAccess controls whether and how Dataproc cluster nodes can communicate with Google Services through gRPC over IPv6. These values are directly mapped to corresponding values in the Compute Engine Instance fields.

GkeNodePoolTarget.Types.Role

Role specifies the tasks that will run on the node pool. Roles can be specific to workloads. Exactly one [GkeNodePoolTarget][google.cloud.dataproc.v1.GkeNodePoolTarget] within the virtual cluster must have the DEFAULT role, which is used to run all workloads that are not associated with a node pool.

HadoopJob.DriverOneofCase

Enum of possible cases for the "driver" oneof.

HiveJob.QueriesOneofCase

Enum of possible cases for the "queries" oneof.

InstanceGroupConfig.Types.Preemptibility

Controls the use of preemptible instances within the group.

Job.TypeJobOneofCase

Enum of possible cases for the "type_job" oneof.

JobStatus.Types.State

The job state.

JobStatus.Types.Substate

The job substate.

KubernetesClusterConfig.ConfigOneofCase

Enum of possible cases for the "config" oneof.

LifecycleConfig.TtlOneofCase

Enum of possible cases for the "ttl" oneof.

ListJobsRequest.Types.JobStateMatcher

A matcher that specifies categories of job states.

LoggingConfig.Types.Level

The Log4j level for job execution. When running an Apache Hive job, Cloud Dataproc configures the Hive client to an equivalent verbosity level.

NodeGroup.Types.Role

Node group roles.

NodeGroupName.ResourceNameType

The possible contents of NodeGroupName.

NodeGroupOperationMetadata.Types.NodeGroupOperationType

Operation type for node group resources.

OrderedJob.JobTypeOneofCase

Enum of possible cases for the "job_type" oneof.

ParameterValidation.ValidationTypeOneofCase

Enum of possible cases for the "validation_type" oneof.

PigJob.QueriesOneofCase

Enum of possible cases for the "queries" oneof.

PrestoJob.QueriesOneofCase

Enum of possible cases for the "queries" oneof.

RegionName.ResourceNameType

The possible contents of RegionName.

ReservationAffinity.Types.Type

Indicates whether to consume capacity from an reservation or not.

ServiceName.ResourceNameType

The possible contents of ServiceName.

SparkBatch.DriverOneofCase

Enum of possible cases for the "driver" oneof.

SparkJob.DriverOneofCase

Enum of possible cases for the "driver" oneof.

SparkSqlJob.QueriesOneofCase

Enum of possible cases for the "queries" oneof.

TrinoJob.QueriesOneofCase

Enum of possible cases for the "queries" oneof.

VirtualClusterConfig.InfrastructureConfigOneofCase

Enum of possible cases for the "infrastructure_config" oneof.

WorkflowMetadata.Types.State

The operation state.

WorkflowNode.Types.NodeState

The workflow node state.

WorkflowTemplateName.ResourceNameType

The possible contents of WorkflowTemplateName.

WorkflowTemplatePlacement.PlacementOneofCase

Enum of possible cases for the "placement" oneof.

YarnApplication.Types.State

The application state, corresponding to <code>YarnProtos.YarnApplicationStateProto</code>.