Mantenha tudo organizado com as coleções
Salve e categorize o conteúdo com base nas suas preferências.
Ao criar um cluster do Dataproc, coloque-o no modo de alta disponibilidade (HA) do Hadoop especificando o número de instâncias mestre no cluster. O número de mestres só pode ser especificado no momento da criação do cluster.
Atualmente, o Dataproc aceita duas configurações de mestre:
1 mestre (padrão, não HA)
3 mestres (HA do Hadoop)
Comparação entre os modos padrão e de alta disponibilidade do Hadoop
Falha do Compute Engine: no caso raro de uma
falha inesperada no Compute Engine, vai ocorrer uma
reinicialização de máquina das instâncias do Dataproc. A configuração padrão de um único mestre do Dataproc foi projetada para recuperar e continuar processando novos trabalhos nesses casos, mas necessariamente ocorrerá falha dos trabalhos em andamento e o HDFS ficará inacessível até que o único NameNode se recupere completamente ao reinicializar. No modo HA, a alta disponibilidade do HDFS e a alta disponibilidade do YARN são configuradas para permitir operações ininterruptas neles, mesmo com as falhas de nó único/reinicializações.
Encerramento do driver do job: o programa principal/de driver de quaisquer jobs executados ainda representará um
ponto único de falha, se a correção do job depender da
execução bem-sucedida do programa de driver. Os jobs enviados pela Dataproc Jobs API não são considerados de "alta disponibilidade" e ainda serão terminados em caso de falha do node mestre que executa os programas correspondentes do driver do job. Para que os jobs individuais sejam resistentes a falhas de node único usando um cluster de alta disponibilidade do Cloud Dataproc, eles precisam: 1) ser executados sem um programa de driver síncrono ou 2) precisam executar o próprio programa de driver em um contêiner YARN e serem gravados para processar as reinicializações do programa de driver. Veja em
Iniciar o Spark no YARN um exemplo de como os
programas de driver reinicializáveis podem ser executados em contêineres YARN para tolerância a falhas.
Falha zonal: como acontece com todos os clusters do Dataproc, os nós em um cluster de alta disponibilidade ficam na mesma zona. Se houver uma falha que
afete todos os nós em uma zona, ela não será atenuada.
Nomes de instâncias
O mestre padrão é chamado cluster-name-m. Os mestres de alta disponibilidade são chamados cluster-name-m-0, cluster-name-m-1, cluster-name-m-2.
Apache ZooKeeper
Em um cluster HA do Dataproc, o componente do Zookeeper é instalado automaticamente nos nós mestres do cluster. Todos os mestres participam de um cluster do ZooKeeper, que ativa o failover automático para outros serviços do Hadoop.
HDFS
Em um cluster padrão do Dataproc:
cluster-name-m executa:
NameNode
NameNode secundário
Em um cluster de alta disponibilidade do Dataproc:
Para criar um cluster de alta disponibilidade, selecione "Alta disponibilidade" (três mestres, N workers) na
seção "Tipo de cluster" do painel "Configurar cluster" na
página
Criar um cluster
do Dataproc.
[[["Fácil de entender","easyToUnderstand","thumb-up"],["Meu problema foi resolvido","solvedMyProblem","thumb-up"],["Outro","otherUp","thumb-up"]],[["Difícil de entender","hardToUnderstand","thumb-down"],["Informações incorretas ou exemplo de código","incorrectInformationOrSampleCode","thumb-down"],["Não contém as informações/amostras de que eu preciso","missingTheInformationSamplesINeed","thumb-down"],["Problema na tradução","translationIssue","thumb-down"],["Outro","otherDown","thumb-down"]],["Última atualização 2025-09-04 UTC."],[[["\u003cp\u003eDataproc clusters can be configured in Hadoop High Availability (HA) mode by setting the number of master instances to 3 during cluster creation, as opposed to the default of 1.\u003c/p\u003e\n"],["\u003cp\u003eHA mode provides uninterrupted YARN and HDFS operations despite single-node failures or reboots, unlike the default mode where in-flight jobs may fail during a Compute Engine failure, necessitating job retries.\u003c/p\u003e\n"],["\u003cp\u003eJobs submitted through the Dataproc Jobs API are not considered "high availability" and will be terminated if the master node running the job driver fails; if a job requires high availability, it must be launched without a driver program, or the driver program must be launched within a YARN container.\u003c/p\u003e\n"],["\u003cp\u003eIn an HA cluster, all master nodes participate in a ZooKeeper cluster to enable automatic failover, and each node runs ResourceManager, while in a default cluster, the single master runs the NameNode, Secondary NameNode, and ResourceManager.\u003c/p\u003e\n"],["\u003cp\u003eCreating an HA cluster involves using either the gcloud command with \u003ccode\u003e--num-masters=3\u003c/code\u003e, the REST API by setting \u003ccode\u003emasterConfig.numInstances\u003c/code\u003e to \u003ccode\u003e3\u003c/code\u003e, or by selecting "High Availability (3 masters, N workers)" in the Dataproc console.\u003c/p\u003e\n"]]],[],null,["When creating a Dataproc cluster, you can put the cluster into\nHadoop High Availability (HA) mode by\nspecifying the number of master instances in the\ncluster. The number of masters can only be specified at cluster creation time.\n\nCurrently, Dataproc supports two master configurations:\n\n- 1 master (default, non HA)\n- 3 masters (Hadoop HA)\n\nComparison of default and Hadoop High Availability mode Due to the complexity and higher cost of HA mode, use the default mode unless your use case requires HA mode.\n\n- **Compute Engine failure:** In the rare case of an\n unexpected Compute Engine failure, Dataproc\n instances will experience a machine reboot. The default single-master\n configuration for Dataproc is designed to recover and continue processing\n new work in such cases, but in-flight jobs will necessarily fail and need to be\n retried, and HDFS will be inaccessible until the single NameNode fully recovers\n on reboot. In **HA mode** , [HDFS High Availability](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html) and\n [YARN High Availability](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html)\n are configured to allow uninterrupted YARN and HDFS operations despite any\n single-node failures/reboots.\n\n- **Job driver termination:** The driver/main program of any jobs you run still represents a\n potential single point of failure if the correctness of your job depends on the\n driver program running successfully. Jobs submitted through the Dataproc\n Jobs API are not considered \"high availability,\" and will still be terminated on\n failure of the master node that runs the corresponding job driver programs. For\n individual jobs to be resilient against single-node failures using a HA Cloud\n Dataproc cluster, the job must either 1) run without a synchronous driver\n program or 2) it must run the driver program itself inside a YARN container and\n be written to handle driver-program restarts. See\n [Launching Spark on YARN](http://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn) for an example\n of how restartable driver programs can run inside YARN containers for fault\n tolerance.\n\n- **Zonal failure:** As is the case with all Dataproc clusters, all nodes in a High\n Availability cluster reside in the same zone. If there is a failure that\n impacts all nodes in a zone, the failure will not be mitigated.\n\nInstance Names\n\nThe default master is named `cluster-name-m`; HA masters are named\n`cluster-name-m-0`, `cluster-name-m-1`, `cluster-name-m-2`.\n\nApache ZooKeeper\n\nIn an HA Dataproc cluster, the\n[Zookeeper component](/dataproc/docs/concepts/components/zookeeper)\nis automatically installed on cluster master nodes. All masters\nparticipate in a ZooKeeper cluster, which enables automatic failover for\nother Hadoop services.\n\nHDFS\n\nIn a standard Dataproc cluster:\n\n- `cluster-name-m` runs:\n - NameNode\n - Secondary NameNode\n\nIn a High Availability Dataproc cluster:\n\n- `cluster-name-m-0` and `cluster-name-m-1` run:\n - NameNode\n - ZKFailoverController\n- All masters run JournalNode\n- There is no Secondary NameNode\n\nPlease see the [HDFS High Availability](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html)\ndocumentation for additional details on components.\n\nYARN\n\nIn a standard Dataproc cluster, `cluster-name-m` runs ResourceManager.\n\nIn a High Availability Dataproc cluster, all masters run ResourceManager.\n\nPlease see the [YARN High Availability](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html)\ndocumentation for additional details on components.\n\nCreate a High Availability cluster \n\ngcloud command\n\n\n| **gcloud CLI setup:** You must [setup and configure](/sdk/docs/quickstarts) the gcloud CLI to use the Google Cloud CLI.\nTo create an HA cluster with [gcloud dataproc clusters create](/sdk/gcloud/reference/dataproc/clusters/create), run the following command: \n\n```\ngcloud dataproc clusters create cluster-name \\\n --region=region \\\n --num-masters=3 \\\n ... other args\n```\n\n\u003cbr /\u003e\n\nREST API\n\n\nTo create an HA cluster, use the\n[clusters.create](/dataproc/docs/reference/rest/v1/projects.regions.clusters/create)\nAPI, setting [masterConfig.numInstances](/dataproc/docs/reference/rest/v1/ClusterConfig#InstanceGroupConfig)\nto `3`.\n| An easy way to construct the JSON body of an HA cluster create request is to create the request from the Dataproc [Create a cluster](https://console.cloud.google.com/dataproc/clustersAdd) page of the Google Cloud console. Select High Availability (3 masters, N workers) in the Cluster type section of the Set up cluster panel, then click the Equivalent REST button at the bottom of the left panel. Here's a snippet of a sample JSON output produced by the console for an HA cluster create request: \n|\n| ```\n| ...\n| masterConfig\": {\n| \"numInstances\": 3,\n| \"machineTypeUri\": \"n1-standard-4\",\n| \"diskConfig\": {\n| \"bootDiskSizeGb\": 500,\n| \"numLocalSsds\": 0\n| }\n| }\n| ...\n| ```\n\n\u003cbr /\u003e\n\nConsole\n\n\nTo create an HA cluster, select High Availability (3 masters, N workers) in\nthe Cluster type section of the Set up cluster panel on the\nDataproc\n[Create a cluster](https://console.cloud.google.com/dataproc/clustersAdd)\npage."]]
