Stay organized with collections
Save and categorize content based on your preferences.
Dataproc is a managed Spark and Hadoop service that lets you take advantage of open
source data tools for batch processing, querying, streaming, and machine learning.
Dataproc automation helps you create clusters quickly, manage them easily, and save
money by turning clusters off when you don't need them. With less time and money spent on
administration, you can focus on your jobs and your data.
Advantages of Dataproc
When compared to traditional, on-premises products and competing cloud
services, Dataproc has a number of unique advantages for clusters of
three to hundreds of nodes:
Low cost — Dataproc is
priced at only 1 cent per virtual CPU in your cluster per hour, on
top of the other Cloud Platform resources you use. In addition to this
low price, Dataproc clusters can include
preemptible instances that have lower
compute prices, reducing your costs even further. Instead of rounding
your usage up to the nearest hour, Dataproc charges you only for
what you really use with second-by-second billing and a low,
one-minute-minimum billing period.
Super fast — Without using Dataproc, it can take
from five to 30 minutes to create Spark and Hadoop clusters on-premises
or through IaaS providers. By comparison, Dataproc clusters are
quick to start, scale, and shutdown, with each of these operations taking
90 seconds or less, on average. This means you can spend less time
waiting for clusters and more hands-on time working with your data.
Integrated — Dataproc has built-in integration with
other Google Cloud Platform services, such as
BigQuery,
Cloud Storage,
Cloud Bigtable,
Cloud Logging, and
Cloud Monitoring, so you have more than just
a Spark or Hadoop cluster—you have a complete data platform. For
example, you can use Dataproc to effortlessly ETL terabytes of raw
log data directly into BigQuery for business reporting.
Managed — Use Spark and Hadoop clusters without the
assistance of an administrator or special software. You can easily
interact with clusters and Spark or Hadoop jobs through the
Google Cloud console, the Cloud SDK, or the Dataproc REST
API. When you're done with a cluster, you can simply turn it off, so you
don’t spend money on an idle cluster. You won’t need to worry about
losing data, because Dataproc is integrated with
Cloud Storage, BigQuery, and
Cloud Bigtable.
Simple and familiar — You don't need to learn new tools
or APIs to use Dataproc, making it easy to move existing projects
into Dataproc without redevelopment. Spark, Hadoop, Pig, and Hive
are frequently updated, so you can be productive faster.
What is included in Dataproc
For a list of the open source (Hadoop, Spark, Hive, and Pig) and Google Cloud
connector versions supported by
Dataproc, see the
Dataproc version list.
Getting Started with Dataproc
To quickly get started with Dataproc, see the Dataproc
quickstarts. You can access Dataproc in the following ways:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-25 UTC."],[[["\u003cp\u003eDataproc is a managed service for Spark and Hadoop that simplifies batch processing, querying, streaming, and machine learning with open-source tools.\u003c/p\u003e\n"],["\u003cp\u003eDataproc offers cost savings through low per-vCPU pricing, preemptible instances, and second-by-second billing, only charging for actual usage.\u003c/p\u003e\n"],["\u003cp\u003eClusters in Dataproc are created, scaled, and shut down quickly, often in 90 seconds or less, minimizing wait times and increasing efficiency.\u003c/p\u003e\n"],["\u003cp\u003eDataproc seamlessly integrates with other Google Cloud Platform services, forming a complete data platform that enables functionalities like ETL directly into BigQuery.\u003c/p\u003e\n"],["\u003cp\u003eDataproc provides a managed environment, eliminating the need for administrators or special software while offering easy interaction with clusters and jobs through the Google Cloud console, Cloud SDK, or REST API.\u003c/p\u003e\n"]]],[],null,["# Dataproc overview\n\nDataproc is a managed Spark and Hadoop service that lets you take advantage of open\nsource data tools for batch processing, querying, streaming, and machine learning.\nDataproc automation helps you create clusters quickly, manage them easily, and save\nmoney by turning clusters off when you don't need them. With less time and money spent on\nadministration, you can focus on your jobs and your data. \n\n### Advantages of Dataproc\n\nWhen compared to traditional, on-premises products and competing cloud\nservices, Dataproc has a number of unique advantages for clusters of\nthree to hundreds of nodes:\n\n- **Low cost** --- Dataproc is [priced](/dataproc/docs/resources/pricing) at only 1 cent per virtual CPU in your cluster per hour, on top of the other Cloud Platform resources you use. In addition to this low price, Dataproc clusters can include [preemptible instances](/preemptible-vms) that have lower compute prices, reducing your costs even further. Instead of rounding your usage up to the nearest hour, Dataproc charges you only for what you really use with second-by-second billing and a low, one-minute-minimum billing period.\n- **Super fast** --- Without using Dataproc, it can take from five to 30 minutes to create Spark and Hadoop clusters on-premises or through IaaS providers. By comparison, Dataproc clusters are quick to start, scale, and shutdown, with each of these operations taking 90 seconds or less, on average. This means you can spend less time waiting for clusters and more hands-on time working with your data.\n- **Integrated** --- Dataproc has built-in integration with other Google Cloud Platform services, such as [BigQuery](/bigquery), [Cloud Storage](/storage), [Cloud Bigtable](/bigtable), [Cloud Logging](/logging), and [Cloud Monitoring](/monitoring), so you have more than just a Spark or Hadoop cluster---you have a complete data platform. For example, you can use Dataproc to effortlessly ETL terabytes of raw log data directly into BigQuery for business reporting.\n- **Managed** --- Use Spark and Hadoop clusters without the assistance of an administrator or special software. You can easily interact with clusters and Spark or Hadoop jobs through the Google Cloud console, the Cloud SDK, or the Dataproc REST API. When you're done with a cluster, you can simply turn it off, so you don't spend money on an idle cluster. You won't need to worry about losing data, because Dataproc is integrated with [Cloud Storage](/storage), [BigQuery](/bigquery), and [Cloud Bigtable](/bigtable).\n- **Simple and familiar** --- You don't need to learn new tools or APIs to use Dataproc, making it easy to move existing projects into Dataproc without redevelopment. Spark, Hadoop, Pig, and Hive are frequently updated, so you can be productive faster.\n\n### What is included in Dataproc\n\nFor a list of the open source (Hadoop, Spark, Hive, and Pig) and Google Cloud\nconnector versions supported by\nDataproc, see the\n[Dataproc version list](/dataproc/docs/concepts/dataproc-versions).\n\n### Getting Started with Dataproc\n\nTo quickly get started with Dataproc, see the Dataproc\nquickstarts. You can access Dataproc in the following ways:\n\n- Through the [REST API](/dataproc/docs/quickstarts/create-cluster-template)\n- Using the [Cloud SDK](/dataproc/docs/quickstarts/create-cluster-gcloud)\n- Using the [Dataproc UI](/dataproc/docs/quickstarts/create-cluster-console)\n- Through the [Cloud Client Libraries](/dataproc/docs/quickstarts/create-cluster-client-libraries)"]]
