Deploy a Dataproc Metastore service

Required Roles

To get the permissions that you need to create a Dataproc Metastore and a Dataproc cluster, ask your administrator to grant you the following IAM roles:

• To grant full access to all Dataproc Metastore resources, including setting IAM permissions: (roles/metastore.admin) on the user account or service account
• To grant full control of Dataproc Metastore resources: Dataproc Metastore Editor (roles/metastore.editor) on the user account or service account
• To create a Dataproc cluster: (roles/dataproc.worker) on the service account

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to create a Dataproc Metastore and a Dataproc cluster. To see the exact permissions that are required, expand the Required permissions section:

You might also be able to get these permissions with custom roles or other predefined roles.

Create a Dataproc Metastore service

The following instructions show you how to create a basic Dataproc Metastore service using the provided default settings.

 gcloud metastore services create example-service \
     --location=us-central1 \
     --instance-size=MEDIUM

Create a Dataproc cluster and connect to Dataproc Metastore

Next, you create a Dataproc cluster and connect to your metastore from the cluster. After that, your cluster uses the metastore service as it's HMS. The cluster you create here uses the default provided settings.

 gcloud dataproc clusters create example-cluster \
    --dataproc-metastore=projects/PROJECT_ID/locations/us-central1/services/example-service \
    --region=us-central1

Connect to Apache Hive with a Dataproc cluster

This next steps show you how to run some example commands in Apache Hive to create a database and a table.

Next, open an SSH session on the Dataproc cluster and launch a Hive session.

1. In the Google Cloud console, go to the VM Instances page.
2. In the list of virtual machine instances, click SSH next to example-cluster.

A browser window opens in your home directory on the node with an output similar to the following:

Connected, host fingerprint: ssh-rsa ...
Linux cluster-1-m 3.16.0-0.bpo.4-amd64 ...
...
example-cluster@cluster-1-m:~$

To start Hive and create a database and table, run the following commands in the SSH session:

1. Start Hive.

   hive
   

2. Create a database called myDatabase.

   create database myDatabase;
   

3. Show the database you created.

   show databases;
   

4. Use the database you created.

   use myDatabase;
   

5. Create a table called myTable.

   create table myTable(id int,name string);
   

6. List the tables under myDatabase.

   show tables;
   

7. Describe the schema of the table you created.

   desc MyTable;
   

Running these commands show an output similar to the following:

$hive

hive> show databases;
OK
default
hive> create database myDatabase;
OK
hive> use myDatabase;
OK
hive> create table myTable(id int,name string);
OK
hive> show tables;
OK
myTable
hive> desc myTable;
OK
id                      int
name                    string