Mit Sammlungen den Überblick behalten
Sie können Inhalte basierend auf Ihren Einstellungen speichern und kategorisieren.
Selbstverwalteten MySQL-Metastore zu Dataproc Metastore migrieren
Auf dieser Seite wird beschrieben, wie Sie Ihren externen selbstverwalteten MySQL-Metastore zu Dataproc Metastore migrieren, indem Sie eine MySQL-Dumpdatei erstellen und die Metadaten in einen vorhandenen Dataproc Metastore-Dienst importieren.
Hinweise
Sign in to your Google Cloud account. If you're new to
Google Cloud,
create an account to evaluate how our products perform in
real-world scenarios. New customers also get $300 in free credits to
run, test, and deploy workloads.
In the Google Cloud console, on the project selector page,
select or create a Google Cloud project.
Bitten Sie Ihren Administrator, Ihnen die folgenden IAM-Rollen zuzuweisen, um die Berechtigungen zu erhalten, die Sie zum Erstellen eines Dataproc Metastore und zum Importieren eines MySQL-Metastore benötigen:
So erstellen Sie einen Dienst und importieren Metadaten:
So verwenden Sie das Cloud Storage-Objekt (SQL-Dumpdatei) für den Import:
Storage-Objekt-Betrachter (roles/storage.objectViewer)
für den Dataproc Metastore-Dienst-Agent. Beispiel: service-CUSTOMER_PROJECT_NUMBER@gcp-sa-metastore.iam.gserviceaccount.com
Diese vordefinierten Rollen enthalten die Berechtigungen, die zum Erstellen eines Dataproc Metastore und zum Importieren eines MySQL-Metastore erforderlich sind. Erweitern Sie den Abschnitt Erforderliche Berechtigungen, um die erforderlichen Berechtigungen anzuzeigen:
Erforderliche Berechtigungen
Die folgenden Berechtigungen sind zum Erstellen eines Dataproc Metastore und zum Importieren eines MySQL-Metastore erforderlich:
So erstellen Sie einen Dienst:
metastore.services.create
für das Projekt.
So importieren Sie Metadaten:
metastore.imports.create
für das Projekt.
So verwenden Sie das Cloud Storage-Objekt (SQL-Dumpdatei) für den Import:
storage.objects.get
auf dem Dataproc Metastore-Dienst-Agent. z. B. service-CUSTOMER_PROJECT_NUMBER@gcp-sa-metastore.iam.gserviceaccount.com.
Sie müssen jetzt die in Ihrer Hive-Metastore-Datenbank gespeicherten Metadaten für den Import vorbereiten, indem Sie eine MySQL-Dumpdatei erstellen und in einen Cloud Storage-Bucket einfügen.
[[["Leicht verständlich","easyToUnderstand","thumb-up"],["Mein Problem wurde gelöst","solvedMyProblem","thumb-up"],["Sonstiges","otherUp","thumb-up"]],[["Schwer verständlich","hardToUnderstand","thumb-down"],["Informationen oder Beispielcode falsch","incorrectInformationOrSampleCode","thumb-down"],["Benötigte Informationen/Beispiele nicht gefunden","missingTheInformationSamplesINeed","thumb-down"],["Problem mit der Übersetzung","translationIssue","thumb-down"],["Sonstiges","otherDown","thumb-down"]],["Zuletzt aktualisiert: 2025-08-27 (UTC)."],[[["\u003cp\u003eThis guide details how to migrate a self-managed MySQL metastore to Dataproc Metastore by creating and importing a MySQL dump file.\u003c/p\u003e\n"],["\u003cp\u003eBefore starting, enable the Dataproc Metastore API and ensure you have the necessary IAM roles, such as Dataproc Metastore Editor, Admin, and Storage Object Viewer, for both the service agent and user account.\u003c/p\u003e\n"],["\u003cp\u003eTo migrate, you will need to create a Dataproc Metastore service, such as \u003ccode\u003eexample-service\u003c/code\u003e, specifying a data location, and then prepare the metadata for import by making a MySQL dump file and placing it in a Cloud Storage bucket.\u003c/p\u003e\n"],["\u003cp\u003eAfter importing the metadata, create and attach a Dataproc cluster to your new \u003ccode\u003eexample-service\u003c/code\u003e Dataproc Metastore.\u003c/p\u003e\n"],["\u003cp\u003eThe guide provides instructions on cleaning up resources after the migration, including deleting the Dataproc Metastore service and the associated Cloud Storage bucket.\u003c/p\u003e\n"]]],[],null,["# Quickstart: Migrate a self-managed MySQL metastore to Dataproc Metastore\n\nMigrate a self-managed MySQL metastore to Dataproc Metastore\n============================================================\n\nThis page shows you how to migrate your external self-managed MySQL metastore to\nDataproc Metastore by creating a MySQL dump file and importing\nthe metadata into an existing Dataproc Metastore service.\n\nBefore you begin\n----------------\n\nSign in to your Google Cloud account. If you're new to Google Cloud, [create an account](https://console.cloud.google.com/freetrial) to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.\n\nIn the Google Cloud console, on the project selector page,\nselect or create a Google Cloud project.\n| **Note**: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.\n\n[Go to project selector](https://console.cloud.google.com/projectselector2/home/dashboard)\n\n\n[Verify that billing is enabled for your Google Cloud project](/billing/docs/how-to/verify-billing-enabled#confirm_billing_is_enabled_on_a_project).\n\n\nEnable the Dataproc Metastore API.\n\n\n[Enable the API](https://console.cloud.google.com/flows/enableapi?apiid=metastore.googleapis.com) \nIn the Google Cloud console, on the project selector page,\nselect or create a Google Cloud project.\n| **Note**: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.\n\n[Go to project selector](https://console.cloud.google.com/projectselector2/home/dashboard)\n\n\n[Verify that billing is enabled for your Google Cloud project](/billing/docs/how-to/verify-billing-enabled#confirm_billing_is_enabled_on_a_project).\n\n\nEnable the Dataproc Metastore API.\n\n\n[Enable the API](https://console.cloud.google.com/flows/enableapi?apiid=metastore.googleapis.com)\n1.\n\n\n Enable the Dataproc Metastore API.\n\n\n [Enable the API](https://console.cloud.google.com/flows/enableapi?apiid=metastore.googleapis.com)\n\n### Required Roles\n\n\nTo get the permissions that\nyou need to create a Dataproc Metastore and import a MYSQL metastore,\n\nask your administrator to grant you the\nfollowing IAM roles:\n\n- To create a service and import metadata:\n - [Dataproc Metastore Editor](/iam/docs/roles-permissions/metastore#metastore.editor) (`roles/metastore.editor`) on the project\n - [Dataproc Metastore Admin](/iam/docs/roles-permissions/metastore#metastore.admin) (`roles/metastore.admin`) on the project.\n- To use the Cloud Storage object (SQL dump file) for import:\n - [Storage Object Viewer](/iam/docs/roles-permissions/storage#storage.objectViewer) (`roles/storage.objectViewer`) on the Dataproc Metastore service agent. For example, `service-`\u003cvar translate=\"no\"\u003eCUSTOMER_PROJECT_NUMBER\u003c/var\u003e`@gcp-sa-metastore.iam.gserviceaccount.com`.\n - [Storage Object Viewer](/iam/docs/roles-permissions/storage#storage.objectViewer) (`roles/storage.objectViewer`) on the user account.\n\n\nFor more information about granting roles, see [Manage access to projects, folders, and organizations](/iam/docs/granting-changing-revoking-access).\n\n\nThese predefined roles contain\n\nthe permissions required to create a Dataproc Metastore and import a MYSQL metastore. To see the exact permissions that are\nrequired, expand the **Required permissions** section:\n\n\n#### Required permissions\n\nThe following permissions are required to create a Dataproc Metastore and import a MYSQL metastore:\n\n- To create a service: ` metastore.services.create` on the project.\n- To import metadata: ` metastore.imports.create` on the project.\n- To use the Cloud Storage object (SQL dump file) for import:\n - ` storage.objects.get` on the Dataproc Metastore service agent. For example, `service-`\u003cvar translate=\"no\"\u003eCUSTOMER_PROJECT_NUMBER\u003c/var\u003e`@gcp-sa-metastore.iam.gserviceaccount.com`\n - ` storage.objects.get` on the user account.\n\n\nYou might also be able to get\nthese permissions\nwith [custom roles](/iam/docs/creating-custom-roles) or\nother [predefined roles](/iam/docs/roles-overview#predefined).\nFor more information about specific Dataproc Metastore roles and permissions, see [Manage Dataproc access with IAM](/dataproc-metastore/docs/iam-and-access-control).\n\n\u003cbr /\u003e\n\nCreate a Dataproc Metastore service\n-----------------------------------\n\nThe following instructions demonstrate how to create a Dataproc Metastore\nservice that you can migrate your SQL dump file to. \n\n### Console\n\n1. In the Google Cloud console, open the **Create service** page:\n\n [Go to Create service](https://console.cloud.google.com/dataproc/metastore/services/create)\n The Dataproc MetastoreThe Create service page\n2. In the **Service name** field, enter `example-service`.\n\n3. Select the **Data location** .\n For information on selecting a region, see\n [Cloud locations](/about/locations).\n\n4. For other service configuration options, use the provided defaults.\n\n5. To create and start the service, click the **Submit** button.\n\nYour new service appears in the Service list.\n\n### gcloud\n\nRun the following `gcloud metastore services create`\ncommand to create a service:\n\n\u003cbr /\u003e\n\n```\n gcloud metastore services create example-service \\\n --location=LOCATION\n \n```\n\n\u003cbr /\u003e\n\nReplace \u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e with the Compute Engine region\nwhere you plan to create the service. Make sure [Dataproc Metastore is available](/about/locations)\nin the region.\n\n### REST\n\nFollow the API instructions to [create a service](/dataproc-metastore/docs/reference/rest/v1/projects.locations.services/create)\nby using the API Explorer.\n\nPrepare for migration\n---------------------\n\nYou must now prepare the metadata stored in your Hive metastore database for\nimport by making a MySQL dump file and placing it into a Cloud Storage\nbucket.\n\nSee [Prepare the import files before import](/dataproc-metastore/docs/import-metadata#prepare-the-import)\nfor steps to prepare for migration.\n\nImport the metadata\n-------------------\n\nNow that you've prepared the dump file, import it into your\nDataproc Metastore service.\n\nSee [Import the files into Dataproc Metastore](/dataproc-metastore/docs/import-metadata#perform-the-import)\nfor steps to import your metadata into your `example-service` service.\n| **Note:** If you're using [VPC Service Controls](/dataproc-metastore/docs/vpc-sc), then you can only import data from a Cloud Storage bucket that resides in the same service perimeter as the Dataproc Metastore service.\n\nCreate and attach a Dataproc cluster\n------------------------------------\n\nAfter you import your metadata into your Dataproc Metastore\n`example-service` service, [create and attach a Dataproc cluster](/dataproc-metastore/docs/attach-dataproc)\nthat uses the service as its Hive metastore.\n\nClean up\n--------\n\n\nTo avoid incurring charges to your Google Cloud account for\nthe resources used on this page, follow these steps.\n\n| **Caution** : Deleting a project has the following effects:\n|\n| - **Everything in the project is deleted.** If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.\n| - **Custom project IDs are lost.** When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an `appspot.com` URL, delete selected resources inside the project instead of deleting the whole project.\n|\n|\n| If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects\n| can help you avoid exceeding project quota limits.\n1. In the Google Cloud console, go to the **Manage resources** page.\n\n [Go to Manage resources](https://console.cloud.google.com/iam-admin/projects)\n2. In the project list, select the project that you want to delete, and then click **Delete**.\n3. In the dialog, type the project ID, and then click **Shut down** to delete the project.\n\n\u003cbr /\u003e\n\nAlternatively, you can delete the resources used in this tutorial:\n\n1. Delete the Dataproc Metastore service.\n\n ### Console\n\n 1. In the Google Cloud console, open the Dataproc Metastore page:\n\n [Open Dataproc Metastore in the\n Google Cloud console](https://console.cloud.google.com/dataproc/metastore/services)\n 2. On the left of the service name, select `example-service` by checking\n the box.\n\n 3. At the top of the **Dataproc Metastore** page, click **Delete**\n to delete the service.\n\n 4. On the dialog, click **Delete** to confirm the deletion.\n\n Your service no longer appears in the Service list.\n\n ### gcloud\n\n Run the following `gcloud metastore services delete` command to delete a\n service:\n\n \u003cbr /\u003e\n\n ```\n gcloud metastore services delete example-service \\\n --location=LOCATION\n ```\n\n \u003cbr /\u003e\n\n Replace \u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e with the Compute Engine\n region where you created the service.\n\n ### REST\n\n Follow the API instructions to [delete a service](/dataproc-metastore/docs/reference/rest/v1/projects.locations.services/delete)\n by using the API Explorer.\n\n All deletions succeed immediately.\n | **Note:** Deleting the service itself deletes all nested imports under that service.\n2. [Delete the Cloud Storage bucket](/storage/docs/deleting-buckets)\n for the Dataproc Metastore service.\n\n | **Note:** Deleting the Dataproc Metastore service doesn't delete its bucket.\n\nWhat's next\n-----------\n\n- [What is Dataproc Metastore?](/dataproc-metastore/docs/overview)\n- [Import metadata into a service](/dataproc-metastore/docs/import-metadata)\n- [Attach a Dataproc cluster or self-managed cluster](/dataproc-metastore/docs/attach-dataproc)"]]
