Stay organized with collections
Save and categorize content based on your preferences.
This document provides an overview of the managed connectivity pipelines that
you can use to import metadata from third-party sources into Dataplex Universal Catalog.
Managed connectivity lets you import metadata into Dataplex Universal Catalog at
scale. A managed connectivity pipeline extracts metadata from your data sources
and then imports the metadata into Dataplex Universal Catalog. If necessary, the pipeline
also creates Dataplex Universal Catalog entry groups in your
Google Cloud project. You can orchestrate the workflows, and schedule the
import jobs based on your requirements.
You build your own custom connectors to extract metadata from third-party
sources. For example, you can build a connector to extract metadata from sources
like MySQL, SQL Server, Oracle, Snowflake, Databricks, and others. For steps to build
a sample custom connector, see
Develop a custom connector for metadata import.
You can also use the
community-contributed custom connectors
that are available for a variety of third-party sources.
The managed connectivity pipeline does the following things:
Creates a target entry group based on your configuration, if the entry
group doesn't exist yet.
Runs the connector. The connector extracts the metadata from your data
source and generates a metadata import file that can be imported into
Dataplex Universal Catalog.
Monitors the progress of the metadata extraction.
Runs a metadata import job to import the metadata into
Dataplex Universal Catalog.
Monitors the progress of the metadata import job.
The managed connectivity pipeline uses Dataproc Serverless to run the
connector, and Dataplex Universal Catalog metadata import API methods
to run the metadata import job.
To import metadata from third-party sources, you can use custom connectors that
are contributed by the community. See each connector's README file for setup
instructions and more information about the connector.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-25 UTC."],[[["\u003cp\u003eManaged connectivity pipelines facilitate the import of metadata from various third-party sources into Dataplex at scale.\u003c/p\u003e\n"],["\u003cp\u003eUsers can create custom connectors to extract metadata from sources like MySQL, SQL Server, Oracle, Snowflake, and Databricks.\u003c/p\u003e\n"],["\u003cp\u003eThe managed connectivity pipeline uses Dataproc Serverless to run connectors and imports the metadata into Dataplex Catalog.\u003c/p\u003e\n"],["\u003cp\u003eThe pipeline automatically creates Dataplex Catalog entry groups if they don't exist, and then runs the connector to extract and import metadata.\u003c/p\u003e\n"],["\u003cp\u003eThe workflows used in the pipelines allow for the scheduling and orchestration of metadata import jobs based on user needs.\u003c/p\u003e\n"]]],[],null,["# Managed connectivity overview\n\nThis document provides an overview of the managed connectivity pipelines that\nyou can use to import metadata from third-party sources into Dataplex Universal Catalog.\n\nManaged connectivity lets you import metadata into Dataplex Universal Catalog at\nscale. A managed connectivity pipeline extracts metadata from your data sources\nand then imports the metadata into Dataplex Universal Catalog. If necessary, the pipeline\nalso creates Dataplex Universal Catalog entry groups in your\nGoogle Cloud project. You can orchestrate the workflows, and schedule the\nimport jobs based on your requirements.\n\nYou build your own custom connectors to extract metadata from third-party\nsources. For example, you can build a connector to extract metadata from sources\nlike MySQL, SQL Server, Oracle, Snowflake, Databricks, and others. For steps to build\na sample custom connector, see\n[Develop a custom connector for metadata import](/dataplex/docs/develop-custom-connector).\nYou can also use the\n[community-contributed custom connectors](#community-contributed-connectors)\nthat are available for a variety of third-party sources.\n\nFor steps to run a managed connectivity pipeline, see\n[Import metadata from a custom source using Workflows](/dataplex/docs/import-using-workflows-custom-source).\n\nHow managed connectivity works\n------------------------------\n\nThe following diagram shows a managed connectivity pipeline.\n\nAt a high level, here's how managed connectivity works:\n\n1. You\n [build a connector for your data source](/dataplex/docs/develop-custom-connector).\n\n The connector must be an Artifact Registry image that can run on\n Dataproc Serverless.\n2. You [run the managed connectivity pipeline](/dataplex/docs/import-using-workflows-custom-source)\n in Workflows, an orchestration platform.\n\n3. The managed connectivity pipeline does the following things:\n\n 1. Creates a target entry group based on your configuration, if the entry group doesn't exist yet.\n 2. Runs the connector. The connector extracts the metadata from your data source and generates a metadata import file that can be imported into Dataplex Universal Catalog.\n 3. Monitors the progress of the metadata extraction.\n 4. Runs a metadata import job to import the metadata into Dataplex Universal Catalog.\n 5. Monitors the progress of the metadata import job.\n\nThe managed connectivity pipeline uses Dataproc Serverless to run the\nconnector, and Dataplex Universal Catalog metadata import API methods\nto run the metadata import job.\n\nThe metadata that you import consists of Dataplex Universal Catalog\nentries and their aspects. For more information about\nDataplex Universal Catalog metadata, see\n[About metadata management in Dataplex Universal Catalog](/dataplex/docs/catalog-overview).\n\nCommunity-contributed custom connectors\n---------------------------------------\n\nTo import metadata from third-party sources, you can use custom connectors that\nare contributed by the community. See each connector's README file for setup\ninstructions and more information about the connector.\n| **Note:** These connectors are not officially supported by Google.\n\nWhat's next\n-----------\n\n- [Import metadata from a custom source using Workflows](/dataplex/docs/import-using-workflows-custom-source)\n- [Develop a custom connector for metadata import](/dataplex/docs/develop-custom-connector)\n- [Import metadata using a custom pipeline](/dataplex/docs/import-metadata)"]]
