Manage metadata of Dataform repositories with Dataplex

This document describes using Dataplex to view metadata of Dataform repositories and annotate them with aspects.

Dataplex lets you store and manage your metadata. Dataform repositories are available in Dataplex by default, without any additional configuration. Dataplex automatically retrieves the following metadata from Dataform repositories:

Data asset name
Data asset parent
Data asset location
Data asset type
Corresponding Google Cloud project
Third-party repository source
Service account

Before you begin

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector
Enable the Dataplex API.
Enable the API
Ensure that you have the necessary Identity and Access Management (IAM) permissions.

Required roles

To manage Dataform repositories in Dataplex, you need the predefined Dataform roles that give you access to Dataform resources, and required Dataplex roles.

Manage metadata in Dataplex

Dataplex logs Dataform repositories as entries with the following entry values:

System entry group: The system entry group for Dataform repositories is @dataform. To view details of repository entries in Dataplex, you need to view the dataform system entry group. For instructions about how to view a list of all entries in an entry group, see View details of an entry group in the Dataplex documentation.
System entry type: The system entry type for Dataform repositories is dataform-repository. To view details of Dataform repositories, you need to view the dataform-repository system entry type. For instructions about how to view details of a selected entry type, see View details of an entry type in the Dataplex documentation. For instructions about how to view details of a selected entry, see View details of an entry in the Dataplex documentation.

Search for assets

For instructions about how to search for data assets in Dataplex, see Search for data assets in Dataplex in the Dataplex documentation.

Annotate repositories

To provide additional context to Dataform repositories in Dataplex, you can annotate Dataform entries with aspects.

The system aspect type for Dataform repositories is dataform-repository.

For instructions about how to annotate entries with aspects, see Manage aspects and enrich metadata in the Dataplex documentation.

Limitations

Managing Dataform repositories in Dataplex is available with the following limitations:

You cannot view or manage Dataform file-level assets in Dataplex—for example, tables, or views.

Locations

You can use Dataplex to manage Dataform repositories from all Dataform locations.

Quotas

Managing Dataform repositories in Dataplex is subject to Dataplex quotas and limits.

Pricing

Managing Dataform repositories in Dataplex is subject to Dataplex pricing.

What's next

To learn more about Dataplex, see Dataplex overview in the Dataplex documentation.