Introduction to BigLake metastore

BigLake metastore is a fully managed metastore for data analytics products on Google Cloud. It provides a single source of truth for managing metadata from multiple sources. The metastore is accessible from BigQuery and various open data processing engines, making it a useful tool for data analysts and engineers.

For example, you can use BigLake metastore as the catalog with open source query engines such as Apache Spark. Tables created using Spark can be queried using BigQuery without requiring you to synchronize your metadata.

Benefits

BigLake metastore offers several advantages for data management and analysis:

  • Serverless architecture. BigLake metastore provides a serverless architecture, eliminating the need for server or cluster management. This helps reduce operational overhead, simplifies deployment, and allows for automatic scaling based on demand.
  • Engine interoperability. BigLake metastore provides you with direct table access in BigQuery, allowing you to query open-format tables stored in BigQuery without additional configuration. For example, you can create a table in Spark and then query it directly in BigQuery. This helps streamline your analytics workflow and reduces the need for complex data movement or ETL processes.

  • Unified user experience. BigLake metastore provides a unified workflow across BigQuery and BigQuery Studio. This lets you use Spark directly in BigQuery and BigQuery Studio. For example:

    First, you can create a table in Spark with a BigQuery Studio notebook.

    Create table in BigLake metastore

    Next, you can query the same Spark table in the Google Cloud console.

    Query table in BigLake metastore

Supported integrations

You can use BigLake metastore with the Google Cloud console, gcloud CLI, the BigQuery REST APIs, or the Iceberg REST API.

BigLake metastore supports the following integrations:

Differences with BigLake metastore (classic)

BigLake metastore is the recommended metastore on Google Cloud.

The core differences between BigLake metastore and BigLake metastore (classic) include the following details:

  • BigLake metastore (classic) is a standalone metastore service that is distinct from BigQuery and only supports Iceberg tables. It has a different three-part resource model. BigLake metastore (classic) tables aren't automatically discovered from BigQuery.

  • BigLake metastore is based on the BigQuery catalog and directly integrates with BigQuery. Tables in BigLake metastore are mutable from multiple open source engines and the same tables can be queried from BigQuery. When you use BigQuery, there is only one source of truth for your metadata. For example, BigLake metastore supports direct integration with Spark. This integration provides a more seamless workflow and helps reduce redundancy when storing metadata and running jobs.

Limitations

The following limitations apply to BigLake metastore tables:

  • You can't create or modify BigLake metastore tables with DDL or DML statements using the BigQuery engine. You can modify BigLake metastore tables using the BigQuery API (with the bq command-line tool or client libraries), but doing so risks making changes that are incompatible with the external engine.
  • BigLake metastore tables don't support renaming operations or ALTER TABLE ... RENAME TO Spark SQL statements.
  • BigLake metastore tables are subject to the same quotas and limits as standard tables.
  • Query performance for BigLake metastore tables from the BigQuery engine might be slow compared to querying data in a standard BigQuery table. In general, the query performance for a BigLake metastore table should be equivalent to reading the data directly from Cloud Storage.
  • A dry run of a query that uses a BigLake metastore table might report a lower bound of 0 bytes of data, even if rows are returned. This result occurs because the amount of data that is processed from the table can't be determined until the actual query completes. Running the query incurs a cost for processing this data.
  • You can't reference a BigLake metastore table in a wildcard table query.
  • You can't use the tabledata.list method to retrieve data from BigLake metastore tables. Instead, you can save query results to a destination table, then use the tabledata.list method on that table.
  • BigLake metastore tables don't support clustering.
  • BigLake metastore tables don't support flexible column names.
  • The display of table storage statistics for BigLake metastore tables isn't supported.

What's next