Requires Apache Beam SDK for Java version 2.58.0 or later.
Configuration
Managed I/O uses the following configuration parameters for Apache Iceberg:
Read and write configuration
Data type
Description
table
string
The identifier of the Apache Iceberg table. Example:
"db.table1".
catalog_name
string
The name of the catalog. Example: "local".
catalog_properties
map
A map of configuration properties for the Apache Iceberg
catalog. The required properties depend on the catalog. For more
information, see
CatalogUtil in the Apache Iceberg documentation.
config_properties
map
An optional set of Hadoop configuration properties. For more
information, see
CatalogUtil in the Apache Iceberg documentation.
Write configuration
Data type
Description
triggering_frequency_seconds
integer
For streaming write pipelines, the frequency at which the sink
attempts to produce snapshots, in seconds.
For more information and code examples, see the following topics:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-04-02 UTC."],[[["Managed I/O for Apache Iceberg supports various catalogs, including Hadoop, Hive, REST-based catalogs, and BigQuery metastore, enabling batch and streaming read and write operations."],["Write capabilities include batch writes, streaming writes, dynamic destinations, and dynamic table creation, providing flexibility in data management."],["For BigQuery tables, the `BigQueryIO` connector with the BigQuery Storage API is used, but dynamic table creation is not supported."],["Configuration parameters like `table`, `catalog_name`, `catalog_properties`, `config_properties`, and `triggering_frequency_seconds` allow for customization of Apache Iceberg operations."],["The usage of this feature requires Apache Beam SDK for Java version 2.58.0 or later, while using the BigQuery Metastore requires 2.62.0 or later if not using Runner V2."]]],[]]
