Managed I/O supports the following capabilities for BigQuery:
- Dynamic table creation
- Dynamic destinations
- For reads, the connector uses the BigQuery Storage Read API.
For writes, the connector uses the following BigQuery methods:
- If the source is unbounded, the connector performs direct writes to BigQuery, by using the BigQuery Storage Write API in exactly-once mode.
- If the source is bounded, the connector uses BigQuery file loads.
Requirements
Requires Apache Beam SDK for Java version 2.61.0 or later.
Configuration
| Read and write configuration | Data type | Description |
|---|---|---|
table |
string | The BigQuery table to read or write. Format as
"PROJECT.DATASET.TABLE". Example:
"my_project.dataset1.table1". |
kms_key |
string | Specifies a Cloud Key Management Service (Cloud KMS) key to encrypt the BigQuery table when writing, or to encrypt any temporary tables created during reads. |
| Read configuration | Data type | Description |
fields |
list of strings | A list of columns to read from the table. This parameter allows efficient reads when a table contains many columns. |
query |
string | A SQL query to read from. If specified, the connector runs the query on BigQuery and reads the query results. |
row_restriction |
string | A predicate that filters data on the server side. Example:
"age > 18". |
| Write configuration | Data type | Description |
triggering_frequency |
integer | For unbounded sources, specifies the frequency at which file writes are triggered, in seconds. |
For reads, you must specify either table or query. For writes, you must
specify table.
For more information, see the following topics: