Tetap teratur dengan koleksi
Simpan dan kategorikan konten berdasarkan preferensi Anda.
Halaman ini memberikan ringkasan tujuan BigQuery. Dokumen ini
menjelaskan perilaku penulisan dan batasan Datastream yang diketahui saat Anda
mereplikasi data ke BigQuery.
Perilaku penulisan
Ukuran peristiwa maksimum saat Anda melakukan streaming data ke BigQuery adalah 20 MB.
Saat mengonfigurasi stream, Anda dapat memilih cara Datastream menulis data perubahan ke BigQuery. Untuk mengetahui informasi selengkapnya, lihat
Mengonfigurasi mode tulis.
Metadata tabel
Datastream menambahkan kolom STRUCT bernama datastream_metadata ke setiap
tabel yang ditulis ke tujuan BigQuery.
Mode penulisan penggabungan
Jika tabel memiliki kunci utama di sumber, kolom akan berisi kolom berikut:
UUID: Kolom ini memiliki jenis data STRING.
SOURCE_TIMESTAMP: Kolom ini memiliki jenis data INTEGER.
Jika tabel tidak memiliki kunci utama, kolom akan berisi kolom tambahan: IS_DELETED. Kolom ini memiliki jenis data BOOLEAN, dan menunjukkan apakah data yang dialirkan Datastream ke tujuan terkait dengan operasi DELETE di sumber. Tabel tanpa kunci utama hanya dapat ditambahkan.
Mode penulisan hanya tambahkan
Kolom datastream_metadata berisi kolom yang sama untuk tabel dengan dan tanpa kunci primer:
UUID: Kolom ini memiliki jenis data STRING.
SOURCE_TIMESTAMP: Kolom ini memiliki jenis data INTEGER.
CHANGE_SEQUENCE_NUMBER: Kolom ini memiliki jenis data STRING. Nomor urut internal yang digunakan oleh Datastream untuk setiap peristiwa perubahan.
CHANGE_TYPE: Kolom ini memiliki jenis data STRING. Menunjukkan jenis
peristiwa perubahan: INSERT, UPDATE-INSERT, UPDATE-DELETE, atau DELETE.
SORT_KEYS: Kolom ini berisi array nilai STRING. Anda dapat menggunakan
nilai untuk mengurutkan peristiwa perubahan.
Mengontrol biaya BigQuery
Biaya BigQuery ditagih secara terpisah dari Datastream. Untuk mempelajari cara mengontrol biaya BigQuery, lihat Harga CDC BigQuery.
Batasan umum
Batasan umum untuk menggunakan BigQuery sebagai tujuan meliputi:
Anda hanya dapat mereplikasi data ke set data BigQuery yang berada di project yang sama dengan aliran Datastream. Google Cloud
Secara default, Datastream tidak mendukung penambahan kunci utama ke tabel yang sudah direplikasi ke BigQuery tanpa kunci utama, atau penghapusan kunci utama dari tabel yang direplikasi ke BigQuery dengan kunci utama. Jika Anda perlu melakukan perubahan tersebut, hubungi Dukungan Google. Untuk mengetahui informasi tentang cara mengubah definisi kunci utama untuk tabel sumber yang sudah memiliki kunci utama, lihat Mendiagnosis masalah.
Kunci utama di BigQuery harus berupa jenis data berikut:
DATE
BOOL
GEOGRAPHY
INT64
NUMERIC
BIGNUMERIC
STRING
TIMESTAMP
DATETIME
Tabel yang berisi kunci utama jenis data yang tidak didukung tidak direplikasi oleh Datastream.
BigQuery tidak mendukung nama tabel dengan karakter ., $, /, @, atau +. Datastream mengganti karakter tersebut dengan garis bawah saat membuat tabel tujuan.
Misalnya, table.name di database sumber menjadi table_name di
BigQuery.
Untuk mengetahui informasi selengkapnya tentang nama tabel di BigQuery, lihat Penamaan
tabel.
BigQuery tidak mendukung lebih dari empat kolom pengelompokan. Saat mereplikasi tabel dengan lebih dari empat kolom kunci utama, Datastream menggunakan empat kolom kunci utama sebagai kolom pengelompokan.
Datastream memetakan literal tanggal dan waktu di luar rentang seperti jenis tanggal tak terbatas PostgreSQL ke nilai berikut:
DATE positif ke nilai 9999-12-31
DATE negatif ke nilai 0001-01-01
TIMESTAMP positif ke nilai 9999-12-31 23:59:59.999000 UTC
TIMESTAMP negatif ke nilai 0001-01-01 00:00:00 UTC
BigQuery tidak mendukung tabel streaming yang memiliki kunci primer
dengan jenis data FLOAT atau REAL. Tabel tersebut tidak direplikasi.
Untuk mempelajari lebih lanjut jenis dan rentang tanggal BigQuery, lihat Jenis
data.
Jika sumber Anda adalah Salesforce, opsi konfigurasi Set data untuk setiap skema tidak didukung.
[[["Mudah dipahami","easyToUnderstand","thumb-up"],["Memecahkan masalah saya","solvedMyProblem","thumb-up"],["Lainnya","otherUp","thumb-up"]],[["Sulit dipahami","hardToUnderstand","thumb-down"],["Informasi atau kode contoh salah","incorrectInformationOrSampleCode","thumb-down"],["Informasi/contoh yang saya butuhkan tidak ada","missingTheInformationSamplesINeed","thumb-down"],["Masalah terjemahan","translationIssue","thumb-down"],["Lainnya","otherDown","thumb-down"]],["Terakhir diperbarui pada 2025-08-12 UTC."],[[["\u003cp\u003eThis guide outlines the process of setting up a BigQuery destination to receive streamed data from a source database via Datastream.\u003c/p\u003e\n"],["\u003cp\u003eYou can choose to have Datastream create a separate dataset in BigQuery for each schema from the source database, or use a single designated dataset for all schemas.\u003c/p\u003e\n"],["\u003cp\u003eDatastream supports two write modes: "Merge," which keeps BigQuery in sync with the source without historical records, and "Append-only," which maintains a history of all data changes.\u003c/p\u003e\n"],["\u003cp\u003eEach table in the BigQuery destination will contain a \u003ccode\u003edatastream_metadata\u003c/code\u003e column, the fields of which will vary based on the table's primary key and write mode.\u003c/p\u003e\n"],["\u003cp\u003eDatastream includes mapping data types from various source databases (MySQL, Oracle, PostgreSQL, SQL Server, and Salesforce) to their corresponding BigQuery data types.\u003c/p\u003e\n"]]],[],null,["# BigQuery destination\n\nThis page gives you an overview of the BigQuery destination. It\ndescribes write behavior and known limitations of Datastream when you\nreplicate data to BigQuery.\n\nWrite behavior\n--------------\n\n- The maximum event size when you stream data into BigQuery is 20 MB.\n\n- When you configure your stream, you can select the way that Datastream\n writes your change data to BigQuery. For more information, see\n [Configure write mode](/datastream/docs/configure-bigquery-destination#configure-write-mode).\n\n### Table metadata\n\nDatastream appends a `STRUCT` column named `datastream_metadata` to each\ntable that's written to the BigQuery destination.\n\n#### Merge write mode\n\nIf a table has a primary key at the source, then the column contains the following fields:\n\n- `UUID`: This field has the `STRING` data type.\n- `SOURCE_TIMESTAMP`: This field has the `INTEGER` data type.\n\nIf a table doesn't have a primary key, then the column contains an additional field: `IS_DELETED`. This field has the `BOOLEAN` data type, and it indicates whether the data that Datastream streams to the destination is associated with a `DELETE` operation at the source. Tables without primary keys are append-only.\n\n#### Append-only write mode\n\nThe `datastream_metadata` column contains the same fields for tables with and\nwithout primary keys:\n\n- `UUID`: This field has the `STRING` data type.\n- `SOURCE_TIMESTAMP`: This field has the `INTEGER` data type.\n- `CHANGE_SEQUENCE_NUMBER`: This field has the `STRING` data type. It's an internal sequence number used by Datastream for each change event.\n- `CHANGE_TYPE`: This field has the `STRING` data type. It indicates the type of the change event: `INSERT`, `UPDATE-INSERT`, `UPDATE-DELETE`or `DELETE`.\n- `SORT_KEYS`: This field contains an array of `STRING` values. You can use the values to sort the change events.\n\n### Control BigQuery costs\n\nBigQuery costs are charged separately from Datastream. To learn\nhow to control your BigQuery costs, see\n[BigQuery CDC pricing](/bigquery/docs/change-data-capture#CDC_pricing).\n\nKnown limitations\n-----------------\n\n| **Note:** This section describes limitations for using Datastream with BigQuery as a destination. In addition to these limitations, you need to familiarize yourself with BigQuery change data capture limitations. For more information, see [Stream table updates with change data capture](/bigquery/docs/change-data-capture#limitations).\n\nKnown limitations for using BigQuery as a destination include:\n\n- You can only replicate data into a BigQuery dataset that resides in the same Google Cloud project as the Datastream stream.\n- By default, Datastream doesn't support adding a primary key to a table that's already replicated to BigQuery without a primary key, or removing a primary key from a table that's replicated to BigQuery with a primary key. If you need to perform such changes, contact Google Support. For information about *changing* the primary key definition for a source table that already has a primary key, see [Diagnose\n issues](/datastream/docs/diagnose-issues#bq-errors).\n- Primary keys in BigQuery must be of the following data types:\n\n - `DATE`\n - `BOOL`\n - `GEOGRAPHY`\n - `INT64`\n - `NUMERIC`\n - `BIGNUMERIC`\n - `STRING`\n - `TIMESTAMP`\n - `DATETIME`\n\n Tables that contain primary keys of unsupported data types aren't replicated\n by Datastream.\n- BigQuery doesn't support table names with `.`, `$`, `/`, `@` ,\n or `+` characters. Datastream replaces such characters with\n underscores when creating destination tables.\n\n For example, `table.name` in the source database becomes `table_name` in\n BigQuery.\n\n For more information on table names in BigQuery, see [Table\n naming](/bigquery/docs/tables#table_naming).\n- BigQuery doesn't support more than four clustering columns. When\n replicating a table with more than four primary key columns,\n Datastream uses four primary key columns as the clustering columns.\n\n- Datastream maps out-of-range date and time literals such as\n PostgreSQL infinity date types to the following values:\n\n - Positive `DATE` to the value of `9999-12-31`\n - Negative `DATE` to the value of `0001-01-01`\n - Positive `TIMESTAMP` to the value of `9999-12-31 23:59:59.999000 UTC`\n - Negative `TIMESTAMP` to the value of `0001-01-01 00:00:00 UTC`\n- BigQuery doesn't support streaming tables which have primary keys\n of `FLOAT` or `REAL` data types. Such tables aren't replicated.\n To learn more about BigQuery date types and ranges, see [Data\n types](/bigquery/docs/reference/standard-sql/data-types).\n\n- If your source is Salesforce, the **Dataset for each schema** configuration\n option isn't supported.\n\nWhat's next\n-----------\n\n- Learn how to [replicate data from a source database to BigQuery datasets\n using Datastream](/datastream/docs/quickstart-replication-to-bigquery)."]]
