Tetap teratur dengan koleksi
Simpan dan kategorikan konten berdasarkan preferensi Anda.
Dataflow mendukung dua mode untuk tugas streaming:
Mode tepat satu kali. Mode ini adalah default untuk semua
tugas streaming Dataflow. Dalam mode ini, Dataflow
memastikan bahwa rekaman tidak dihapus atau diduplikasi saat data bergerak melalui
pipeline.
Mode minimal satu kali. Mode ini menjamin bahwa rekaman diproses setidaknya sekali (yaitu, tidak ada rekaman input yang hilang). Namun, duplikat data
mungkin terjadi dalam mode ini. Untuk kasus penggunaan yang dapat mentoleransi duplikat,
mode minimal sekali dapat secara signifikan menurunkan biaya dan latensi tugas Anda.
Memilih mode streaming yang akan digunakan
Pilih mode tepat satu kali jika Anda perlu memastikan hasil yang tepat dari pipeline
dan semantik yang dapat diprediksi. Contoh:
Pipeline dengan
agregasi,
seperti jumlah, total, atau rata-rata.
Kasus penggunaan penting untuk bisnis yang mengandalkan pemrosesan data satu kali dan hanya
satu kali. Contohnya mencakup deteksi penipuan, deteksi ancaman jaringan, dan dasbor inventaris e-commerce.
Pilih mode streaming minimal sekali jika beban kerja Anda dapat mentoleransi duplikasi
data dan mungkin mendapatkan manfaat dari pengurangan biaya atau latensi. Contoh:
Workload yang penghapusan duplikasinya dilakukan di hilir dari
Dataflow. Misalnya, pipeline yang menulis ke BigQuery atau penyimpanan data SQL.
Pipeline hanya peta tanpa agregasi. Contohnya mencakup pemrosesan log, pengambilan data perubahan, atau tugas ekstrak, transformasi, dan pemuatan (ETL), yang pipeline-nya hanya melakukan transformasi per elemen, seperti terjemahan skema.
Pipeline yang sink outputnya tidak dapat menjamin pengiriman tepat satu kali, seperti
Pub/Sub. Dalam hal ini, penghapusan duplikat dalam pipeline mungkin tidak diperlukan, dan Anda dapat memperoleh manfaat dari pengurangan biaya dan latensi mode streaming minimal sekali.
Pipeline yang membaca dari Pub/Sub. Membaca dari Pub/Sub dioptimalkan secara signifikan saat menggunakan mode minimal satu kali.
Pertimbangan lainnya
Mode minimal sekali dapat mengurangi biaya dan latensi pipeline secara signifikan. Dampak persisnya bergantung pada spesifikasi pipeline. Uji streaming minimal sekali di bawah beban yang realistis untuk mengevaluasi dampaknya.
Saat menggunakan mode minimal satu kali, rasio duplikat data bergantung pada
jumlah percobaan ulang. Tingkat dasar biasanya rendah (<1%). Namun, lonjakan
dapat terjadi jika node pekerja gagal atau kondisi lain menyebabkan panggilan RPC berulang.
Mode streaming memengaruhi cara Streaming Engine memproses data, tetapi tidak mengubah semantik konektor I/O. Sebaiknya sesuaikan semantik I/O dengan mode streaming. Misalnya, jika Anda menggunakan mode streaming minimal sekali dengan konektor I/O BigQuery, tetapkan mode penulisan ke STORAGE_API_AT_LEAST_ONCE.
Template Dataflow yang disediakan Google
secara otomatis mengaktifkan opsi ini saat Anda menggunakan streaming minimal sekali.
Transformasi per elemen seperti Map tidak selalu bersifat idempoten. Misalnya, pertimbangkan fungsi yang menerima pesan dan menambahkan stempel waktu saat ini ke pesan tersebut. Dalam hal ini, kumpulan data duplikat dapat menghasilkan beberapa output yang berbeda.
Mode minimal sekali mungkin tidak sesuai untuk pipeline tersebut.
Menetapkan mode streaming
Pemrosesan tepat sekali adalah setelan default untuk semua tugas Dataflow. Untuk mengaktifkan mode streaming minimal sekali, tetapkan
streaming_mode_at_least_onceopsi layanan.
Kedua opsi ini saling eksklusif. Jika Anda tidak menyetel salah satu opsi ini, template akan menggunakan mode streaming default yang ditentukan oleh metadata template. Untuk mengetahui informasi selengkapnya, lihat
Template kustom.
Kolom metadata ini memungkinkan pengguna memilih mode streaming saat men-deploy
template di konsol Google Cloud . Kolom defaultStreamingMode bersifat opsional dan menentukan mode streaming default untuk template. Jika Anda tidak menentukan defaultStreamingMode dan template mendukung kedua mode, maka mode tepat sekali adalah defaultnya.
Untuk mengetahui informasi selengkapnya, lihat bagian berikut dalam dokumentasi template Dataflow:
Mode minimal sekali selalu menggunakan
penagihan berbasis resource,
dengan Anda ditagih untuk total resource yang digunakan oleh tugas Anda.
Biaya per unit
Unit Komputasi Streaming Engine adalah
sama terlepas dari mode streaming. Namun, dalam sebagian besar kasus, pipeline
menggunakan total resource yang jauh lebih sedikit saat menggunakan mode minimal sekali.
[[["Mudah dipahami","easyToUnderstand","thumb-up"],["Memecahkan masalah saya","solvedMyProblem","thumb-up"],["Lainnya","otherUp","thumb-up"]],[["Sulit dipahami","hardToUnderstand","thumb-down"],["Informasi atau kode contoh salah","incorrectInformationOrSampleCode","thumb-down"],["Informasi/contoh yang saya butuhkan tidak ada","missingTheInformationSamplesINeed","thumb-down"],["Masalah terjemahan","translationIssue","thumb-down"],["Lainnya","otherDown","thumb-down"]],["Terakhir diperbarui pada 2025-08-18 UTC."],[[["\u003cp\u003eDataflow supports two streaming modes: exactly-once, which ensures no data is lost or duplicated, and at-least-once, which guarantees no data loss but may result in duplicates.\u003c/p\u003e\n"],["\u003cp\u003eExactly-once mode is the default for all Dataflow streaming jobs and is recommended for pipelines requiring precise results, such as those involving aggregations or business-critical applications like fraud detection.\u003c/p\u003e\n"],["\u003cp\u003eAt-least-once mode can significantly reduce costs and latency and is suitable for workloads that can tolerate duplicate records, such as those with downstream deduplication or map-only pipelines.\u003c/p\u003e\n"],["\u003cp\u003eThe choice of streaming mode affects how Streaming Engine processes data but does not alter the semantics of I/O connectors; it is recommended to use the right connector to complement the streaming mode.\u003c/p\u003e\n"],["\u003cp\u003eYou can set at-least-once streaming mode using the \u003ccode\u003estreaming_mode_at_least_once\u003c/code\u003e service option in Java, Python, Go, console, or via the REST API, and it will be automatically applied in Dataflow templates when they support it.\u003c/p\u003e\n"]]],[],null,["# Set the pipeline streaming mode\n\nDataflow supports two modes for streaming jobs:\n\n- **Exactly-once mode**. This mode is the default for all Dataflow streaming jobs. In this mode, Dataflow ensures that records are not dropped or duplicated as the data moves through the pipeline.\n- **At-least-once mode**. This mode guarantees that records are processed at least once (that is, no input records are lost). However, duplicate records are possible in this mode. For use cases that can tolerate duplicates, at-least-once mode can significantly lower the cost and latency of your job.\n\nChoose which streaming mode to use\n----------------------------------\n\nChoose exactly-once mode if you need to ensure exact results from the pipeline\nand predictable semantics. For example:\n\n- Pipelines with [aggregations](https://beam.apache.org/documentation/basics/#aggregation), such as count, sum, or mean.\n- Business-critical use cases that rely on records being processed once and only once. Examples include fraud detection, network threat detection, and ecommerce inventory dashboards.\n\nChoose at-least-once streaming mode if your workload can tolerate duplicated\nrecords and might benefit from reduced cost or latency. For example:\n\n- Workloads where deduplication is performed downstream from Dataflow. For example, pipelines that write to BigQuery or a SQL datastore.\n- Map-only pipelines with no aggregations. Examples include log processing, change data capture, or extract, transform, and load (ETL) jobs, in which the pipeline performs only per-element transforms, such as schema translation.\n- Pipelines where the output sink can't guarantee exactly-once delivery, such as Pub/Sub. In that case, deduplication *within* the pipeline might be unnecessary, and you can benefit from the reduced cost and latency of at-least-once streaming mode.\n- Pipelines that read from Pub/Sub. Reading from Pub/Sub is significantly optimized when using at-least-once mode.\n\n### Additional considerations\n\n- At-least-once mode can significantly reduce the cost and latency of a\n pipeline. The exact impact depends on the specifics of the pipeline. Test\n at-least-once streaming under realistic loads to evaluate the impact.\n\n- When using at-least-once mode, the rate of duplicate records depends on the\n number of retries. The baseline rate is typically low (\\\u003c1%). However, spikes\n can occur if worker nodes fail or other conditions cause repeated RPC calls.\n\n- The streaming mode affects how Streaming Engine processes records, but does\n not change the semantics of I/O connectors. It is recommended to align your\n I/O semantics with the streaming mode. For example, if you use at-least-once\n streaming mode with the\n [BigQuery I/O connector](/dataflow/docs/guides/write-to-bigquery),\n set the write mode to `STORAGE_API_AT_LEAST_ONCE`.\n [Google-provided Dataflow templates](/dataflow/docs/guides/templates/provided-templates)\n automatically enable this option when you use at-least-once streaming.\n\n- Element-wise transforms such as `Map` are not always idempotent. For example,\n consider a function that receives a message and appends the current timestamp\n to it. In that case, a duplicate record can produce several distinct outputs.\n At-least-once mode might not be appropriate for that pipeline.\n\nSet the streaming mode\n----------------------\n\nExactly-once processing is the default setting for all Dataflow\njobs. To enable at-least-once streaming mode, set the\n`streaming_mode_at_least_once`\n[service option](/dataflow/docs/reference/service-options). \n\n### Java\n\n --dataflowServiceOptions=streaming_mode_at_least_once\n\n### Python\n\n --dataflow_service_options=streaming_mode_at_least_once\n\n### Go\n\n --dataflow_service_options=streaming_mode_at_least_once\n\nIf you don't specify the `streaming_mode_at_least_once` option, then\nDataflow uses exactly-once streaming mode.\n\nIf you set the `streaming_mode_at_least_once` option, Dataflow\nautomatically enables [Streaming Engine](/dataflow/docs/streaming-engine) with\n[resource-based billing](/dataflow/docs/streaming-engine#compute-unit-pricing).\n\nTo update the streaming mode on a running job,\n[launch a replacement job](/dataflow/docs/guides/updating-a-pipeline#Launching).\n\nSelect the streaming mode for a template\n----------------------------------------\n\nTo select the streaming mode when you run a Dataflow streaming\ntemplate, perform the following steps: \n\n### Console\n\n1. In the Google Cloud console, go to the Dataflow **Jobs**\n page.\n\n [Go to Jobs](https://console.cloud.google.com/dataflow/jobs)\n2. Click **Create job from template**.\n\n3. Select the template that you want to run from the\n **Dataflow template** drop-down menu.\n\n4. For **Streaming mode**, select the streaming mode. If the template supports\n only one mode, then this option is disabled.\n\n### gcloud\n\nTo enable at-least-once mode, set the `streaming_mode_at_least_once` option\nin the `additional-experiments` flag: \n\n --additional-experiments=streaming_mode_at_least_once\n\nTo enable exactly-once mode, set the `streaming_mode_exactly_once` option\nin the `additional-experiments` flag: \n\n --additional-experiments=streaming_mode_exactly_once\n\nThese two options are mutually exclusive. If you don't set one of these\noptions, then the template defaults to a streaming mode that is determined by\nthe template metadata. For more information, see\n[Custom templates](#custom-templates).\n\n### REST\n\nUse the `additionalExperiments` field in the\n[`FlexTemplateRuntimeEnvironment`](/dataflow/docs/reference/rest/v1b3/projects.locations.flexTemplates/launch#FlexTemplateRuntimeEnvironment)\n(Flex templates) or\n[`RuntimeEnvironment`](/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment)\n(classic templates) object. \n\n {\n additionalExperiments : [\"streaming_mode_at_least_once\"]\n ...\n }\n\n### Custom templates\n\nIf you create a custom template that supports at-least-once processing, add the\nfollowing top-level fields to the template metadata file: \n\n {\n \"streaming\": true,\n \"supportsAtLeastOnce\": true,\n \"supportsExactlyOnce\": true,\n \"defaultStreamingMode\": \"AT_LEAST_ONCE\"\n }\n\nThese metadata fields enable users to select the streaming mode when deploying\nthe template in the Google Cloud console. The `defaultStreamingMode` field\nis optional and specifies the default streaming mode for the template. If you\ndon't specify `defaultStreamingMode` and the template supports both modes, then\nexactly-once mode is the default.\n\nFor more information, see the following sections in the Dataflow\ntemplates documentation:\n\n- Flex templates: [Metadata](/dataflow/docs/guides/templates/configuring-flex-templates#metadata)\n- Classic templates: [Use metadata in your pipeline code](/dataflow/docs/guides/templates/creating-templates#use-metadata-in-your-pipeline-code)\n\nView a job's streaming mode\n---------------------------\n\nTo view the streaming mode for a job, go to the **Jobs** page in the\nGoogle Cloud console.\n\n[Go to Jobs](https://console.cloud.google.com/dataflow/jobs)\n\nThe streaming mode is also listed on the job details page, in the **Job\ninfo** panel.\n\nLimitations\n-----------\n\nAt-least-once streaming mode requires\n[Streaming Engine](/dataflow/docs/streaming-engine) with\n[resource-based billing](/dataflow/docs/streaming-engine#compute-unit-pricing).\n\nPricing\n-------\n\nAt-least-once mode always uses\n[resource-based billing](/dataflow/docs/streaming-engine#compute-unit-pricing),\nwhere you're billed for the total resources that are consumed by your job.\n\nThe per-unit cost of\n[Streaming Engine Compute Units](/dataflow/pricing#streaming-compute-units) is\nthe same regardless of the streaming mode. However, in most cases a pipeline\nconsumes significantly fewer total resources when using at-least-once mode.\n\nWhat's next\n-----------\n\n- Learn more about [exactly-once processing](/dataflow/docs/concepts/exactly-once)."]]
