Stay organized with collections
Save and categorize content based on your preferences.
Dataflow supports two modes for streaming jobs:
Exactly-once mode. This mode is the default for all
Dataflow streaming jobs. In this mode, Dataflow
ensures that records are not dropped or duplicated as the data moves through
the pipeline.
At-least-once mode. This mode guarantees that records are processed at
least once (that is, no input records are lost). However, duplicate records
are possible in this mode. For use cases that can tolerate duplicates,
at-least-once mode can significantly lower the cost and latency of your job.
Choose which streaming mode to use
Choose exactly-once mode if you need to ensure exact results from the pipeline
and predictable semantics. For example:
Pipelines with
aggregations,
such as count, sum, or mean.
Business-critical use cases that rely on records being processed once and only
once. Examples include fraud detection, network threat detection, and
ecommerce inventory dashboards.
Choose at-least-once streaming mode if your workload can tolerate duplicated
records and might benefit from reduced cost or latency. For example:
Workloads where deduplication is performed downstream from
Dataflow. For example, pipelines that write to
BigQuery or a SQL datastore.
Map-only pipelines with no aggregations. Examples include log processing,
change data capture, or extract, transform, and load (ETL) jobs, in which the
pipeline performs only per-element transforms, such as schema translation.
Pipelines where the output sink can't guarantee exactly-once delivery, such
as Pub/Sub. In that case, deduplication within the pipeline might
be unnecessary, and you can benefit from the reduced cost and latency of
at-least-once streaming mode.
Pipelines that read from Pub/Sub. Reading from Pub/Sub is
significantly optimized when using at-least-once mode.
Additional considerations
At-least-once mode can significantly reduce the cost and latency of a
pipeline. The exact impact depends on the specifics of the pipeline. Test
at-least-once streaming under realistic loads to evaluate the impact.
When using at-least-once mode, the rate of duplicate records depends on the
number of retries. The baseline rate is typically low (<1%). However, spikes
can occur if worker nodes fail or other conditions cause repeated RPC calls.
The streaming mode affects how Streaming Engine processes records, but does
not change the semantics of I/O connectors. It is recommended to align your
I/O semantics with the streaming mode. For example, if you use at-least-once
streaming mode with the
BigQuery I/O connector,
set the write mode to STORAGE_API_AT_LEAST_ONCE.
Google-provided Dataflow templates
automatically enable this option when you use at-least-once streaming.
Element-wise transforms such as Map are not always idempotent. For example,
consider a function that receives a message and appends the current timestamp
to it. In that case, a duplicate record can produce several distinct outputs.
At-least-once mode might not be appropriate for that pipeline.
Set the streaming mode
Exactly-once processing is the default setting for all Dataflow
jobs. To enable at-least-once streaming mode, set the
streaming_mode_at_least_onceservice option.
These two options are mutually exclusive. If you don't set one of these
options, then the template defaults to a streaming mode that is determined by
the template metadata. For more information, see
Custom templates.
These metadata fields enable users to select the streaming mode when deploying
the template in the Google Cloud console. The defaultStreamingMode field
is optional and specifies the default streaming mode for the template. If you
don't specify defaultStreamingMode and the template supports both modes, then
exactly-once mode is the default.
For more information, see the following sections in the Dataflow
templates documentation:
At-least-once mode always uses
resource-based billing,
where you're billed for the total resources that are consumed by your job.
The per-unit cost of
Streaming Engine Compute Units is
the same regardless of the streaming mode. However, in most cases a pipeline
consumes significantly fewer total resources when using at-least-once mode.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-26 UTC."],[[["\u003cp\u003eDataflow supports two streaming modes: exactly-once, which ensures no data is lost or duplicated, and at-least-once, which guarantees no data loss but may result in duplicates.\u003c/p\u003e\n"],["\u003cp\u003eExactly-once mode is the default for all Dataflow streaming jobs and is recommended for pipelines requiring precise results, such as those involving aggregations or business-critical applications like fraud detection.\u003c/p\u003e\n"],["\u003cp\u003eAt-least-once mode can significantly reduce costs and latency and is suitable for workloads that can tolerate duplicate records, such as those with downstream deduplication or map-only pipelines.\u003c/p\u003e\n"],["\u003cp\u003eThe choice of streaming mode affects how Streaming Engine processes data but does not alter the semantics of I/O connectors; it is recommended to use the right connector to complement the streaming mode.\u003c/p\u003e\n"],["\u003cp\u003eYou can set at-least-once streaming mode using the \u003ccode\u003estreaming_mode_at_least_once\u003c/code\u003e service option in Java, Python, Go, console, or via the REST API, and it will be automatically applied in Dataflow templates when they support it.\u003c/p\u003e\n"]]],[],null,["# Set the pipeline streaming mode\n\nDataflow supports two modes for streaming jobs:\n\n- **Exactly-once mode**. This mode is the default for all Dataflow streaming jobs. In this mode, Dataflow ensures that records are not dropped or duplicated as the data moves through the pipeline.\n- **At-least-once mode**. This mode guarantees that records are processed at least once (that is, no input records are lost). However, duplicate records are possible in this mode. For use cases that can tolerate duplicates, at-least-once mode can significantly lower the cost and latency of your job.\n\nChoose which streaming mode to use\n----------------------------------\n\nChoose exactly-once mode if you need to ensure exact results from the pipeline\nand predictable semantics. For example:\n\n- Pipelines with [aggregations](https://beam.apache.org/documentation/basics/#aggregation), such as count, sum, or mean.\n- Business-critical use cases that rely on records being processed once and only once. Examples include fraud detection, network threat detection, and ecommerce inventory dashboards.\n\nChoose at-least-once streaming mode if your workload can tolerate duplicated\nrecords and might benefit from reduced cost or latency. For example:\n\n- Workloads where deduplication is performed downstream from Dataflow. For example, pipelines that write to BigQuery or a SQL datastore.\n- Map-only pipelines with no aggregations. Examples include log processing, change data capture, or extract, transform, and load (ETL) jobs, in which the pipeline performs only per-element transforms, such as schema translation.\n- Pipelines where the output sink can't guarantee exactly-once delivery, such as Pub/Sub. In that case, deduplication *within* the pipeline might be unnecessary, and you can benefit from the reduced cost and latency of at-least-once streaming mode.\n- Pipelines that read from Pub/Sub. Reading from Pub/Sub is significantly optimized when using at-least-once mode.\n\n### Additional considerations\n\n- At-least-once mode can significantly reduce the cost and latency of a\n pipeline. The exact impact depends on the specifics of the pipeline. Test\n at-least-once streaming under realistic loads to evaluate the impact.\n\n- When using at-least-once mode, the rate of duplicate records depends on the\n number of retries. The baseline rate is typically low (\\\u003c1%). However, spikes\n can occur if worker nodes fail or other conditions cause repeated RPC calls.\n\n- The streaming mode affects how Streaming Engine processes records, but does\n not change the semantics of I/O connectors. It is recommended to align your\n I/O semantics with the streaming mode. For example, if you use at-least-once\n streaming mode with the\n [BigQuery I/O connector](/dataflow/docs/guides/write-to-bigquery),\n set the write mode to `STORAGE_API_AT_LEAST_ONCE`.\n [Google-provided Dataflow templates](/dataflow/docs/guides/templates/provided-templates)\n automatically enable this option when you use at-least-once streaming.\n\n- Element-wise transforms such as `Map` are not always idempotent. For example,\n consider a function that receives a message and appends the current timestamp\n to it. In that case, a duplicate record can produce several distinct outputs.\n At-least-once mode might not be appropriate for that pipeline.\n\nSet the streaming mode\n----------------------\n\nExactly-once processing is the default setting for all Dataflow\njobs. To enable at-least-once streaming mode, set the\n`streaming_mode_at_least_once`\n[service option](/dataflow/docs/reference/service-options). \n\n### Java\n\n --dataflowServiceOptions=streaming_mode_at_least_once\n\n### Python\n\n --dataflow_service_options=streaming_mode_at_least_once\n\n### Go\n\n --dataflow_service_options=streaming_mode_at_least_once\n\nIf you don't specify the `streaming_mode_at_least_once` option, then\nDataflow uses exactly-once streaming mode.\n\nIf you set the `streaming_mode_at_least_once` option, Dataflow\nautomatically enables [Streaming Engine](/dataflow/docs/streaming-engine) with\n[resource-based billing](/dataflow/docs/streaming-engine#compute-unit-pricing).\n\nTo update the streaming mode on a running job,\n[launch a replacement job](/dataflow/docs/guides/updating-a-pipeline#Launching).\n\nSelect the streaming mode for a template\n----------------------------------------\n\nTo select the streaming mode when you run a Dataflow streaming\ntemplate, perform the following steps: \n\n### Console\n\n1. In the Google Cloud console, go to the Dataflow **Jobs**\n page.\n\n [Go to Jobs](https://console.cloud.google.com/dataflow/jobs)\n2. Click **Create job from template**.\n\n3. Select the template that you want to run from the\n **Dataflow template** drop-down menu.\n\n4. For **Streaming mode**, select the streaming mode. If the template supports\n only one mode, then this option is disabled.\n\n### gcloud\n\nTo enable at-least-once mode, set the `streaming_mode_at_least_once` option\nin the `additional-experiments` flag: \n\n --additional-experiments=streaming_mode_at_least_once\n\nTo enable exactly-once mode, set the `streaming_mode_exactly_once` option\nin the `additional-experiments` flag: \n\n --additional-experiments=streaming_mode_exactly_once\n\nThese two options are mutually exclusive. If you don't set one of these\noptions, then the template defaults to a streaming mode that is determined by\nthe template metadata. For more information, see\n[Custom templates](#custom-templates).\n\n### REST\n\nUse the `additionalExperiments` field in the\n[`FlexTemplateRuntimeEnvironment`](/dataflow/docs/reference/rest/v1b3/projects.locations.flexTemplates/launch#FlexTemplateRuntimeEnvironment)\n(Flex templates) or\n[`RuntimeEnvironment`](/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment)\n(classic templates) object. \n\n {\n additionalExperiments : [\"streaming_mode_at_least_once\"]\n ...\n }\n\n### Custom templates\n\nIf you create a custom template that supports at-least-once processing, add the\nfollowing top-level fields to the template metadata file: \n\n {\n \"streaming\": true,\n \"supportsAtLeastOnce\": true,\n \"supportsExactlyOnce\": true,\n \"defaultStreamingMode\": \"AT_LEAST_ONCE\"\n }\n\nThese metadata fields enable users to select the streaming mode when deploying\nthe template in the Google Cloud console. The `defaultStreamingMode` field\nis optional and specifies the default streaming mode for the template. If you\ndon't specify `defaultStreamingMode` and the template supports both modes, then\nexactly-once mode is the default.\n\nFor more information, see the following sections in the Dataflow\ntemplates documentation:\n\n- Flex templates: [Metadata](/dataflow/docs/guides/templates/configuring-flex-templates#metadata)\n- Classic templates: [Use metadata in your pipeline code](/dataflow/docs/guides/templates/creating-templates#use-metadata-in-your-pipeline-code)\n\nView a job's streaming mode\n---------------------------\n\nTo view the streaming mode for a job, go to the **Jobs** page in the\nGoogle Cloud console.\n\n[Go to Jobs](https://console.cloud.google.com/dataflow/jobs)\n\nThe streaming mode is also listed on the job details page, in the **Job\ninfo** panel.\n\nLimitations\n-----------\n\nAt-least-once streaming mode requires\n[Streaming Engine](/dataflow/docs/streaming-engine) with\n[resource-based billing](/dataflow/docs/streaming-engine#compute-unit-pricing).\n\nPricing\n-------\n\nAt-least-once mode always uses\n[resource-based billing](/dataflow/docs/streaming-engine#compute-unit-pricing),\nwhere you're billed for the total resources that are consumed by your job.\n\nThe per-unit cost of\n[Streaming Engine Compute Units](/dataflow/pricing#streaming-compute-units) is\nthe same regardless of the streaming mode. However, in most cases a pipeline\nconsumes significantly fewer total resources when using at-least-once mode.\n\nWhat's next\n-----------\n\n- Learn more about [exactly-once processing](/dataflow/docs/concepts/exactly-once)."]]