Dataflow HPC highly parallel workloads

When you're working with high-volume grid-computing, use Dataflow to run HPC highly parallel workloads in a fully managed system. With Dataflow, you can run your highly parallel workloads in a single pipeline, improving efficiency and making your workflow easier to manage. The data stays in one system both for pre- and post-processing and for task processing. Dataflow automatically manages performance, scalability, availability, and security needs.
Follow this tutorial to see an end-to-end example of an HPC highly parallel pipeline that uses custom containers with C++ libraries.
Learn about best practices to consider when designing your HPC highly parallel pipeline.


Using GPUs in Dataflow jobs can accelerate image processing and machine learning processing tasks.
HSBC used a Dataflow HPC highly parallel workflow to increase calculation capacity and speed while lowering costs.
The Dataflow HPC highly parallel pipeline example and the corresponding source code are available on GitHub.