Stay organized with collections
Save and categorize content based on your preferences.
Pub/Sub Lite is
a real-time messaging service built for low cost and offers lower reliability
compared to Pub/Sub. Pub/Sub Lite offers zonal and regional
topics for storage.
The Pub/Sub Lite Spark Connector
supports Pub/Sub Lite as an input source to Apache Spark Structured Streaming in
the default micro-batch processing and experimental continuous
processing modes.
To get started, clone the java-pubsublite-spark GitHub repository:
git clone https://github.com/googleapis/java-pubsublite-spark
cd java-pubsublite-spark/samples
Python / Scala
The connector is available from the Maven Central repository.
You can download and provide it via the --packages option when using the
spark-submit command or set it via the spark.jars.packages
configuration property.
For more information
See Using Pub/Sub Lite with Apache Spark,
a quickstart that runs a Python script on a Dataproc cluster to
read and write data from and to Pub/Sub Lite.
Select the version of the Pub/Sub Lite Spark Connector here, then download its JAR on the linked page.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-03-21 UTC."],[[["Pub/Sub Lite is a real-time messaging service designed for low cost, providing zonal and regional storage topics, but with lower reliability than standard Pub/Sub."],["The Pub/Sub Lite Spark Connector allows Pub/Sub Lite to serve as an input source for Apache Spark Structured Streaming in both micro-batch and experimental continuous processing modes."],["A Java-based Spark example using Pub/Sub Lite with Dataproc is available in the `samples` directory of the `java-pubsublite-spark` GitHub repository."],["The Pub/Sub Lite Spark connector is available on Maven Central repository, and can be downloaded via the `--packages` option in spark-submit or with spark.jars.packages."],["You can read and write data from and to Pub/Sub Lite by following the instructions in the quickstart guide, \"Using Pub/Sub Lite with Apache Spark\", which runs a Python script on a Dataproc cluster."]]],[]]
