Cloud Data Fusion Plugins

Plugins are customizable modules that extend the capabilities of Cloud Data Fusion. The following is a complete list of plugins available in Cloud Data Fusion, but you can also build your own plugin with plugin APIs.

Contact us Contact us

Most popular

Quickly discover and access the most popular Cloud Data Fusion plugins.

BigQuery

This source reads the entire contents of a BigQuery table.

BigQuery Multi Table

This sink writes to multiple BigQuery tables. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.

BigQuery Table

This sink writes to a BigQuery table. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.

Cloud Storage

This source plugin reads objects from a path in a Cloud Storage bucket.

Cloud Storage

This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.

Multi Database Table

Reads from multiple tables within a database. Used with multi sink plugins like Cloud Storage multi sink and BigQuery multi sink to load data from entire database in a single pipeline.

MySQL

Reads from a MySQL instance using a configurable SQL query. Outputs one record for each row returned by the query.

Oracle

Reads from an Oracle table using a configurable SQL query. Outputs one record for each row returned by the query.

Salesforce

This source reads sObjects such as opportunities, contacts, accounts, leads, any custom object, etc. from Salesforce in batch.

SAP ODP

Connects to an SAP source for batch-based and delta-based data extraction through Operational Data Provisioning (ODP).

SQL Server

Reads from a SQL Server using a configurable SQL query. Outputs one record for each row returned by the query.

Google Cloud

See all of the plugins available for Google Cloud products including Cloud Storage, BigQuery, Cloud Bigtable, Datastore, Pub/Sub, and Google Workspace.

BigQuery

This source reads the entire contents of a BigQuery table.

BigQuery Argument Setter

Reads from a BigQuery table to fetch arguments to set in the pipeline.

BigQuery Execute

This plugin executes a BigQuery SQL Query.

BigQuery Multi Table

This sink writes to multiple BigQuery tables. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.

BigQuery Table

This sink writes to a BigQuery table. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.

Cloud Bigtable

Batch sink that writes data to Cloud Bigtable.

Cloud Bigtable

This source reads data from Cloud Bigtable.

Dataplex Sink

Ingests data to any of the Dataplex supported assets.

Dataplex Source

Reads data from Dataplex entities (tables) residing on Cloud Storage or BigQuery assets.

Datastore

This source reads data from Datastore. Datastore is a NoSQL document database built for automatic scaling and high performance.

Datastore

This sink writes data to Cloud Datastore. Datastore is a NoSQL document database built for automatic scaling and high performance.

Firestore

Reads from Cloud Firestore in batch pipelines.

Firestore

Plugin to write to Cloud Firestore in batch and streaming pipelines.

Cloud Storage

This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.

Cloud Storage

This source plugin reads objects from a path in a Cloud Storage bucket.

Cloud Storage Argument Setter

Reads from a Cloud Storage file to fetch arguments to set in the pipeline.

Cloud Storage Delete

This plugin deletes objects in a Cloud Storage bucket.

Cloud Storage Copy

This plugin copies objects from one Cloud Storage bucket to another. A single object can be copied, or a directory of objects can be copied.

Cloud Storage Create

This plugin creates objects in a Cloud Storage bucket.

Cloud Storage Done File Marker

Creates an empty DONE (or SUCCESS) file in the given Cloud Storage bucket upon pipeline completion, success, or failure, to orchestrate dependent processes.

Cloud Storage Move

This plugin moves objects from one Cloud Storage bucket to another. A single object can be moved, or a directory of objects can be moved.

Cloud Storage Multi FIle

This plugin is used in conjunction with the MultiTableDatabase source to write records from multiple databases into separate directories.

Cloud DLP Decrypt

Decrypts sensitive data that was earlier encrypted by Cloud Data Loss Prevention (DLP) using a reversible encryption transform, such as Format Preserving Encryption.

Cloud DLP PII Filter

Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.

Cloud DLP Redact

Redacts sensitive records using Cloud Data Loss Prevention (DLP) by matching some pre-defined DLP filters or a custom user-defined template.

Google Drive

Reads files from specified Google Drive directory via Google Drive API.

Google Drive

Sink plugin saves files from the pipeline to Google Drive directory via Google Drive API.

Google Sheets

Writes spreadsheets to specified Google Drive directory.

Google Sheets

Reads spreadsheets from specified Google Drive directory.

Pub/Sub

This sink writes to a Pub/Sub topic in streaming pipelines.

Pub/Sub

This sources reads from a Pub/Sub subscription in realtime.

Spanner

This sink writes to a Cloud Spanner table.

Spanner

This source reads from a Cloud Spanner table.

Speech To Text

This plugin converts audio files to text by using Speech-to-Text.

Aggregate

Plugins to aggregate functions like as avg, count, count(*), first, last, max, min, sum, collect including table joins, etc.

Group By

Groups by one or more fields, then performs aggregate functions such as avg, count, count(*), first, last, max, min, sum, collect on each group.

Joiner

Joins records from one or more input based on join keys. Supports inner and outer joins, selection, and renaming of output fields.

Top N

Top N returns the top "n" records from the input set, based on the criteria specified in the plugin configuration.

Window Aggregation

Applies aggregation functions over a window of data.

Alerting

Plugins for alerting.

Email

Sends an email at the end of a pipeline run.

Email by SendGrid

Action to send an email at the end of a pipeline run.

Cloud Storage Done File Marker

Creates an empty DONE (or SUCCESS) file in the given Cloud Storage bucket upon pipeline completion, success, or failure, to orchestrate dependent processes.

HTTP Callback

Performs an HTTP request at the end of a pipeline run.

Kafka Alert Publisher

Kafka Alert Publisher that allows you to publish alerts to Kafka as JSON objects.

MariaDB

Runs a MariaDB query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Netezza

Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Oracle

Runs an Oracle query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

PostgreSQL

Runs a PostgreSQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

SQL Server

Runs a SQL Server query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Teradata

Runs a Teradata query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

TMS

Publishes alerts to the CDAP Transactional Messaging System (TMS) as JSON objects.

Apache

Plugins for Apache.

Cassandra

Batch sink that writes data to Apache Cassandra.

Cassandra

Batch source that reads from Apache Cassandra.

HBase

Writes records to an HBase table with one field mapping to the row key, and all other record fields mapping to column qualifiers.

HBase

Batch source that reads from a column family in an HBase table.

Hive Bulk Export

Hive Export will take select query as input to run that query on hive table and store results in the specified location.

Hive Bulk Import

Imports data from a directory/file into a Hive table using Hive Load Statement.

HTTP to HDFS

Action to fetch data from an external http endpoint and create a file in HDFS.

Kafka Alert Publisher

Kafka Alert Publisher that allows you to publish alerts to Kafka as JSON objects.

Kafka Consumer

Kafka batch source that reads events from Apache Kafka in batch pipelines.

Kafka Consumer

Kafka streaming source that reads events from Apache Kafka in streaming pipelines.

Kafka Producer

Kafka sink that allows you to write events into CSV or JSON to Kafka.

Kudu

CDAP plugin for ingesting data into Apache Kudu. Plugin can be configured for both batch and real-time pipelines.

Kudu

CDAP plugin for reading data from Apache Kudu table.

PySpark Program

Executes user-provided Spark code in Python.

Repartitioner

This plugin re-partitions a Spark RDD.

Scala Spark Compute

Executes user-provided Spark code in Scala that transforms RDD to RDD with full access to all Spark features.

Scala Spark Program

Executes user-provided Spark code in Scala.

Spark

Executes user-provided Spark code in Scala that operates on an input RDD or Dataframe with full access to all Spark features.

Webserver Log Parser

Parses logs from any input source for relevant information such as URI, IP, browser, device, HTTP status code, and timestamp.

APIs

Plugins for various APIs.

Anaplan

Source plugin for loading data from an Anaplan model into a data pipeline.

Anaplan

Sink plugin for supporting data import from CDAP to Anaplan server.

Anaplan Function

Performs an Action/Process that is predefined in the given Anaplan model. This is most commonly used to prepare for the data import and export.

Facebook Ads

Reads Ads data from Facebook Insights API in batch pipelines.

GitHub

Reads from GitHub in batch pipelines.

Google Ads Multiple Reports

Reads all of the Google Ads reports from a Google Ads account in batch pipelines.

Google Ads Single Report

Reads the specified report type from Google Ads accounts in batch pipelines.

Google Analytics Batch

Reads from Google Analytics 360 in batch pipelines.

Google Analytics Streaming

Reads from Google Analytics 360 in streaming pipelines.

Google Drive

Reads files from specified Google Drive directory via Google Drive API.

Google Drive

Sink plugin saves files from the pipeline to Google Drive directory via Google Drive API.

Google Search Console

Reads from Google Search Console in batch pipelines.

Google Sheets

Writes spreadsheets to specified Google Drive directory.

Google Sheets

Reads spreadsheets from specified Google Drive directory.

HTTP

Reads data in JSON, XML, CSV, TSV, TEXT and BLOB formats from HTTP/HTTPS APIs in batch pipelines.

HTTP

Sink plugin to send the messages from the pipeline to an external http endpoint.

HTTP Callback

Performs an HTTP request at the end of a pipeline run.

HTTP Poller

This is a streaming source that will fetch data from a specified URL at a given interval and pass the results to the next plugin.

HTTP Streaming

Reads data in JSON, XML, CSV, TSV, TEXT and BLOB formats from HTTP/HTTPS APIs in streaming pipelines.

HTTP to HDFS

Action to fetch data from an external http endpoint and create a file in HDFS.

HubSpot

Plugin to write data to HubSpot CRM in batch and streaming pipelines.

HubSpot

Reads data from HubSpot CRM in streaming pipelines.

HubSpot

Reads data from HubSpot CRM in batch pipelines.

Salesforce

This source reads sObjects such as opportunities, contacts, accounts, leads, any custom object, etc. from Salesforce in batch.

Salesforce

This source tracks updates in Salesforce sObjects. Examples of sObjects are opportunities, contacts, accounts, leads, any custom object, etc.

Salesforce

A batch sink that inserts sObjects such as opportunities, contacts, accounts, leads, any custom objects, etc. into Salesforce.

Salesforce Marketing Cloud

This source tracks updates in Salesforce sObjects such as opportunities, contacts, accounts, leads, any custom object, etc.

Salesforce Marketing Cloud

This sink inserts records into a Salesforce Marketing Cloud (SFMC) Data Extension via server-to-server integration with the SFMC API.

Salesforce Multi Objects

This source reads multiple sObjects from Salesforce. It should be used in conjunction with multi-sinks.

ServiceNow

Reads from one or multiple tables within ServiceNow.

Speech To Text

This plugin converts audio files to text by using Speech-to-Text.

Splunk

Reads from Splunk in streaming pipelines.

Splunk

Reads from Splunk in batch pipelines.

Zendesk

Reads objects such as Article Comments, Post Comments, Requests Comments, Ticket Comments, Groups, Organizations, Satisfaction Ratings, Tags, Ticket Fields, Ticket Metrics, Ticket Metric Events, Tickets, Users from Zendesk.

Zendesk Multi Objects

Multi-source that reads multiple objects from Zendesk and routes them to specific destinations. Should be used in conjunction with a multi-sink.

Zuora

Plugin to write data to Zuora in batch and streaming pipelines.

Zuora

Reads data from Zuora in batch pipelines.

Avro

Plugins for Avro.

File

Writes to a filesystem in various formats.

File

This source is used whenever you need to read from a distributed file system.

Cloud Storage

Writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.

AWS

Available plugins for AWS.

Amazon S3

This source is used whenever you need to read from Amazon S3.

Amazon S3

This sink is used to write to Amazon S3 in various formats.

Amazon S3 Client

The Amazon S3 Client Action is used to work with S3 buckets and objects before or after the execution of a pipeline.

Kinesis Sink

Kinesis sink that outputs to a specified Amazon Kinesis Stream.

Kinesis Source

Spark streaming source that reads from AWS Kinesis streams.

Redshift to S3

The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.

S3 to Redshift

S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.

Cleansing

Plugins for cleansing.

Deduplicate

De-duplicates records either using one or more fields or by using the record as a whole. Additionally, it supports logically choosing a record out of the duplicate records based on a filter field. Supported logical functions are max, min, first, and last.

Distinct

De-duplicates input records so that all output records are distinct.

Fast Filter

Filters out messages based on a specified criteria.

Cloud DLP PII Filter

Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.

Wrangler

This plugin applies data transformation directives generated either through an interactive UI or manually on your data records.

Cloud Storage

Available plugins for cloud storage.

Amazon S3

This source is used whenever you need to read from Amazon S3.

Azure Blob Store

Batch source to use Microsoft Azure Blob Storage as a source.

Azure Decompress

Azure Decompress Action plugin decompress gz files from a container on Azure Storage Blob service into another container.

Cloud Storage to Snowflake

Loads data from external or internal staged files on Cloud Storage to an existing Snowflake table.

Cloud Storage

This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.

Cloud Storage

This plugin reads objects from a path in a Cloud Storage bucket.

Cloud Storage Argument Setter

Reads from a Cloud Storage file to fetch arguments to set in the pipeline.

Cloud Storage Delete

This plugin deletes objects in a Cloud Storage bucket.

Cloud Storage Copy

This plugin copies objects from one Cloud Storage bucket to another. A single object can be copied, or a directory of objects can be copied.

Cloud Storage Create

This plugin creates objects in a Cloud Storage bucket.

Cloud Storage Move

This plugin moves objects from one Cloud Storage bucket to another. A single object can be moved, or a directory of objects can be moved.

Cloud Storage Multi FIle

This plugin is used in conjunction with the MultiTableDatabase source to write records from multiple databases into separate directories.

Cloud Storage Done File Marker

Creates an empty DONE (or SUCCESS) file in the given Cloud Storage bucket upon pipeline completion, success, or failure, to orchestrate dependent processes.

Microsoft Azure Delete

Azure Delete Action plugin deletes a container on Azure Storage Blob service.

Snowflake to Cloud Storage

Unloads data from a Snowflake table to an internal or external Cloud Storage location (stage).

Code

Plugins to execute your own code. 

JavaScript

Executes user-provided JavaScript that transforms one record into zero or more records.

PySpark Program

Executes user-provided Spark code in Python.

Python Evaluator

Executes user-provided Python code for transforming input records, which are available as Python dictionary.

Scala Spark Compute

Executes user-provided Spark code in Scala that transforms RDD to RDD with full access to all Spark features.

Scala Spark Program

Executes user-provided Spark code in Scala.

Spark

Executes user-provided Spark code in Scala that operates on an input RDD or Dataframe with full access to all Spark features.

Compression

Plugins for compression.

Compressor

Compresses configured fields using different compression algorithms such as SNAPPY, ZIP, and GZIP.

Azure Decompress

The Decompress Action is used to expand archived or compressed (.zip, .gz, .tar, .tar.gz, etc) files before further processing in a pipeline.

Field Decompresser

Decompresses configured fields using compression algorithms such as SNAPPY, ZIP, and GZIP.

Conditional

Pipelines also offer control flow plugins in the form of conditions. Condition plugins allow you to branch your pipeline into two separate paths, depending on whether the specified condition predicate evaluates to true or false.

Conditional

A control flow plugin that allows conditional execution within pipelines based on expressions.

Error Collector

Collects errors emitted from the previous stage and adds an error message, code, and stage to the record.

Fail Pipeline

Batch Sink that causes a pipeline to fail immediately after receiving the first record.

Null Field Splitter

This plugin sends records with a null value for a given field to the null port and those with a non-null value to the nonnull port.

Decompress

Routes a record to an appropriate port based on the evaluation of an expression on the value of one or more of its fields.

Union Splitter

The union splitter is used to split data by a union schema, so that type-specific logic can be done downstream.

Control Flow

Plugins for control flow.

Anaplan Function

Performs an Action/Process that is predefined in the given Anaplan model. This is most commonly used to prepare for the data import and export.

BigQuery Execute

Execute a SQL query at the beginning or end of a pipeline.

Database

Action that runs a database command.

Database Query

Runs a database query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

DB2 Execute

Action that runs a DB2 command.

Azure Decompress

The Decompress Action is used to expand archived or compressed (.zip, .gz, .tar, .tar.gz, etc) files before further processing in a pipeline.

File Contents Checketer

This action plugin can be used to check if a file is empty or if the contents of a file match a given pattern.

File Delete

Deletes a file or files.

File Move

Moves a file or files.

FTP Copy

Copy files from FTP server to the specified destination.

FTP Put

Copy files to FTP server from a specified destination.

Cloud Storage Delete

This plugin deletes objects in a Cloud Storage bucket.

Cloud Storage Copy

This plugin copies objects from one Cloud Storage bucket to another. A single object can be copied, or a directory of objects can be copied.

Cloud Storage Create

This plugin creates objects in a Cloud Storage bucket.

Cloud Storage Move

This plugin moves objects from one Cloud Storage bucket to another. A single object can be moved, or a directory of objects can be moved.

HTTP to HDFS

Action to fetch data from an external http endpoint and create a file in HDFS.

MariaDB Execute

Action that runs a MariaDB command.

MySQL Execute

Action that runs a MySQL command.

Netezza Execute

Action that runs a Netezza command.

Oracle Execute

Action that runs an Oracle command.

Oracle Export

An Action plugin to efficiently export data from Oracle to a file system using Oracle's command-line tools.

PostgreSQL Execute

Action that runs a PostgreSQL command.

Redshift to S3

The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.

Remote Program Executor

Establishes an SSH connection with remote machine to execute command on that machine.

S3 to Redshift

S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.

SFTP Copy

SFTP copy allows copying of the files from the specified directory on SFTP servers to the specified destination.

SFTP Delete

CDAP Action plugin to delete the specified files from an SFTP server.

SFTP Put

SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used to filter only the files that are of interest.

Snowflake Run SQL

Runs a specified SQL query in Snowflake.

SQL Server Execute

Action that runs a SQL Server command.

Teradata Execute

Action that runs a Teradata command.

Windows Share Copy

Copies a file or files on a Microsoft Windows share to a filesystem.

CRM

Plugins for CRM platforms like Salesforce.

HubSpot

Plugin to write data to HubSpot CRM in batch and streaming pipelines.

HubSpot

Reads data from HubSpot CRM in streaming pipelines.

HubSpot

Reads data from HubSpot CRM in batch pipelines.

Salesforce

This source reads sObjects such as opportunities, contacts, accounts, leads, any custom object, etc. from Salesforce in batch.

Salesforce

A batch sink that inserts sObjects such as opportunities, contacts, accounts, leads, any custom objects, etc. into Salesforce.

Salesforce

This source tracks updates in Salesforce sObjects. Examples of sObjects are opportunities, contacts, accounts, leads, any custom object, etc.

Salesforce Marketing Cloud

This source tracks updates in Salesforce sObjects such as opportunities, contacts, accounts, leads, any custom object, etc.

Salesforce Marketing Cloud

This sink inserts records into a Salesforce Marketing Cloud (SFMC) Data Extension via server-to-server integration with the SFMC API.

Salesforce Multi Objects

This source reads multiple sObjects from Salesforce. It should be used in conjunction with multi-sinks.

CSV

Plugins for CSV.

CSV Formatter

Formats input into a CSV Record using formats such as DELIMITED, EXCEL, MYSQL, RFC4180, and TDF.

CSV Parser

Parses a CSV Record formatted as DEFAULT, EXCEL, MYSQL, RFC4180, Tab Delimited, Pipe Delimited into a Structured Record.

File

Writes to a filesystem in various formats.

File

This source is used whenever you need to read from a distributed file system.

Cloud Storage

This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.

Kafka Producer

Kafka sink that allows you to write events into CSV or JSON to Kafka.

Database

Plugins for your databases.

Cloud Bigtable

Batch sink that writes data to Cloud Bigtable.

Cloud Bigtable

This source reads data from Cloud Bigtable.

Database

Action that runs a database command.

Database

Reads from a database using a configurable SQL query. Outputs one record for each row returned by the query.

Database

Writes records to a database table. Each record will be written to a row in the table.

Database Argument Setter

Reads from a relational database table to fetch arguments to set in the pipeline.

Database Query

Runs a database query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

DB2

Reads from a DB2 using a configurable SQL query. Outputs one record for each row returned by the query.

DB2

Runs a DB2 query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

DB2 Execute

Action that runs a DB2 command.

IBM DB2

Writes records to a DB2 table. Each record will be written to a row in the table.

MariaDB

Writes records to a MariaDB table. Each record will be written to a row in the table.

MariaDB

Reads from a MariaDB using a configurable SQL query. Outputs one record for each row returned by the query.

MariaDB

Runs a MariaDB query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

MariaDB Execute

Action that runs a MariaDB command.

Multi Database Table

Reads from multiple tables within a database. Used with multi sink plugins like, Cloud Storage multi sink and BigQuery multi sink to load data from entire database in a single pipeline.

MySQL Query

Runs a MySQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

MySQL

Writes records to a MySQL table. Each record will be written to a row in the table.

MySQL

Reads from a MySQL instance using a configurable SQL query. Outputs one record for each row returned by the query.

MySQL Execute

Action that runs a MySQL command.

Oracle

Reads from an Oracle table using a configurable SQL query. Outputs one record for each row returned by the query.

Oracle

Writes records to an Oracle table. Each record will be written to a row in the table.

Oracle

Runs an Oracle query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Oracle Execute

Action that runs an Oracle command.

Oracle Export

An Action plugin to efficiently export data from Oracle to a file system using Oracle's command-line tools.

PostgreSQL

Reads from a PostgreSQL using a configurable SQL query. Outputs one record for each row returned by the query.

PostgreSQL

Writes records to a PostgreSQL table. Each record will be written to a row in the table.

PostgreSQL

Runs a PostgreSQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

PostgreSQL Execute

Action that runs a PostgreSQL command.

Spanner

This sink writes to a Cloud Spanner table.

Spanner

This source reads from a Cloud Spanner table.

SQL Server

Writes records to a SQL Server table. Each record will be written to a row in the table.

SQL Server

Runs a SQL Server query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

SQL Server

Reads from a SQL Server using a configurable SQL query. Outputs one record for each row returned by the query.

SQL Server Execute

Action that runs a SQL Server command.

IBM DB2

DB2 Runs a DB2 query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Data Warehouse

Plugins for your data warehouse.

BigQuery

This source reads the entire contents of a BigQuery table.

BigQuery Argument Setter

Reads from a BigQuery table to fetch arguments to set in the pipeline.

BigQuery Execute

This plugin executes a BigQuery SQL query.

BigQuery Multi Table

This sink writes to multiple BigQuery tables. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.

BigQuery Table

This sink writes to a BigQuery table. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.

Cloud Storage to Snowflake

Loads data from external or internal staged files on Cloud Storage to an existing Snowflake table.

Hive Bulk Export

Hive Export will take select query as input to run that query on hive table and store results in the specified location.

Hive Bulk Import

Imports data from a directory/file into a Hive table using Hive Load Statement.

Netezza

Writes records to a Netezza table. Each record will be written to a row in the table.

Netezza

Reads from a Netezza using a configurable SQL query. Outputs one record for each row returned by the query.

Netezza

Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Netezza Execute

Action that runs a Netezza command.

Redshift to S3

The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.

S3 to Redshift

S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.

SCD2 Spark Compute

Creates an SCD2 representation of a record by grouping source records by the specified key and generating the start and end (expiry) dates for it.

Snowflake

Writes data to a Snowflake table in batch pipelines.

Snowflake

Reads data from Snowflake tables in batch pipelines.

Snowflake Run SQL

Runs a specified SQL query in Snowflake.

Snowflake to Cloud Storage

Unloads data from a Snowflake table to an internal or external Cloud Storage location (stage).

Teradata

Writes records to a Teradata table. Each record will be written to a row in the table.

Teradata

Reads from a Teradata using a configurable SQL query. Outputs one record for each row returned by the query.

Teradata

Runs a Teradata query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Teradata

Action that runs a Teradata command.

Vertica Bulk Export Action

Bulk exports data in a Vertica table into a file.

Vertica Bulk Import Action

Vertica Bulk Import Action plugin reads all the files in a given directory and bulk imports the contents into a Vertica table.

DLP

Plugins for Cloud Data Loss Prevention (DLP).

Cloud DLP Decrypt

Decrypts sensitive data that was earlier encrypted by Cloud Data Loss Prevention (DLP) using a reversible encryption transform, such as Format Preserving Encryption.

Cloud DLP PII Filter

Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.

Cloud DLP Redact

Redacts sensitive records using Cloud Data Loss Prevention (DLP) by matching some pre-defined DLP filters or a custom user-defined template.

Dynamic

Dynamic plugins.

Argument Setter

Performs an HTTP request to fetch arguments to set in the pipeline.

Database Argument Setter

Reads from a relational database table to fetch arguments to set in the pipeline.

Cloud Storage Argument Setter

Reads from a Cloud Storage file to fetch arguments to set in the pipeline.

Email

Plugins for email.

Email

Sends an email at the end of a pipeline run.

Email by SendGrid

Action to send an email at the end of a pipeline run.

SendGrid

Reads data about existing marketing campaigns, email analytics, bounces, spam reports from SendGrid.

SendGrid

Writes data to SendGrid in batch and streaming pipelines.

File

File plugins.

CSV Formatter

Formats input into a CSV Record using formats such as DELIMITED, EXCEL, MYSQL, RFC4180, and TDF.

CSV Parser

Parses a CSV Record formatted as DEFAULT, EXCEL, MYSQL, RFC4180, Tab Delimited, Pipe Delimited into a Structured Record.

File

Writes to a filesystem in various formats.

File

This source is used whenever you need to read from a distributed file system.

File

File streaming source that watches a directory and streams file contents of any new files added to the directory.

File Appender

This sink plugin appends to an existing file until it hits either a size or an age based threshold.

File Contents Checketer

This action plugin can be used to check if a file is empty or if the contents of a file match a given pattern.

File Delete

Deletes a file or files.

File Move

Moves a file or files.

FTP

Batch source to read from FTP (ftp://..) or SFTP (sftp://...).

FTP Copy

Copy files from FTP server to the specified destination.

FTP Put

Copy files to FTP server from a specified destination.

PDF Extractor

This transform leverages the Apache PDF Box library to extract text and metadata from a PDF file.

Redshift to S3

The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.

Amazon S3

This sink is used to write to Amazon S3 in various formats.

S3 to Redshift

S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.

SFTP Copy

SFTP copy allows copying of the files from the specified directory on SFTP servers to the specified destination.

SFTP Delete

CDAP Action plugin to delete the specified files from an SFTP server.

SFTP Put

SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used to filter only the files that are of interest.

Filter

Filter plugins.

Deduplicate

De-duplicates records either using one or more fields or by using the record as a whole. Additionally, it supports logically choosing a record out of the duplicate records based on a filter field. Supported logical functions are max, min, first, and last.

Distinct

De-duplicates input records so that all output records are distinct.

Fast Filter

Filters out messages based on a specified criteria.

Cloud DLP PII Filter

Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.

Null Field Splitter

This plugin sends records with a null value for a given field to the null port and those with a non-null value to the nonnull port.

Sampling

Samples large dataset flowing through this plugin to pull random records using systematic or reservoir sampling.

Top N

Top N returns the top "n" records from the input set, based on the criteria specified in the plugin configuration.

FTP

FTP plugins.

FTP

Batch source to read from FTP (ftp://..) or SFTP (sftp://...).

FTP Copy

Copy files from FTP server to the specified destination.

FTP Put

Copy files to FTP server from a specified destination.

SFTP Copy

SFTP copy allows copying of the files from the specified directory on SFTP servers to the specified destination.

SFTP Delete

CDAP Action plugin to delete the specified files from an SFTP server.

SFTP Put

SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used to filter only the files that are of interest.

Google Ads

Plugins for Google Ads.

Google Ads Multiple Reports

Reads all of the Google Ads reports from a Google Ads account in batch pipelines.

Google Ads Single Report

Reads the specified report type from Google Ads accounts in batch pipelines.

Google Search Console

Reads from Google Search Console in batch pipelines.

Google Analytics 360

Plugins for Google Analytics 360.

Google Analytics Batch

Reads from Google Analytics 360 in batch pipelines.

Google Analytics Streaming

Reads from Google Analytics 360 in streaming pipelines.

Google Workspace

Plugins for Google Workspace including Google Sheets and Google Drive.

Google Drive

Reads files from specified Google Drive directory via Google Drive API.

Google Drive

Sink plugin saves files from the pipeline to Google Drive directory via Google Drive API.

Google Sheets

Reads spreadsheets from specified Google Drive directory.

Google Sheets

Writes spreadsheets to specified Google Drive directory.

Hadoop

Plugins for Hadoop.

HBase

Writes records to an HBase table with one field mapping to the row key, and all other record fields mapping to column qualifiers.

HBase

Batch source that reads from a column family in an HBase table.

Hive Bulk Export

Hive Export will take select query as input to run that query on hive table and store results in the specified location.

Hive Bulk Import

Imports data from a directory/file into a Hive table using Hive Load Statement.

HTTP to HDFS

Action to fetch data from an external http endpoint and create a file in HDFS.

HDFS

Plugins for HDFS.

XML Reader

The XML Reader plugin is a source plugin that allows users to read XML files stored on HDFS.

HTTP to HDFS

Action to fetch data from an external http endpoint and create a file in HDFS.

HP

Plugins for HP.

Vertica Bulk Export Action

Bulk exports data in a Vertica table into a file.

Vertica Bulk Import Action

Vertica Bulk Import Action plugin reads all the files in a given directory and bulk imports the contents into a Vertica table.

HTTP

Plugins for HTTP.

Argument Setter

Performs an HTTP request to fetch arguments to set in the pipeline.

IBM

Plugins for IBM.

DB2

Reads from a DB2 using a configurable SQL query. Outputs one record for each row returned by the query.

DB2

Runs a DB2 query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

DB2 Execute

Action that runs a DB2 command.

IBM DB2

Writes records to a DB2 table. Each record will be written to a row in the table.

Netezza

Writes records to a Netezza table. Each record will be written to a row in the table.

Netezza

Reads from a Netezza using a configurable SQL query. Outputs one record for each row returned by the query.

Netezza

Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Netezza Execute

Action that runs a Netezza command.

JDBC

Plugins for JDBC.

Database

Reads from a database using a configurable SQL query. Outputs one record for each row returned by the query.

Database

Writes records to a database table. Each record will be written to a row in the table.

MongoDB

Reads documents from a MongoDB collection and converts each document into a StructuredRecord with the help of a specified schema.

MongoDB

This sink writes to a MongoDB collection.

MySQL Query

Runs a MySQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

MySQL

Writes records to a MySQL table. Each record will be written to a row in the table.

MySQL

Reads from a MySQL instance using a configurable SQL query. Outputs one record for each row returned by the query.

MySQL Execute

Action that runs a MySQL command.

Netezza

Writes records to a Netezza table. Each record will be written to a row in the table.

Netezza

Reads from a Netezza using a configurable SQL query. Outputs one record for each row returned by the query.

Netezza

Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Netezza Execute

Action that runs a Netezza command.

Oracle

Reads from an Oracle table using a configurable SQL query. Outputs one record for each row returned by the query.

Oracle

Writes records to an Oracle table. Each record will be written to a row in the table.

Oracle

Runs an Oracle query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Oracle Execute

Action that runs an Oracle command.

Oracle Export

An Action plugin to efficiently export data from Oracle to a file system using Oracle's command-line tools.

PostgreSQL

Reads from a PostgreSQL using a configurable SQL query. Outputs one record for each row returned by the query.

PostgreSQL

Writes records to a PostgreSQL table. Each record will be written to a row in the table.

PostgreSQL

Runs a PostgreSQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

PostgreSQL Execute

Action that runs a PostgreSQL command.

SQL Server

Writes records to a SQL Server table. Each record will be written to a row in the table.

SQL Server

Runs a SQL Server query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

SQL Server

Reads from a SQL Server using a configurable SQL query. Outputs one record for each row returned by the query.

SQL Server Execute

Action that runs a SQL Server command.

Teradata

Writes records to a Teradata table. Each record will be written to a row in the table.

Teradata

Reads from a Teradata using a configurable SQL query. Outputs one record for each row returned by the query.

Teradata

Runs a Teradata query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Teradata Execute

Action that runs a Teradata command.

JSON

Plugins for JSON.

Elasticsearch

Writes data to Elasticsearch as a JSON string, then indexes it using the index, type, and idField specified by the user.

Cloud Storage

Writes records to one or more files in a directory on Cloud Stoage in various formats such as CSV, Avro, Parquet, and JSON.

JSON Formatter

Formats a structured record as JSON Object.

JSON Parser

Parses an input JSON event into a record using JSON paths for extracting fields from complex nested input JSON.

Kafka Producer

Kafka sink that allows you to write events into CSV or JSON to Kafka.

XML to JSON

Accepts a field that contains a properly formatted XML string and outputs a properly formatted JSON string version of the data.

Kafka

Plugins for Kafka.

Confluent

Reads from Confluent Kafka in streaming pipelines.

Confluent

Plugin to write to Confluent Kafka in streaming pipelines.

Kafka Consumer

Kafka batch source that reads events from Apache Kafka in batch pipelines.

Kafka Consumer

Kafka streaming source that reads events from Apache Kafka in streaming pipelines.

Kafka Alert Publisher

Kafka Alert Publisher that allows you to publish alerts to Kafka as JSON objects.

Kafka Producer

Kafka sink that allows you to write events into CSV or JSON to Kafka.

Logs

Plugins for logs.

Splunk

Reads from Splunk in batch pipelines.

Splunk

Reads from Splunk in streaming pipelines.

Webserver Log Parser

Parses logs from any input source for relevant information such as URI, IP, browser, device, HTTP status code, and timestamp.

Mapping

Plugins for mapping.

Add field

Adds a new field to each record. The field value can either be a new UUID, or it can be set directly through configuration.

Clone Record

Makes a copy of every input record received for a configured number of times on the output.

Create Record

Plugin to generate hierarchical structures from flat schemas.

Date Transform

This transform takes a date in either a Unix timestamp or a string and converts it to a formatted string. (Macro-enabled).

Cloud DLP Redact

Redacts sensitive records using Cloud Data Loss Prevention (DLP) by matching some pre-defined DLP filters or a custom user-defined template.

Lookup

Performs a lookup of a given field within a lookup dataset by matching it with input dataset and includes the field, and it's value in resulting dataset.

Multi Field Adder

Multi Field Adder transform allows you to add one or more fields to the output.

NGramTransform Spark Compute

Transforms the input features into n-grams.

Projection

The Projection transform lets you drop, keep, rename, and cast fields to a different type.

Record Splitter

Given a field and a delimiter, emits one record for each split of the field.

Row Denormalizer

Converts raw data into denormalized data based on a key column.

SCD2 Spark Compute

Creates an SCD2 representation of a record by grouping source records by the specified key and generating the start and end (expiry) dates for it.

Transpose

Normalizes data by converting columns to rows.

Value Mapper

Value Mapper is a transform plugin that maps string values of a field in the input record to a mapping value using a mapping dataset.

Wrangler

This plugin applies data transformation directives generated either through an interactive UI or manually on your data records.

Masking

Plugins for masking.

Field Decoder

Decodes configured fields using decoding methods like STRING_BASE64, BASE64, BASE32, STRING_BASE32, and HEX.

Field Decryptor

Decrypts one or more fields in input records using a keystore that must be present on all nodes of the cluster.

Field Encryptor

Encrypts one or more fields in input records using a Java keystore that must be present on all nodes of the cluster.

Field Endcoder

Encodes configured fields using encoding methods like STRING_BASE64, BASE64, BASE32, STRING_BASE32, and HEX.

MD5/SHA Field Encoder

Encodes field values using digest algorithms like MD2, MD5, SHA1, SHA256, SHA384 and SHA512.

Microsoft

Plugins for Microsoft Excel and Windows.

Excel

The Excel plugin provides user the ability to read data from one or more Excel file(s).

Windows Share Copy

Copies a file or files on a Microsoft Windows share to a filesystem.

Microsoft Azure

Plugins for Microsoft Azure data warehouse, databases, cloud storage, and more.

ADLS

Writes data to Azure Data Lake Store directory in avro, orc, or text format.

ADLS Multi File

This sink writes to multiple Azure Data Lake directory.

ADLS Delete

Deletes a file or files within ADLS file system.

Azure Blob Store

Batch source to use Microsoft Azure Blob Storage as a source.

Azure Data Lake Store

Azure Data Lake Store Batch Source reads data from Azure Data Lake Store files and converts it into StructuredRecord.

Azure Decompress

Azure Decompress Action plugin decompress gz files from a container on Azure Storage Blob service into another container.

Azure Face Extractor

This transform leverages the Azure Cognitive APIs to extract faces and other metadata, including emotions, from a set of images.

Microsoft Azure Delete

Azure Delete Action plugin deletes a container on Azure Storage Blob service

SQL Server

Runs a SQL Server query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

SQL Server

Reads from a SQL Server using a configurable SQL query. Outputs one record for each row returned by the query.

SQL Server

Writes records to a SQL Server table. Each record will be written to a row in the table.

SQL Server Execute

Action that runs a SQL Server command.

Nested

Plugins for nested data.

Flatten Record

Plugin to transform nested data into flat data.

NoSQL

Plugins for NoSQL.

Cassandra

Batch sink that writes data to Apache Cassandra.

Cassandra

Batch source that reads from Apache Cassandra.

Datastore

This source reads data from Datastore. Datastore is a NoSQL document database built for automatic scaling and high performance.

Datastore

This sink writes data to Cloud Datastore. Datastore is a NoSQL document database built for automatic scaling and high performance.

Firestore

Reads from Cloud Firestore in batch pipelines.

Firestore

Plugin to write to Cloud Firestore in batch and streaming pipelines.

HBase

Writes records to an HBase table with one field mapping to the row key, and all other record fields mapping to column qualifiers.

HBase

Batch source that reads from a column family in an HBase table.

MongoDB

Reads documents from a MongoDB collection and converts each document into a StructuredRecord with the help of a specified schema.

MongoDB

This sink writes to a MongoDB collection.

Oracle

Plugins for Oracle.

Oracle

Reads from an Oracle table using a configurable SQL query. Outputs one record for each row returned by the query.

Oracle

Writes records to an Oracle table. Each record will be written to a row in the table.

Oracle

Runs an Oracle query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Oracle Execute

Action that runs an Oracle command.

Oracle Export

An Action plugin to efficiently export data from Oracle to a file system using Oracle's command-line tools.

Parquet

Plugins for Parquet.

File

Writes to a filesystem in various formats.

File

This source is used whenever you need to read from a distributed file system.

Cloud Storage

Writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.

Python

Plugins for Python.

PySpark Program

Executes user-provided Spark code in Python.

Python Evaluator

Executes user-provided Python code for transforming input records, which are available as Python dictionary.

S3

Plugins for S3.

Amazon S3

This source is used whenever you need to read from Amazon S3.

SAP

Accelerator: SAP Order to Cash

Accelerates the SAP Table Batch Source feature for your order to cash process.

Accelerator: SAP Procure-to-Pay

Accelerates the SAP Table Batch Source feature for your procure-to-pay process.

SAP Ariba

Connects to an SAP Ariba source for batch data integrations into Google Cloud.

SAP BW Open Hub Batch Source

Connects to an SAP source for batch-based and delta-based extraction through BW Open Hub Service.

SAP OData

Connects to an SAP source for batch-based data extraction using the Open Data Protocol (OData).

SAP ODP

Connects to an SAP source for batch-based and delta-based data extraction through Operational Data Provisioning (ODP).

SAP SLT Replication

Connects to an SAP source to replicate data into Google Cloud using SAP Landscape Transformation (SLT).

SAP SuccessFactors Batch Source

Connects to an SAP source for batch-based data extraction.

SAP Table Batch Source

Connects to an SAP source for bulk data transfers from SAP tables and views.

SaaS

Plugins for common SaaS products like Salesforce, ServiceNow, Zendesk, and Zuora.

Anaplan

Source plugin for loading data from an Anaplan model into a data pipeline.

Anaplan

Sink plugin for supporting data import from CDAP to Anaplan server.

Anaplan Function

Performs an Action/Process that is predefined in the given Anaplan model. This is most commonly used to prepare for the data import and export.

HubSpot

Plugin to write data to HubSpot CRM in batch and streaming pipelines.

HubSpot

Reads data from HubSpot CRM in streaming pipelines.

HubSpot

Reads data from HubSpot CRM in batch pipelines.

Salesforce

This source reads sObjects such as opportunities, contacts, accounts, leads, any custom object, etc. from Salesforce in batch.

Salesforce

A batch sink that inserts sObjects such as opportunities, contacts, accounts, leads, any custom objects, etc. into Salesforce.

Salesforce Marketing Cloud

This source tracks updates in Salesforce sObjects such as opportunities, contacts, accounts, leads, any custom object, etc.

Salesforce Marketing Cloud

This sink inserts records into a Salesforce Marketing Cloud (SFMC) Data Extension via server-to-server integration with the SFMC API.

Salesforce Multi Objects

This source reads multiple sObjects from Salesforce. It should be used in conjunction with multi-sinks.

ServiceNow

Reads from one or multiple tables within ServiceNow.

Zendesk

Reads objects such as Article Comments, Post Comments, Requests Comments, Ticket Comments, Groups, Organizations, Satisfaction Ratings, Tags, Ticket Fields, Ticket Metrics, Ticket Metric Events, Tickets, Users from Zendesk.

Zendesk Multi Objects

Multi-source that reads multiple objects from Zendesk and routes them to specific destinations. Should be used in conjunction with a multi-sink.

Zuora

Plugin to write data to Zuora in batch and streaming pipelines.

Zuora

Reads data from Zuora in batch pipelines.

Search

Plugins for search.

Elasticsearch

Pulls documents from Elasticsearch according to the query specified by the user.

Elasticsearch

Writes data to Elasticsearch as a JSON string, then indexes it using the index, type, and idField specified by the user.

Google Search Console

Reads from Google Search Console in batch pipelines.

Snowflake

Plugins for Snowflake.

Cloud Storage to Snowflake

Loads data from external or internal staged files on Cloud Storage to an existing Snowflake table.

Snowflake

Writes data to a Snowflake table in batch pipelines.

Snowflake

Reads data from Snowflake tables in batch pipelines.

Snowflake Run SQL

Runs a specified SQL query in Snowflake.

Snowflake to Cloud Storage

Unloads data from a Snowflake table to an internal or external Cloud Storage location (stage).

Spark

Plugins for Spark.

Log Parser

Parses logs from any input source for relevant information such as URI, IP, browser, device, HTTP status code, and timestamp.

PySpark Program

Executes user-provided Spark code in Python.

Repartitioner

This plugin re-partitions a Spark RDD.

Scala Spark Compute

Executes user-provided Spark code in Scala that transforms RDD to RDD with full access to all Spark features.

Scala Spark Program

Executes user-provided Spark code in Scala.

Spark

Executes user-provided Spark code in Scala that operates on an input RDD or Dataframe with full access to all Spark features.

SSH

Plugins for SSH.

Remote Program Executor

Establishes an SSH connection with remote machine to execute command on that machine.

Run

Runs an executable binary which is installed and available on the local filesystem of the Hadoop nodes.

Statistics

Plugins for statistics.

Data Profiler

Calculates statistics such as total, nulls, min, max, mean, stddev, zero, positive count, and negative count.

Group By

Groups by one or more fields, then performs aggregate functions such as avg, count, count(*), first, last, max, min, sum, collect on each group.

Streaming

Plugins for streaming products.

Amazon SQS

Reads from Amazon Simple Queue Service (SQS) in streaming pipelines.

Amazon SQS

Plugin to write to Amazon Simple Queue Service (SQS) in streaming pipelines.

Confluent

Reads from Confluent Kafka in streaming pipelines.

Confluent

Plugin to write to Confluent Kafka in streaming pipelines.

File

File streaming source that watches a directory and streams file contents of any new files added to the directory.

Google Analytics Streaming

Reads from Google Analytics 360 in streaming pipelines.

HTTP Poller

This is a streaming source that will fetch data from a specified URL at a given interval and pass the results to the next plugin.

HTTP Streaming

Reads data in JSON, XML, CSV, TSV, TEXT and BLOB formats from HTTP/HTTPS APIs in streaming pipelines.

HubSpot

Reads data from HubSpot CRM in streaming pipelines.

Kafka Consumer

Kafka streaming source that reads events from Apache Kafka in streaming pipelines.

Kafka Alert Publisher

Kafka Alert Publisher that allows you to publish alerts to Kafka as JSON objects.

Kafka Producer

Kafka sink that allows you to write events into CSV or JSON to Kafka.

Kinesis Sink

Kinesis sink that outputs to a specified Amazon Kinesis Stream.

Kinesis Sourcef

Spark streaming source that reads from AWS Kinesis streams.

Pub/Sub

This sink writes to a Pub/Sub topic in streaming pipelines.

Pub/Sub

This sources reads from a Pub/Sub subscription in realtime.

Salesforce

This source tracks updates in Salesforce sObjects. Examples of sObjects are opportunities, contacts, accounts, leads, any custom object, etc.

Splunk

Reads from Splunk in streaming pipelines.

Splunk

Plugin to write to Splunk in batch and streaming pipelines.

Twitter Tweet Stream

Samples tweets in real time through Spark streaming. Output records will have this schema.

Windower

The Window plugin is used to window a part of a streaming pipeline.

Testing

Plugins for testing.

Trash

Trash consumes all the records on the input and eats them all, which means no output is generated or no output is stored anywhere.

XML

Plugins for XML.

XML Multi Parser

The XML Multi Parser transform uses XPath to extract fields from an XML document.

XML Parser

The XML Parser transform uses XPath to extract fields from a complex XML event.

XML Reader

The XML Reader plugin is a source plugin that allows users to read XML files stored on HDFS.

XML to JSON

Accepts a field that contains a properly formatted XML string and outputs a properly formatted JSON string version of the data.