Cloud Data Fusion release notes

The Cloud Data Fusion version 6.10.1.2 patch revision is generally available (GA). 6.10.1.2 includes the following changes:

• You can generate audit logs that record data plane activities within your Cloud Data Fusion instance. Data plane audit logging is available in Preview for RBAC-enabled instances.

• To improve the API response time, by default, all program activity records older than 30 days are cleaned up. Any activity older than 30 days isn't visible in the Cloud Data Fusion studio (CDAP-14950).

• When using role-based access control, performing the List Pipelines operation requires datafusion.pipelines.list permission, in addition to datafusion.namespaces.get permission. For more information, see RBAC roles and permissions (CDAP-20931).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to overcount in cases where servers were restarted when a pipeline run was in progress (CDAP-21046).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to be stale after a restart when no pipelines were running (CDAP-21048).

• Fixed an issue causing the default max concurrent runs limit for triggers not to appear in the web interface, making it difficult to tell if triggers were working as intended (CDAP-21072).

• Fixed an issue causing the top panel of the Studio tab to disappear when you edited a pipeline draft that's based on a pipeline from an earlier Cloud Data Fusion version (CDAP-21073).

• Improved performance by removing a call to the list apps API during pipeline deployment when checking if a pipeline already exists (CDAP-21074).

The Snowflake plugin version 1.1.4 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1816):

• Fixed an issue in the Snowflake source causing pipelines to fail if fields contained decimals.
• Fixed an issue in the Snowflake source causing pipelines to fail if data contained a backslash (\). You can set a new escape character using the cdap.snowflake.source.escape runtime argument.

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.10.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017).

The HTTP plugin version 1.4.3 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1810):

• In the HTTP streaming source, batch source, and batch sink, a PATCH option was added to the HTTP Method field.
• Fixed an issue in the HTTP sink causing data loss when a pipeline didn't fail by default when a non-2xx response code was received.
• Fixed an issue in the HTTP source causing a pipeline not to fail when the get_schema method was called and a non-2xx response code was received.

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.8.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017). This note is incorrect; see entry for November 27, 2024.

The SAP table batch source plugin version 0.11.5 is available in Cloud Data Fusion version 6.8.0 and later. This release fixes an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage.

The Multiple table plugin version 1.4.1 is available in Cloud Data Fusion versions 6.10.1 and later. This release fixes an issue causing pipelines to fail if a Multiple database tables batch source's Reference Name field contains spaces. The field no longer accepts spaces (PLUGIN-1752).

The Oracle plugin version 1.11.4 is available in Cloud Data Fusion versions 6.10.1 and later. This release includes the following change:

• Fixed an issue causing pipelines with an Oracle sink that has date columns in the input schema to fail (PLUGIN-1812).

The SAP ODATA batch source plugin version 0.11.5 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes:

• Fixed an issue causing pipelines to fail when the source plugin contained nested fields (an array of records): ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):o.a.s.u.Utils@98] - Aborting task io.cdap.cdap.api.data.format.UnexpectedFormatException: Schema must be a record with at least one field.

• Added a Read Timeout plugin property, which lets you configure the read timeout value for long-running pipelines.

The SAP ODP batch source plugin version 0.11.3 is available in Cloud Data Fusion versions 6.8.0 and later. This release includes the following changes:

• Fixed an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage. To address the issue, you must upgrade the Cloud Storage client library to version 2.3.0 or later.

• Fixed an issue causing memory errors in the SAP system. You can choose to load changed data without loading historical data first. You can select this option in the plugin properties.

The Cloud Data Fusion version 6.10.1.1 patch revision is generally available (GA). 6.10.1.1 includes the following changes:

• Cloud Data Fusion stores lineage-related information for 30 days by default, in addition to cleaning up run records (CDAP-21053).
• Added support to disable Field level lineage. For more information, see Explore Data Lineage using metadata (CDAP-21007).
• Cloud Data Fusion supports running pipelines using Dataproc 2.2. For more information about its limitations in Cloud Data Fusion, which are unchanged from Dataproc 2.1, see Change the Dataproc image to version 2.1 (CDAP-21047). This note is incorrect; see entry for October 24, 2024.

The Cloud Data Fusion version 6.9.2.4 patch revision is GA. 6.9.2.4 includes the following changes:

• Cloud Data Fusion stores lineage-related information for 30 days by default, in addition to cleaning up run records (CDAP-21053).
• Added support to disable Field level lineage. For more information, see Explore Data Lineage using metadata (CDAP-21007).
• Fixed an issue causing the maximum concurrent runs setting not to work as expected for scheduled pipeline runs (CDAP-20988).
• Fixed an issue causing upgrades to fail when the schedule name had hyphens, spaces, or other symbols (CDAP-20999).

The SAP SLT No RFC Replication plugin version 0.11.3 is available in Cloud Data Fusion version 6.8.0 and later. This release fixes an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage.

The CloudSQL MySQL plugin version 1.10.7 is available in Cloud Data Fusion versions 6.9.0 and 6.10.0. This plugin version lets you use a macro to specify the name of the CloudSQL instance in the plugin's Connection name field.

Excel plugin version 2.12.3 is available in Cloud Data Fusion 6.10.0 and later. This version fixes an issue in the Excel batch source causing pipelines with large XLSX files to consume high memory and fail (PLUGIN-1771 and PLUGIN-1795).

Excel plugin version 2.11.5 is available in Cloud Data Fusion 6.9 versions. This version fixes an issue in the Excel batch source causing pipelines with large XLSX files to consume high memory and fail (PLUGIN-1771 and PLUGIN-1795).

Excel plugin version 2.10.3 is available in Cloud Data Fusion 6.8 versions. This version fixes an issue in the Excel batch source causing pipelines with large XLSX files to consume high memory and fail (PLUGIN-1771 and PLUGIN-1795).

The Python transform plugin version 2.3.1 is available in Cloud Data Fusion version 6.10.1. The release lets you use a macro in the Script plugin property field to pass runtime arguments (CDAP-21054).

The Cloud Storage Copy/Move plugin version 0.23.2, which is bundled with Google Cloud Platform plugin, is available in Cloud Data Fusion versions 6.10.0 and later. The release lets you use a wildcard character (*) in the source path to copy and move multiple files. For example, the source path gs://demo0/prod/reports/*.csv copies and moves all CSV files in the reports directory (PLUGIN-698).

The Cloud Storage Multi File sink plugin version 0.23.2 is available in Cloud Data Fusion version 6.10.1 and later. The release fixes an issue in the Cloud Storage Multi File sink causing pipelines to fail when the Flexible schema property was set to true (PLUGIN-1780).

The Oracle sink plugin version 1.10.7 is available in Cloud Data Fusion version 6.9. The release fixes an issue in the Oracle sink causing null values to be assigned to fields in the input schema that have lowercase letters in the field name (PLUGIN-1793).

The Google Sheets plugin version 1.4.3, which is bundled with the Google Drive plugins, is available in the Cloud Data Fusion Hub. The release includes the following changes:

• Fixed an issue causing the Google Sheets plugin to incorrectly parse column names that have special characters (PLUGIN-1785).

• Fixed an issue causing pipelines to fail when the Google Sheets plugin is used with Wrangler and any of the fields required to fetch schema was a macro (PLUGIN-1791).

The Decompress plugin version 1.2.1 is available in the Hub in Cloud Data Fusion version 6.10.1 and later. The release fixes an issue in the Decompress plugin causing concatenated GZIP files (.gz) to not decompress as intended. In version 1.2.1, decompression occurs until EOF is reached (PLUGIN-1743).

The Cloud Storage Multi File sink plugin version 0.22.8 is available in Cloud Data Fusion version 6.9.2. The release fixes an issue in the Cloud Storage Multi File sink causing pipelines to fail when a Flexible schema was set to true (PLUGIN-1780).

Fixed in Cloud Data Fusion 6.10.1:

• Fixed an issue causing runtime arguments of pipeline triggers to not propagate to downstream pipelines (CDAP-20947).
• Fixed an issue in Wrangler causing the send-to-error-and-continue directive to not initialize dq_failure when the condition is false (PLUGIN-1736).
• Fixed an issue that occurs if running a replication pipeline when task workers are enabled (CDAP-20951).
• Improved error reporting in the BigQuery Sink. Fixed an issue in BigQuery Argument Setter where validation error wasn't displayed correctly (PLUGIN-788, PLUGIN-781, PLUGIN-782, PLUGIN-1318).
• Improved retries in BigQuery plugin (PLUGIN-1715).
• Fixed an issue with the Python plugin, where running in native mode doesn't work as intended (PLUGIN-1617).
• Fixed an issue causing certain connection parameters to not propagate in a MySQL connection (PLUGIN-1728).
• Fixed an issue causing the Cloud Storage Copy action to timeout while working with large files (PLUGIN-1735).
• Fixed an issue causing Copy and Move plugins to not create buckets at the destination path as expected, resulting in a runtime error (PLUGIN-1738).
• Fixed an issue causing empty source input to fail in multiple plugins (PLUGIN-1742).
• Fixed an issue with remote execution of Wrangler directives causing type information to not be emitted (PLUGIN-1778).
• Fixed an issue causing a No record field providederror (CDAP-21024).
• Streaming pipelines in Cloud Data Fusion support the Excel source. Batch pipelines with an Excel source can consume high memory and fail in large pipelines (PLUGIN-1771).
• Fixed an issue with using the Conditional plugin as a source for Wrangler, causing CDAP not to fetch the necessary schema (CDAP-20890).
• Fixed an issue with instance upgrades causing existing schedule names to be improperly encoded in the URL, resulting in pre-upgrade failure (CDAP-20999).
• Fixed an issue with schedules causing the maximum concurrent run property to not work as intended (CDAP-20988).
• Fixed an issue causing committed ID to incorrectly propagate when pushing pipeline configurations to Git (CDAP-20932).

The SAP SuccessFactors batch source version 1.2.3 is available in the Enterprise edition of Cloud Data Fusion 6.7.0 and later. This release lets you configure a proxy URL and SuccessFactors authentication properties.

The HTTP plugin (version 1.4.2) is available in Cloud Data Fusion versions 6.8.0 and later. The release fixed an issue in the HTTP source causing an error in the retrieved schema when one of the retrieved columns contained a quoted value with a delimiter, such as a comma (PLUGIN-1781).

The Salesforce Marketing Cloud plugin (version 1.3.1) is available in Cloud Data Fusion version 6.8.0 and later. The release fixed an issue in the Salesforce Marketing sink plugin causing upsert operations to fail (PLUGIN-1773).

The SAP Ariba batch source plugin version 1.3.0 is available in the Cloud Data Fusion Enterprise edition, versions 6.7.0 and later. In plugin version 1.3.0, if a failure occurs, the plugin retries API calls. You can configure the retry parameters using the hidden fields in the pipeline configuration JSON.

Salesforce plugin version 1.6.3 is available in Cloud Data Fusion versions 6.8.0 and later. The release includes the following changes:

• Fixed an issue in the Salesforce sink plugin causing an unsupported type datetime error for DateTime type fields in the input schema. In this version, the Salesforce sink plugin supports datetime and decimal logical types (PLUGIN-1749).
• Fixed an issue in all Salesforce plugins causing a pipeline to fail when using an OAuth macro because the oAuth macro value didn't get passed to the plugin as intended. In this version, all Salesforce plugins support an OAuth macro (PLUGIN-1767).
• At the time of failure on the Salesforce sink side, if the Error handling property is set to the Fail on error option, the Salesforce job is aborted, which stops newer batches from being added to the job due to spark retry settings in CDAP (PLUGIN-1768).
• To make debugging easier, additional debug logs and batch results in logs are available.

The Cloud Data Fusion version 6.9.2.3 patch revision is generally available (GA). 6.9.2.3 includes the following fixes:

• Skipped running MetadataConsumerSubscriberService when Dataplex Data Lineage Integration is disabled (CDAP-20947).

• Fixed an issue causing runtime arguments of pipeline triggers to not propagate to downstream pipelines (CDAP-20943).

• Fixed an issue causing pipelines to fail in starting state when the system worker service is intermittently unavailable (CDAP-20956).

• Fixed an issue causing pipelines to fail in starting state when the Compute Engine metadata server is intermittently unavailable (CDAP-20955).

Fixed in Cloud Data Fusion 6.10.0:

• Fixed an issue in the Postgres DB plugin causing macros to be unsupported for database configuration (PLUGIN-1681).
• Fixed an issue causing slowness in the API while fetching runs for all applications in a namespace (CDAP-20587).
• Made the following fixes to Wrangler grammar (CDAP-20839):
  • The NUMERIC token type supports negative numbers.
  • The PROPERTIES token type supports one or more properties.
• Fixed an issue causing columns that have all null values to be dropped in Wrangler (CDAP-20521).
• Fixed an issue causing pipeline upgrades to not have the intended description (CDAP-20815).

The SAP BW Open Hub batch source plugin version 0.11.4 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes:

• Fixed an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage. To address the issue, you must upgrade the Google Storage Client library to version 2.3.0 or later.

• Add a new plugin property, Max. retries for read enqueue.

The Salesforce plugin version 1.6.2 is available in Cloud Data Fusion versions 6.8.0 and later. This version includes the following changes:

• Fixed an issue in the Salesforce plugin causing the following error in some pipelines that run more than 4 hours: java.lang.IllegalStateException: SSLException reading next record: javax.net.ssl.SSLException: Connection reset. The Connection timeout property was added to the Salesforce plugin properties in the web interface with the default value of 3600 seconds (PLUGIN-1719).

• For accuracy, fixed schema handling for referenced object fields: child fields are explicitly marked as non-nullable, regardless of the schema values in the referenced object.

In earlier versions, when retrieving schema information for fields in referenced objects, such as contact.account_lastmodifieddate, the schema inherits properties from the referenced object, causing incorrect non-nullable assumptions (PLUGIN-1720).

• A retry mechanism was added in the Salesforce batch source and Multi-Source plugins for connection timeout issues (PLUGIN-1706).

The SAP ODP batch source plugin version 0.11.0 is available in Cloud Data Fusion versions 6.8.3 and later. The release includes the following changes:

• Improved error logging, providing more information about the occurrence and mode of errors.
• Improved the placeholder values in the Filter options fields.
• Added support for fetching Operational Delta Queue (ODQ) fields by turning on the Fetch delta fields toggle.
• On the Properties page for the plugin, changed the name of the ObjectType field to Context. For more information, see the ODP batch source properties.

The Cloud Data Fusion version 6.9.2.2 patch revision is generally available (GA). 6.9.2.2 includes the following fixes:

• Increased the speed of the batch /runs API call for pipelines that are run thousands of times (CDAP-20587).
• Fixed an issue causing draft pipelines to load incorrectly when you enable an accelerator that automatically installs plugins from a custom hub (CDAP-20628).
• Fixed an issue causing the CDAP service IP to be cached forever (CDAP-20781).
• Fixed an issue causing a KubeTwillRunnerService error on shutdown (CDAP-20792).
• Fixed an issue causing pipelines and streaming jobs to fail after an AppFabric restart. (CDAP-20797).
• Fixed an issue causing a slowdown in deploying applications (CDAP-20820).
• Fixed an issue causing refreshed OAuth tokens to not be logged as expected, making it difficult to identify the root cause of some issues (CDAP-20861).
• Fixed an issue in replication where you couldn't select the Tink transformation in the web interface. (CDAP-20804).

The Cloud Data Fusion version 6.8.3.1 patch revision is generally available (GA). It fixes a regression that causes a pipeline to fail when using Dataproc secondary workers (CDAP-20807).

The Cloud Data Fusion version 6.9.2.1 patch revision is generally available (GA). It fixes a regression that causes a pipeline to fail when using Dataproc secondary workers (CDAP-20807).

The Cloud Data Fusion version 6.7.3.1 patch revision is generally available (GA). It introduces retries in the platform transaction layer to handle a PSQLException error thrown from broken database connections (CDAP-19949 and CDAP-20722).

Fixed in Cloud Data Fusion 6.9.2:

• Added a retry for Pub/Sub snapshot creation and deletion in a real-time pipeline with a Pub/Sub source when a retryable internal error is thrown (PLUGIN-1660).
• Fixed a bug causing the Dynamic Spark plugins to fail when running on Dataproc 1.5 (CDAP-20674).
• Fixed a discrepancy in warning and error counts reported between the pipeline summary tab and system logs (CDAP-20680).
• Fixed a problem when, in rare cases, a cluster couldn't be found with Cluster Reuse (CDAP-20759).
• Fixed a bug causing the JavaScript transform to fail on Dataproc 2.1 (CDAP-20778).

The SAP SuccessFactors Batch Source plugin version 1.2.1 is available in Cloud Data Fusion. The release includes the following changes:

• Fixed an issue causing an inaccurate error message when the plugin's Associated Entity Name property is invalid.

• Fixed an issue causing inaccurate validation errors when the Expand Fields property contains a macro.

In the SAP Table Batch Source plugin version 0.10.0, fixed an issue causing failed data pipeline runs when you clicked the Take a snapshot toggle because the FIELDS parameters weren't exported.

In the SAP SuccessFactors Batch Source plugin version 1.2.0, fixed an issue causing a null pointer exception when you provide a valid entity name that has more than one expand option separated by a '/'.

In Cloud Data Fusion versions 6.7 and later, SAP Ariba plugin version 1.2.1 fixes the issue causing the following error after entering authentication credentials: CDF_ARIBA_01501 - Failed to call given Ariba service. This issue occurs when the SAP Ariba sandbox exists in the following locations:

• US
• EU
• UAE
• KSA

The SAP OData plugin (version 0.9.1) is available in the Cloud Data Fusion SAP Hub (all versions) with the following changes:

• Fixed an issue in the SAP OData batch source causing you not to receive a valid error message if the base URL provided is invalid.
• A warning has been added to the log message when you provide a batch size that is larger than the maximum allowed batch size.

Fixed in Cloud Data Fusion 6.9.1:

• For SQL Server replication sources, fixed an issue on the Review assessment page, where SQL Server DATETIME and DATETIME2 columns were shown as mapped to TIMESTAMP columns in BigQuery. This was a UI bug. The replication job mapped the data types to the BigQuery DATETIME type (CDAP-19389).

• For replication jobs, fixed an issue where retries for transient errors from BigQuery might have resulted in data inconsistency (CDAP-20276).

• Fixed an issue where a replication job got stuck in an infinite retry when it failed to process a DDL operation (CDAP-20301).

• When you duplicate a pipeline, Cloud Data Fusion appends _copy to the pipeline name when it opens in the Pipeline Studio. In previous releases, Cloud Data Fusion appended _<v1, v2, v3> to the name (CDAP-20373).

• Fixed the pipeline stage validation API to return unevaluated macro values to prevent secure macros from being returned (CDAP-20430).

• Fixed an issue that sometimes caused pipelines to fail when running pipelines on Dataproc with the following error: Unsupported program type: Spark. The first time a pipeline that only contained actions ran on a newly created or upgraded instance, it succeeded. However, the next pipeline runs, which included sources or sinks, might have failed with this error (CDAP-20431).

• Fixed an issue where the flow control running count metric (system.flowcontrol.running.count) might be stale if no new pipelines or replication jobs were started (CDAP-20458).

• Fixed an issue where executor resource settings are not honored when app.pipeline.overwriteConfig is set (CDAP-20549).

• In the Oracle batch source, the Oracle NUMBER data type defined without precision and scale by default was mapped to Cloud Data Fusion string data type. If these fields were used by an Oracle Sink to insert into a NUMBER data type field in the Oracle table, the pipeline failed due to incompatibility between string and NUMBER type. Now, the Oracle Sink inserts these string types into NUMBER fields in the Oracle table (PLUGIN-1481).

• For Oracle batch sources, fixed an issue that caused the pipeline to fail when there was a TIMESTAMP WITH LOCAL TIME ZONE column set to NULLABLE and the source had values that were NULL(PLUGIN-1494) .

• Fixed an issue where pipelines that had a Database batch source and an Oracle sink that used a connection object (using SYSDBA) to connect to an Oracle database failed to establish a connection to the Oracle database. This was due to a package conflict between the Database batch source and the Oracle sink plugins (PLUGIN-1503).

• Fixed an issue where pipelines failed when the output schema was overridden in certain source plugins. This was because the output schema didn't match the order of the fields from the query. This happened when the pipeline included any of the following batch sources:

• Database

• Oracle

• MySQL

• SQL Server

• PostgreSQL

• DB2

• MariaDB

• Netezza

• Cloud SQL PostgreSQL

• Cloud SQL MySQL

• Teradata

  Pipelines no longer fail when you override the output schema in these source plugins. Cloud Data Fusion uses the name of the field to match the schema of the field in the result set and the field in the output schema {PLUGIN-1512).

• For BigQuery Pushdown, fixed an issue when BigQuery Pushdown was enabled for an existing dataset, the Location where the BigQuery Sink executed jobs was the location specified in the Pushdown configuration, not the BigQuery Dataset location. The configured Location should have only been used when creating resources. Now, if the dataset already exists, the Location for the existing dataset is used (PLUGIN-1513).

• For the Database sink, fixed an issue where the pipeline didn't fail if there was an error writing data to the database. Now, if there is an error writing data to the database, the pipeline fails and no data is written to the database (PLUGIN-1514).

• Fixed an issue that checks GET permission on a namespace which does not exist yet during the namespace creation flow (CDAP-18394).

• Fixed an issue where Dataproc continued running a job when it couldn't communicate with the Cloud Data Fusion instance if the replication job or pipeline was deleted in Cloud Data Fusion (CDAP-20216).

• Fixed an issue that caused pipelines with triggers with runtime arguments to fail after the instance was upgraded to Cloud Data Fusion 6.8.0 and 6.9.0 (CDAP-20568).

• Fixed an issue where arguments set by actions and pipeline triggers don't overwrite runtime arguments. You must add the following runtime argument: system.skip.normal.macro.evaluation=true (CDAP-20597).

• Fixed an issue that caused the Pipeline Studio page to show an incorrect count of triggers (CDAP-20655).

• Fixed an issue that caused the Trigger's Payload Config to be missing in the UI for an upgraded instance (CDAP-20660).

• Fixed bug where initial offset was not considered in Kafka batch source (PLUGIN-1594).

Fixed in 6.8.3:

• Fixed an issue where the event publish feature did not work with RBAC-enabled instances (CDAP-20375).
• Fixed an issue where executor resource settings were not honored when app.pipeline.overwriteConfig was set (CDAP-20549).
• Fixed an issue that caused pipelines with triggers with runtime arguments to fail after the instance was upgraded to Cloud Data Fusion 6.8.0 and later (CDAP-20568).
• Fixed an issue where arguments set by actions and pipeline triggers don't overwrite runtime arguments. You must add the following runtime argument: system.skip.normal.macro.evaluation=true (CDAP-20597).
• Fixed an issue that caused the Studio page to show an incorrect count of triggers (CDAP-20655).
• Fixed an issue that caused the Trigger's Payload Config to be missing in the UI for an upgraded instance (CDAP-20660).
• Fixed an issue in the BigQuery Sink where the absence of an ordering key caused an exception (PLUGIN-1582).
• Fixed an issue where initial offset was not considered in the Kafka Batch Source (PLUGIN-1594).

Cloud Data Fusion version 6.8.2 fixes an issue in Cloud Data Fusion versions 6.8.0 and 6.8.1 that may cause the following error: Unsupported program type: Spark. The first time a pipeline that only contains actions runs on a newly created or upgraded instance, it succeeds. However, following pipeline runs that also include sources or sinks may fail with this error. For updated settings, see Troubleshooting.

In Cloud Data Fusion version 6.8.1, Dataproc clusters no longer require the following OAUTH scope to function: https://www.googleapis.com/auth/cloud-platform.

Oracle plugins version 1.8.6 is available in Cloud Data Fusion versions 6.7.1, 6.7.2, and 6.7.3. Oracle plugins version 1.9.2 is available in Cloud Data Fusion versions 6.8.0 and later. These plugin versions have the following changes:

For the Oracle Batch Source, fixed a backward compatibility issue. In plugin versions 1.8.3, 1.9.0, and earlier, Cloud Data Fusion maps the Oracle NUMBER data type with undefined precision and scale to CDAP decimal(38,0) , which can cause data loss due to rounding errors. In plugin versions 1.8.4, 1.8.5, and 1.9.1, the Oracle NUMBER data type with undefined precision and scale maps to the CDAP string data type by default, which preserves all decimal digits. In versions 1.8.6 and 1.9.2, the Oracle NUMBER data type with undefined precision and scale gets mapped to CDAP string by default and lets you edit the output schema to use the older mapping to decimal(38, 0) data type. For more information, see Oracle batch source plugin converts NUMBER to string (PLUGIN-1535 ).

Fixed in 6.8.1:

• Fixed an issue in instances with role-based access control where deployment and validations failed with read time out error after a short time.

• Fixed an issue that caused replication assessment for Oracle by Datastream to fail when an instance had role-based access control.

• Fixed an issue that allowed reading secure keys in the system namespace with only the Data Fusion Viewer role (datafusion.viewer) or Instance Accessor role (datafusion.accessor). For more information about predefined roles for role-based access control in Cloud Data Fusion, see the Role-based access control (RBAC) overview.

• Fixed an issue where the Replication Source plugin's event reader was not stopped by the Delta worker when there were errors, which caused leakage of the plugin's resources (CDAP-20394).

• Fixed an issue that occurred in certain upgrade scenarios, where pipelines didn't have the Use Connection property set and the UI didn't display a plugin's connection properties, such as Project ID and Service Account Type (CDAP-20392).

• Fixed an issue that caused pipelines to fail if they used a connection property, such as the Service Account JSON property, that used a secure macro with JSON as the value (CDAP-20271).

• For Oracle by Datastream Replication sources, fixed an issue where the Review Assessment page would freeze when a selected or manually-entered table didn't exist in the source database (CDAP-20257).

• For Oracle by Datastream Replication sources, fixed an issue where the Select tables and transformations page failed to load the list of tables and displayed the error deadline exceeded when the source database contained a large number of tables (CDAP-20199).

• Fixed an error in security-enabled instances that caused pipeline launch to fail and return a token expired error when evaluating secure macros in provisioner properties (CDAP-20146).

• For MySQL Replication sources, fixed an issue that caused Replication jobs to fail during the initial snapshot, if the job included a runtime argument with the Debezium property, binary-handling-mode (CDAP-20121).

• For Replication jobs, increased retry duration for API calls to update state/offsets in Replication jobs (CDAP-20028).

• Fixed an issue that prevented upgrades for MySQL and SQL Server Replication jobs in version 6.6.0. Upgrades are supported from version 6.6.0 to 6.7.3 and 6.8.1 (CDAP-19622).

• Fixed upgrades for Oracle by Datastream Replication jobs. You can upgrade Oracle by Datastream Replication jobs from Cloud Data Fusion versions 6.6.0 and 6.7 to version 6.8.1. (CDAP-20013).

Fixed in 6.7.3:

• Fixed an issue that allowed reading secure keys in the system namespace with only the Data Fusion Viewer role (datafusion.viewer) or Instance Accessor role (datafusion.accessor). For more information about predefined roles for role-based access control in Cloud Data Fusion, see the Role-based access control (RBAC) overview.

• Fixed an issue in the BigQuery Replication Target plugin that caused Replication jobs to fail when the BigQuery target table already existed. The new version of the plugin will automatically be used in new Replication jobs (CDAP-19599).

• Fixed an issue that prevented upgrades for MySQL and SQL Server Replication jobs in version 6.6.0. Upgrades are supported from version 6.6.0 to 6.7.3 and 6.8.1 (CDAP-19622).

• Fixed an issue that prevented upgrades for Oracle by Datastream Replication jobs in version 6.6.0. Upgrades are supported from versions 6.6.0 , 6.7.0, 6.7.1, and 6.7.2 to version 6.7.3 (CDAP-20013).

• Fixed an issue that caused pipelines to fail if they used a connection property, such as the Service Account JSON property, which used a secure macro with JSON as the value (CDAP-20271).

• Fixed an issue that occurred in certain upgrade scenarios, where pipelines didn't have the Use Connection property set and the UI didn't display a plugin's connection properties, such as Project ID and Service Account Type (CDAP-20392).

• Fixed an issue where the Replication Source plugin's event reader was not stopped by the Delta worker when there were errors, which caused leakage of the plugin's resources (CDAP-20394).

• Fixed an error in security-enabled instances that caused pipeline launch to fail and return a token expired error when evaluating secure macros in provisioner properties (CDAP-20146).

• In the Oracle Batch Source (version 1.8.4 and earlier), when the source data included fields with the NUMBER data type with undefined precision and scale, Cloud Data Fusion read it as decimal and set the precision to 38 and the scale to 0. If any values in the field had scale other than 0, values were rounded, which could have resulted in data loss. If the scale for this field is overridden in the plugin output schema, the pipeline fails.
  
  In Oracle batch source version 1.8.5, you can edit the scale of the CDAP decimal data type in the output schema. The overridden value is used to map to the Oracle NUMBER data type without failing the pipeline. If there are any numbers present in the fields with a scale greater than the scale defined in the plugin, Cloud Data Fusion rounds the values based on the scale you set in the output schema. For example, if you specify precision=10, scale=3, the value 123.4567 is rounded to 123.457. For more information about setting precision and scale in a plugin, see Changing the precision and scale for decimal fields in the output schema (PLUGIN-1433).

• Improved performance for batch pipelines with MySQL sinks (PLUGIN-1374).

• For Database plugins (version 2.9.3), fixed a security issue where the database username and password were exposed in the logs (CDAP-20235).

Fixed in 6.8.0:

• For custom Dataproc compute profiles, fixed the issue causing the wrong Cloud Storage bucket to be used to stage data. Cloud Data Fusion uses the bucket specified in the custom compute profile.

• Fixed the issue in the BigQuery Replication Target plugin causing Replication jobs to fail when the BigQuery target table already existed. The new version of the plugin is used by default in new Replication jobs. To use the new plugin version in existing jobs, recreate the job.

• Fixed an issue causing the Replication Assessment to get stuck when the Oracle (by Datastream) storage bucket property was empty, or had an invalid bucket name. Cloud Data Fusion returns a 400 error code during assessment when the property is empty or has an invalid bucket name.

• Fixed an issue causing Replication jobs to fail when the source column name didn't comply with BigQuery table naming conventions. In 6.8.0, if a source column name doesn't comply with BigQuery naming conventions, Cloud Data Fusion replaces invalid characters with an underscore, prepends an underscore when the first character is a number, and truncates the name when it exceeds the maximum length.

• In the File batch source, fixed an issue causing Get Schema to appear only when Format was set to delimited. Get Schema appears for all formats.

• Fixed an issue with the output schema when connecting a Splitter transformation with a Joiner transformation.

• Fixed an issue causing imports in the Cloud Data Fusion UI to fail for pipelines exported through the Pipeline Microservices.

• In the Oracle Batch Source, when the source data included fields with the Numeric data type (undefined precision and scale), Cloud Data Fusion set the precision to 38 and the scale to 0. If any values in the field had scale other than 0, Cloud Data Fusion truncated the values, which could have resulted in data loss. If the scale for a field was overridden in the plugin output schema, the pipeline failed. For more information, see the CDAP 6.8.0 bug fixes.

• In the Wrangler transformation, fixed the issue causing the pipeline to not fail when the Error Handling property was set to Fail Pipeline. This happened when an error was returned, but no exception was thrown, and there were 0 records in the output. For example, this happened when one of the directives (such as.parse-as-simple-date) failed because the input data wasn't in the correct format. This fix is not available by default. Instead, contact support if you need this fix for your projects.

• In Wrangler, fixed the issue causing the Wrangler page to get stuck when a BigQuery table name contained non-alphanumeric characters, such as underscores. Wrangler imports BigQuery tables that follow BigQuery table naming conventions.

Fixed in 6.7.2:

• In the BigQuery Sink plugin (version 0.20.3), fixed an issue that caused a NullPointerException error when table metrics were updated or when the output schema was not defined.
• In the Send Email batch pipeline alert, fixed an issue where emails failed to send when the Protocol was set to TLS.
• (Release note added January 5, 2023) In the Cloud Storage Sink and File Sink plugins, fixed an issue that caused Decimal values to be formatted incorrectly.

Fixed in 6.7.1:

• Fixed an issue that created duplicate entries in the file cache map, which resulted in multiple attempts to delete the same cache file.

• Fixed an issue where the Log service left empty folders, which made the mounting of Persistent Disk slow. This caused the Log service not to start in a timely manner.

• Fixed an issue that caused pipelines to take a long time to launch or get stuck. This was linked to I/O throttling that occurred on the underlying Persistent Disk.

• Fixed an issue that caused pipelines to fail when two or more pipelines were scheduled to start simultaneously on a static Dataproc cluster. This was due to a file upload race condition.

• Fixed an issue in flow control that caused Appfabric to return a 5xx error code in rare scenarios, instead of 429 (Too Many Requests Error), if the number of concurrently launching or running pipelines was above certain thresholds.

• Fixed an issue that resulted in an error when a compute profile was exported from the default namespace after switching from a custom namespace.

• Delete, edit, and export no longer appear as operations for pre-configured connections in Cloud Data Fusion. You can view connection details for those connections.

• Fixed an issue when you started a pipeline multiple times, and then stopped the pipeline before it completed, which resulted in the following UI error: Program <program name> is not running.

• Removed verbose logs from the BigQuery client libraries in pipeline logs.

• Fixed an issue that caused the BigQuery Execute action plugin configured with an Encryption Key Name (CMEK) to fail when the SQL query contained DDL Statements.

• In the BigQuery Execute action plugin, added the property Store Results in a BigQuery Table to the UI, which hides the destination table related properties by default.

Fixed in 6.7.0:

• Fixed an issue in the Group By transformation where Longest String and Shortest String aggregators returned an empty string, even when all records contained null values in the specified field. The Group By transformation returns null for empty input.
• Fixed an issue in the Group By transformation that caused the Concat and Concat Distinct aggregate functions to produce incorrect results in some cases.
• Fixed an issue in the Group By transformation that caused the Variance, Variance If, and Standard Deviation aggregate functions to produce incorrect results in some cases.
• In the Oracle and MySQL Batch Source plugins, fixed an issue to treat all timestamps, specifically the ones older than the Gregorian cutover date (October 15, 1582), from the database in Gregorian calendar format.
• Improved the generic Database source plugin to correctly read data when the data type is NUMBER, scale is set, and the data contains integer values.
• Fixed an issue in sources (such as File and Cloud Storage) that resulted in an error if you clicked Get Schema when the source file contained delimiters used in regular expressions, such as "|" or ".". You no longer need to escape delimiters for sources.
• Fixed an issue where Datastore sources read a maximum of 300 records. Datastore sources read all records.
• Fixed an issue in BigQuery sinks where the output table was not partitioned correctly in the following cases:
  • The output table didn't exist.
  • Partitioning type was set to Time.
  • Operation was set to Upsert.
• Fixed an issue that caused pipelines with BigQuery sinks that have input schemas with nested array fields to fail.
• Fixed issues that caused failures when reading maps and named enums from Avro files.

Fixed in 6.6.0:

• Improved instance stability. Fixed a number of system service unavailability cases resulting in the message "Necessary services are experiencing intermittent problems" and API call failures.
• Fixed an issue that caused pipelines to fail when a Database Batch Source included a decimal column with precision greater than 19.
• Fixed an issue that caused pipelines with a Conditional plugin that were running on MapReduce to fail.
• Fixed an issue that caused pipelines with a Conditional plugin and running on Spark to fail.
• Fixed an issue that caused validation to fail for Cloud Storage Multi File Sinks.

Fixed in 6.5.1 (for more information, see the CDAP release note ):

• Fixed an issue where messages could not be retrieved for Kafka topics.
• Fixed an issue where you could not create a profile for an existing Dataproc cluster.
• Fixed an issue that caused pipelines to fail when Transformation Pushdown was enabled and used macros as properties.
• Fixed an issue that caused long running programs, like Replication, to fail within the default Hadoop delegation token timeout. Now, these tokens get renewed so that the job keeps running.
• Fixed an issue in Replication that caused an error when you clicked Configure.
• Fixed an issue that caused a pipeline to fail with an ACCESS DENIED error when running BigQuery jobs. For more information, see Troubleshooting .
• Fixed an issue in the Cloud Storage connection that prevented browsing and parsing files stored in folders under buckets.
• Fixed an issue that caused custom formats to be unusable in the Cloud Storage source and sink.

Fixed in 6.5.0 preview version (for more information, see the CDAP release note):

• Fixed an issue in Replication that caused jobs to fail if more than 1000 tables were selected for replication.

• Fixed an issue that caused replication jobs to hang when there were too many Delete or DDL events.

• Fixed an issue that caused Wrangler to ignore all the other columns other than the given column when parsing Excel files.

• Fixed Wrangler to fail pipelines upon error. In Wrangler 6.2 and above, there was a backwards-incompatible change where pipelines did not fail if there was an error and instead were marked as completed.

• Improved resilience of TMS.

• Fixed an issue that caused File Source Plugin validation to fail when there was a macro in the Format field.

Fixed in 6.4.1 (for more information, see the CDAP release note):

• Fixed an issue that caused pipelines with aggregations and Decimal fields to fail with an exception.

• Fixed the Join Condition Type so that it is displayed in the Joiner plugin for pipelines that were upgraded from versions before 6.4.0.

• Fixed Wrangler so that pipelines fail when there is an error. In Wrangler 6.2 and above, there was a backwards-incompatible change where pipelines did not fail if there was an error and were instead marked as complete.

• Fixed an issue that prevented new previews from being scheduled after the preview manager had been stopped ten times.

• Fixed an issue while writing non-null values to a nullable field in BigQuery.

• Fixed an issue in the BigQuery plugins to correctly delete temporary storage buckets.

• Fixed an issue in the BigQuery sink that caused pipelines to fail when the input schema was not provided.

• Fixed an issue in the BigQuery sink that caused pipelines to fail or give incorrect results.

• Fixed an issue that caused pipelines to fail when a Pub/Sub source Subscription field was a macro.

Fixed in 6.4.0 (for more information, see the CDAP release note):

• Fixed Bigtable batch source plugin. In previous versions, pipelines that included the Bigtable source would fail.
• FTP batch source now works with empty File System Properties.
• Strings are now supported in Min/Max aggregate functions (used in both Group By and Pivot plugins).
• Fixed Salesforce plugin to correctly parse the schema as Avro schema to be sure all the field names are accepted by Avro.
• Fixed data pipeline with BigQuery sink that failed with INVALID_ARGUMENT exception if the range specified was a macro.
• Fixed a class conflict in the Kinesis Spark Streaming source plugin. You can now run pipelines with this source.
• Fixed an issue in field validation logic in pipelines with BigQuery sink that caused a NullPointerException.
• Fixed the Wrangler Generate UUID directive to correctly generate a universally unique identifier (UUID) of the record.
• Fixed advanced joins to recognize auto broadcast setting.
• Fixed Pipeline Studio to use current namespace when it fetches data pipeline drafts.
• Fixed Replication statistics to display on the dashboard for SQL Server.
• Fixed an issue where clicking the Delete button on Replication Assessment page resulted in an error for the replication job.
• Schema name is now shown when selecting tables to replicate.
• Fixed Replication to correctly insert rows that were previous deleted by a replication job.
• Data pipelines running in Spark 3 enabled Dataproc cluster no longer fail with class not found exception.
• Fixed Replication with a SQL Server source to generate rows correctly in BigQuery target table if snapshot failed and restarted.
• Fixed an issue where SQL Server replication job stopped processing data when the connection was reset by the SQL Server.
• Fixed an error in Replication wizard step to select tables, columns and events to replicate, where selecting no columns for a table caused the wizard to fetch all columns in a table.
• Using a macro for a password in a replication job no longer results in an error.
• Fixed logical type display for data pipeline preview runs.
• Fixed Dashboard API to return programs running but started before the startTime.
• Fixed deployed Replication jobs to show advanced configurations in Ui.
• Fixed data pipeline with Python Evaluator transformation to run without stack trace errors.
• Added loading indicator while fetching logs in Log Viewer.
• Fixed Pipeline preview so logical start time function doesn't display as a macro.
• Fixed fields with a list drop down menu in the Replication wizard to default to Select one.
• Added message in Replication Assessment when there are tables that CDF cannot access.
• Used error message when an invalid expression is added in Wrangler.
• Fixed RENAME directive in Wrangler so it is case sensitive.
• Fixed Pipeline Operations UI to stop showing the loading icon forever when it gets error from backend.
• Fixed Wrangler to no longer generate invalid reference names.
• Fixed Wrangler to display logical types instead of java types.
• Fixed pipelines from Wrangler to no longer generate incorrect for xml files.
• Added connection in Wrangler hard codes the name of the JDBC driver.
• Batch data pipelines with Spark 2.2 engine and HDFS sinks no longer fail with delegation token issue error.

Fixed a bug where the setting for the number of executors in streaming pipelines was ignored.

Fixed a race condition where runtime monitoring failed when programs launched concurrently.

Fixed the preview page in table mode so that it shows multiple inputs and outputs with tabs.

Fixed the stability of state transitions for starting pipelines in AppFabric when AppFabric restarts.

Fixed a bug where a metric incorrectly counted the number of the records written in the Google Cloud Storage sink.

Reduced preview startup by 60%. Also added limit to max concurrent preview runs (10 by default).

Fixed a bug that caused errors when Wrangler's parse-as-csv with header was used when reading multiple small files.

Fixed a bug that caused zombie processes when using the Remote Hadoop Provisioner.

Fixed a bug that caused DBSource plugin to fail in preview mode.

Fixed a race condition that caused a failure when running a Spark program.

Fixed a bug that caused Cloud Data Fusion to launch Dataproc clusters in an incorrect project.

Added support for specifying a subnet for the Dataproc provisioner.

Fixed the Cloud Dataproc provisioner to handle networks that do not use automatic subnet creation.