Troubleshoot Connect clusters and connectors

This document provides troubleshooting guidance for Connect clusters and connectors.

Troubleshoot a connector

If a connector fails to start up or fails while running, the operational state is Failed. A connector can also have errors while running; these appear in the connector logs.

To troubleshoot a connector, perform the following steps.

  1. View the connector logs to identify the error. For more information, see View connector logs.

  2. Identify the root cause of the error. The following topics describe specific errors that might occur for each connector type:

  3. If the connector is mis-configured, update the connector's configuration and then resume the connector. Transient failures might not require any changes to the configuration.

As a best practice, set the connector's task restart policy to restart with exponential backoff. This setting helps to avoid connector failures due to transient errors.

Connect cluster errors

This section describes errors that might occur in a Connect cluster.

Metadata topic has 'cleanup policy=delete'

The Connect cluster logs show an error similar to the following:

org.apache.kafka.common.config.ConfigException: Topic 'connect-offsets-
PROJECT_ID-LOCATION-CONNECT_CLUSTER_NAME-UNIQUE_ID' supplied via the
'offset.storage.topic' property is required to have 'cleanup policy=compact' to
guarantee consistency and durability of source connector offsets, but found the
topic currently has 'cleanup policy=delete'.

This error might occur for the following reasons:

  • The Connect cluster's primary Kafka cluster was deleted and then recreated with the same name.
  • The metadata topics created by the Connect cluster in the primary Kafka cluster were deleted.

To resolve this error, perform the following steps:

  1. Locate the metadata topics in the primary Kafka cluster. The metadata topics are named with the following format:

    connect-status-PROJECT_ID-LOCATION-CONNECT_CLUSTER_NAME-UNIQUE_ID

    For example:

    connect-status-project1-us-central1-connect1-3c563900-a4e4-4eDe-88f7-73b16d6344d2

  2. For each metadata topic, edit the topic and add the key-value pair cleanup.policy:compact to the Non-default parameters field. For more information, see Update a Kafka topic.

What's next