Troubleshoot a Pub/Sub connector

This section provides troubleshooting guidance for common issues encountered when using a Pub/Sub source connector or Pub/Sub sink connector.

Missing Pub/Sub permissions

If the Managed Kafka service account doesn't have permission to access the Pub/Sub resources, the following error occurs:

PERMISSION_DENIED: User not authorized to perform this action.

This error might occur for several reasons.

  • Pub/Sub Source connector: The Managed Kafka service account might not have permission to access the Pub/Sub subscription. To resolve this issue, grant the Managed Kafka service account both of the following Identity and Access Management (IAM) roles:

    • Pub/Sub Subscriber (roles/pubsub.subscriber)
    • Pub/Sub Viewer (roles/pubsub.viewer)

    For more information, see Grant permissions to read from the Pub/Sub subscription.

  • Pub/Sub Sink connector: The Managed Kafka service account might not have permission to publish to the Pub/Sub topic. To resolve this issue, grant the Managed Kafka service the Pub/Sub Publisher (roles/pubsub.publisher) IAM role.

    For more information, see Grant permissions to publish to the Pub/Sub topic.

Error verifying the subscription

If the Pub/Sub Source connector can't find the Pub/Sub subscription, the connector fails with an error similar to the following:

org.apache.kafka.connect.errors.ConnectException: Error verifying the subscription SUBSCRIPTION for project PROJECT

To resolve this issue, verify that the connector's cps.subscription configuration specifies the correct subscription, and that the subscription exists in your project.

Connector tasks failed to convert records

If the connector's key converter or value converter is incompatible with the data type of records, the connector fails with an error similar to the following:

org.apache.kafka.connect.errors.DataException: Invalid schema type for
CONVERTER_NAME: DATA_FORMAT

To resolve this issue, change the key converter or value converter to the correct class to support the schema data type.

For example, if you the set the key converter to ByteArrayConverter, you might see the error "Invalid schema type for ByteArrayConverter: STRING". This error occurs because the ByteArrayConverter doesn't support STRING types. Instead, use org.apache.kafka.connect.storage.StringConverter.

What's next