Retry strategy

C++

By default, operations support retries for the following HTTP error codes, as well as any socket errors that indicate the connection was lost or never successfully established.

• 408 Request Timeout
• 429 Too Many Requests
• 500 Internal Server Error
• 502 Bad Gateway
• 503 Service Unavailable
• 504 Gateway Timeout

All exponential backoff and retry settings in the C++ library are configurable. If the algorithms implemented in the library do not support your needs, you can provide custom code to implement your own strategies.

By default, the C++ library retries all operations with retryable errors, even those that are never idempotent and can delete or create multiple resources when repeatedly successful. To only retry idempotent operations, use the google::cloud::storage::StrictIdempotencyPolicy class.

C#

The C# client library uses exponential backoff by default.

Go

By default, operations support retries for the following errors:

• Connection errors:
  • io.ErrUnexpectedEOF: This may occur due to transient network issues.
  • url.Error containing connection refused: This may occur due to transient network issues.
  • url.Error containing connection reset by peer: This means that Google Cloud has reset the connection.
  • net.ErrClosed: This means that Google Cloud has closed the connection.
• HTTP codes:
  • 408 Request Timeout
  • 429 Too Many Requests
  • 500 Internal Server Error
  • 502 Bad Gateway
  • 503 Service Unavailable
  • 504 Gateway Timeout
• Errors that implement the Temporary() interface and give a value of err.Temporary() == true
• Any of the above errors that have been wrapped using Go 1.13 error wrapping

All exponential backoff settings in the Go library are configurable. By default, operations in Go use the following settings for exponential backoff (defaults are taken from gax):

In general, retrying continues indefinitely unless the controlling context is canceled, the client is closed, or a non-transient error is received. To stop retries from continuing, use context timeouts or cancellation. The only exception to this behavior is when performing resumable uploads using Writer, where the data is large enough that it requires multiple requests. In this scenario, each chunk times out and stops retrying after 32 seconds by default. You can adjust the default timeout by changing Writer.ChunkRetryDeadline.

There is a subset of Go operations that are conditionally idempotent (conditionally safe to retry). These operations only retry if they meet specific conditions:

• GenerationMatch or Generation

  • Safe to retry if a GenerationMatch precondition was applied to the call, or if ObjectHandle.Generation was set.

• MetagenerationMatch

  • Safe to retry if a MetagenerationMatch precondition was applied to the call.

• Etag

  • Safe to retry if the method inserts an etag into the JSON request body. Only used in HMACKeyHandle.Update when HmacKeyMetadata.Etag has been set.

RetryPolicy is set to RetryPolicy.RetryIdempotent by default. See Customize retries for examples on how to modify the default retry behavior.

Java

By default, operations support retries for the following errors:

• Connection errors:
  • Connection reset by peer: This means that Google Cloud has reset the connection.
  • Unexpected connection closure: This means Google Cloud has closed the connection.
• HTTP codes:
  • 408 Request Timeout
  • 429 Too Many Requests
  • 500 Internal Server Error
  • 502 Bad Gateway
  • 503 Service Unavailable
  • 504 Gateway Timeout

All exponential backoff settings in the Java library are configurable. By default, operations through Java use the following settings for exponential backoff:

For more information about the settings, see the Java reference documentation for RetrySettings.Builder and HttpTransportOptions.Builder.

There is a subset of Java operations that are conditionally idempotent (conditionally safe to retry). These operations only retry if they include specific arguments:

• ifGenerationMatch or generation

  • Safe to retry if ifGenerationMatch or generation was passed in as an option to the method.

• ifMetagenerationMatch

  • Safe to retry if ifMetagenerationMatch was passed in as an option.

StorageOptions.setStorageRetryStrategy is set to StorageRetryStrategy#getDefaultStorageRetryStrategy by default. See Customize retries for examples on how to modify the default retry behavior.

Node.js

By default, operations support retries for the following error codes:

• Connection errors:
  • EAI_again: This is a DNS lookup error. For more information, see the getaddrinfo documentation.
  • Connection reset by peer: This means that Google Cloud has reset the connection.
  • Unexpected connection closure: This means Google Cloud has closed the connection.
• HTTP codes:
  • 408 Request Timeout
  • 429 Too Many Requests
  • 500 Internal Server Error
  • 502 Bad Gateway
  • 503 Service Unavailable
  • 504 Gateway Timeout

All exponential backoff settings in the Node.js library are configurable. By default, operations through Node.js use the following settings for exponential backoff:

There is a subset of Node.js operations that are conditionally idempotent (conditionally safe to retry). These operations only retry if they include specific arguments:

• ifGenerationMatch or generation

  • Safe to retry if ifGenerationMatch or generation was passed in as an option to the method. Often, methods only accept one of these two parameters.

• ifMetagenerationMatch

  • Safe to retry if ifMetagenerationMatch was passed in as an option.

retryOptions.idempotencyStrategy is set to IdempotencyStrategy.RetryConditional by default. See Customize retries for examples on how to modify the default retry behavior.

PHP

The PHP client library uses exponential backoff by default.

Python

By default, operations support retries for the following error codes:

• Connection errors:
  • requests.exceptions.ConnectionError
  • requests.exceptions.ChunkedEncodingError (only for operations that fetch or send payload data to objects, like uploads and downloads)
  • ConnectionError
• HTTP codes:
  • 408 Request Timeout
  • 429 Too Many Requests
  • 500 Internal Server Error
  • 502 Bad Gateway
  • 503 Service Unavailable
  • 504 Gateway Timeout

Operations through Python use the following default settings for exponential backoff:

There is a subset of Python operations that are conditionally idempotent (conditionally safe to retry) when they include specific arguments. These operations only retry if a condition case passes:

• DEFAULT_RETRY_IF_GENERATION_SPECIFIED

  • Safe to retry if generation or if_generation_match was passed in as an argument to the method. Often methods only accept one of these two parameters.

• DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED

  • Safe to retry if if_metageneration_match was passed in as an argument to the method.

• DEFAULT_RETRY_IF_ETAG_IN_JSON

  • Safe to retry if the method inserts an etag into the JSON request body. For HMACKeyMetadata.update() this means etag must be set on the HMACKeyMetadata object itself. For the set_iam_policy() method on other classes, this means the etag must be set in the "policy" argument passed into the method.

Ruby

By default, operations support retries for the following error codes:

• Connection errors:
  • SocketError
  • HTTPClient::TimeoutError
  • Errno::ECONNREFUSED
  • HTTPClient::KeepAliveDisconnected
• HTTP codes:
  • 408 Request Timeout
  • 429 Too Many Requests
  • 5xx Server Error

All exponential backoff settings in the Ruby client library are configurable. By default, operations through the Ruby client library use the following settings for exponential backoff:

There is a subset of Ruby operations that are conditionally idempotent (conditionally safe to retry) when they include specific arguments:

• if_generation_match or generation

  • Safe to retry if the generation or if_generation_match parameter is passed in as an argument to the method. Often methods only accept one of these two parameters.

• if_metageneration_match

  • Safe to retry if the if_metageneration_match parameter is passed in as an option.

By default, all idempotent operations are retried, and conditionally idempotent operations are retried only if the condition case passes. Non-idempotent operations are not retried. See Customize retries for examples on how to modify the default retry behavior.

C++

To customize the retry behavior, provide values for the following options when you initialize the google::cloud::storage::Client object:

• google::cloud::storage::RetryPolicyOption: The library provides google::cloud::storage::LimitedErrorCountRetryPolicy and google::cloud::storage::LimitedTimeRetryPolicy classes. You can provide your own class, which must implement the google::cloud::RetryPolicy interface.

• google::cloud::storage::BackoffPolicyOption: The library provides the google::cloud::storage::ExponentialBackoffPolicy class. You can provide your own class, which must implement the google::cloud::storage::BackoffPolicy interface.

• google::cloud::storage::IdempotencyPolicyOption: The library provides the google::cloud::storage::StrictIdempotencyPolicy and google::cloud::storage::AlwaysRetryIdempotencyPolicy classes. You can provide your own class, which must implement the google::cloud::storage::IdempotencyPolicy interface.

For more information, see the C++ client library reference documentation.

namespace gcs = ::google::cloud::storage;
// Create the client configuration:
auto options = google::cloud::Options{};
// Retries only idempotent operations.
options.set<gcs::IdempotencyPolicyOption>(
    gcs::StrictIdempotencyPolicy().clone());
// On error, it backs off for a random delay between [1, 3] seconds, then [3,
// 9] seconds, then [9, 27] seconds, etc. The backoff time never grows larger
// than 1 minute.
options.set<gcs::BackoffPolicyOption>(
    gcs::ExponentialBackoffPolicy(
        /*initial_delay=*/std::chrono::seconds(1),
        /*maximum_delay=*/std::chrono::minutes(1),
        /*scaling=*/3.0)
        .clone());
// Retries all operations for up to 5 minutes, including any backoff time.
options.set<gcs::RetryPolicyOption>(
    gcs::LimitedTimeRetryPolicy(std::chrono::minutes(5)).clone());
return gcs::Client(std::move(options));

C#

You cannot customize the default retry strategy used by the C# client library.

Go

When you initialize a storage client, a default retry configuration will be set. Unless they're overridden, the options in the config are set to the default values. Users can configure non-default retry behavior for a single library call (using BucketHandle.Retryer and ObjectHandle.Retryer) or for all calls made by a client (using Client.SetRetry). To modify retry behavior, pass in the relevant RetryOptions to one of these methods.

See the following code sample to learn how to customize your retry behavior.

import (
	"context"
	"fmt"
	"io"
	"time"

	"cloud.google.com/go/storage"
	"github.com/googleapis/gax-go/v2"
)

// configureRetries configures a custom retry strategy for a single API call.
func configureRetries(w io.Writer, bucket, object string) error {
	// bucket := "bucket-name"
	// object := "object-name"
	ctx := context.Background()
	client, err := storage.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("storage.NewClient: %w", err)
	}
	defer client.Close()

	// Configure retries for all operations using this ObjectHandle. Retries may
	// also be configured on the BucketHandle or Client types.
	o := client.Bucket(bucket).Object(object).Retryer(
		// Use WithBackoff to control the timing of the exponential backoff.
		storage.WithBackoff(gax.Backoff{
			// Set the initial retry delay to a maximum of 2 seconds. The length of
			// pauses between retries is subject to random jitter.
			Initial: 2 * time.Second,
			// Set the maximum retry delay to 60 seconds.
			Max: 60 * time.Second,
			// Set the backoff multiplier to 3.0.
			Multiplier: 3,
		}),
		// Use WithPolicy to customize retry so that all requests are retried even
		// if they are non-idempotent.
		storage.WithPolicy(storage.RetryAlways),
	)

	// Use context timeouts to set an overall deadline on the call, including all
	// potential retries.
	ctx, cancel := context.WithTimeout(ctx, 500*time.Second)
	defer cancel()

	// Delete an object using the specified retry policy.
	if err := o.Delete(ctx); err != nil {
		return fmt.Errorf("Object(%q).Delete: %w", object, err)
	}
	fmt.Fprintf(w, "Blob %v deleted with a customized retry strategy.\n", object)
	return nil
}

Java

When you initialize Storage, an instance of RetrySettings is initialized as well. Unless they are overridden, the options in the RetrySettings are set to the default values. To modify the default automatic retry behavior, pass the custom StorageRetryStrategy into the StorageOptions used to construct the Storage instance. To modify any of the other scalar parameters, pass a custom RetrySettings into the StorageOptions used to construct the Storage instance.

See the following example to learn how to customize your retry behavior:

import com.google.api.gax.retrying.RetrySettings;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.cloud.storage.StorageRetryStrategy;
import org.threeten.bp.Duration;

public final class ConfigureRetries {
  public static void main(String[] args) {
    String bucketName = "my-bucket";
    String blobName = "blob/to/delete";
    deleteBlob(bucketName, blobName);
  }

  static void deleteBlob(String bucketName, String blobName) {
    // Customize retry behavior
    RetrySettings retrySettings =
        StorageOptions.getDefaultRetrySettings()
            .toBuilder()
            // Set the max number of attempts to 10 (initial attempt plus 9 retries)
            .setMaxAttempts(10)
            // Set the backoff multiplier to 3.0
            .setRetryDelayMultiplier(3.0)
            // Set the max duration of all attempts to 5 minutes
            .setTotalTimeout(Duration.ofMinutes(5))
            .build();

    StorageOptions alwaysRetryStorageOptions =
        StorageOptions.newBuilder()
            // Customize retry so all requests are retried even if they are non-idempotent.
            .setStorageRetryStrategy(StorageRetryStrategy.getUniformStorageRetryStrategy())
            // provide the previously configured retrySettings
            .setRetrySettings(retrySettings)
            .build();

    // Instantiate a client
    Storage storage = alwaysRetryStorageOptions.getService();

    // Delete the blob
    BlobId blobId = BlobId.of(bucketName, blobName);
    boolean success = storage.delete(blobId);

    System.out.printf(
        "Deletion of Blob %s completed %s.%n", blobId, success ? "successfully" : "unsuccessfully");
  }
}

Node.js

When you initialize Cloud Storage, a retryOptions config file is initialized as well. Unless they're overridden, the options in the config are set to the default values. To modify the default retry behavior, pass the custom retry configuration retryOptions into the storage constructor upon initialization. The Node.js client library can automatically use backoff strategies to retry requests with the autoRetry parameter.

See the following code sample to learn how to customize your retry behavior.

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The ID of your GCS file
// const fileName = 'your-file-name';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage({
  retryOptions: {
    // If this is false, requests will not retry and the parameters
    // below will not affect retry behavior.
    autoRetry: true,
    // The multiplier by which to increase the delay time between the
    // completion of failed requests, and the initiation of the subsequent
    // retrying request.
    retryDelayMultiplier: 3,
    // The total time between an initial request getting sent and its timeout.
    // After timeout, an error will be returned regardless of any retry attempts
    // made during this time period.
    totalTimeout: 500,
    // The maximum delay time between requests. When this value is reached,
    // retryDelayMultiplier will no longer be used to increase delay time.
    maxRetryDelay: 60,
    // The maximum number of automatic retries attempted before returning
    // the error.
    maxRetries: 5,
    // Will respect other retry settings and attempt to always retry
    // conditionally idempotent operations, regardless of precondition
    idempotencyStrategy: IdempotencyStrategy.RetryAlways,
  },
});
console.log(
  'Functions are customized to be retried according to the following parameters:'
);
console.log(`Auto Retry: ${storage.retryOptions.autoRetry}`);
console.log(
  `Retry delay multiplier: ${storage.retryOptions.retryDelayMultiplier}`
);
console.log(`Total timeout: ${storage.retryOptions.totalTimeout}`);
console.log(`Maximum retry delay: ${storage.retryOptions.maxRetryDelay}`);
console.log(`Maximum retries: ${storage.retryOptions.maxRetries}`);
console.log(
  `Idempotency strategy: ${storage.retryOptions.idempotencyStrategy}`
);

async function deleteFileWithCustomizedRetrySetting() {
  await storage.bucket(bucketName).file(fileName).delete();
  console.log(`File ${fileName} deleted with a customized retry strategy.`);
}

deleteFileWithCustomizedRetrySetting();

PHP

You cannot customize the default retry strategy used by the PHP client library.

Python

To modify the default retry behavior, create a copy of the google.cloud.storage.retry.DEFAULT_RETRY object by calling it with a with_XXX method. The Python client library automatically uses backoff strategies to retry requests if you include the DEFAULT_RETRY parameter.

Note that with_predicate is not supported for operations that fetch or send payload data to objects, like uploads and downloads. It's recommended that you modify attributes one by one. For more information, see the google-api-core Retry reference.

To configure your own conditional retry, create a ConditionalRetryPolicy object and wrap your custom Retry object with DEFAULT_RETRY_IF_GENERATION_SPECIFIED, DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED, or DEFAULT_RETRY_IF_ETAG_IN_JSON.

See the following code sample to learn how to customize your retry behavior.

from google.cloud import storage
from google.cloud.storage.retry import DEFAULT_RETRY


def configure_retries(bucket_name, blob_name):
    """Configures retries with customizations."""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The ID of your GCS object
    # blob_name = "your-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(blob_name)

    # Customize retry with a deadline of 500 seconds (default=120 seconds).
    modified_retry = DEFAULT_RETRY.with_deadline(500.0)
    # Customize retry with an initial wait time of 1.5 (default=1.0).
    # Customize retry with a wait time multiplier per iteration of 1.2 (default=2.0).
    # Customize retry with a maximum wait time of 45.0 (default=60.0).
    modified_retry = modified_retry.with_delay(initial=1.5, multiplier=1.2, maximum=45.0)

    # blob.delete() uses DEFAULT_RETRY_IF_GENERATION_SPECIFIED by default.
    # Override with modified_retry so the function retries even if the generation
    # number is not specified.
    print(
        f"The following library method is customized to be retried according to the following configurations: {modified_retry}"
    )

    blob.delete(retry=modified_retry)
    print(f"Blob {blob_name} deleted with a customized retry strategy.")

Ruby

When you initialize the storage client, all retry configurations are set to the values shown in the table above. To modify the default retry behavior, pass retry configurations while initializing the storage client.

To override the number of retries for a particular operation, pass retries in the options parameter of the operation.

def configure_retries bucket_name: nil, file_name: nil
  # The ID of your GCS bucket
  # bucket_name = "your-unique-bucket-name"

  # The ID of your GCS object
  # file_name = "your-file-name"

  require "google/cloud/storage"

  # Creates a client
  storage = Google::Cloud::Storage.new(

    # The maximum number of automatic retries attempted before returning
    # the error.
    #
    # Customize retry configuration with the maximum retry attempt of 5.
    retries: 5,

    # The total time in seconds that requests are allowed to keep being retried.
    # After max_elapsed_time, an error will be returned regardless of any
    # retry attempts made during this time period.
    #
    # Customize retry configuration with maximum elapsed time of 500 seconds.
    max_elapsed_time: 500,

    # The initial interval between the completion of failed requests, and the
    # initiation of the subsequent retrying request.
    #
    # Customize retry configuration with an initial interval of 1.5 seconds.
    base_interval: 1.5,

    # The maximum interval between requests. When this value is reached,
    # multiplier will no longer be used to increase the interval.
    #
    # Customize retry configuration with maximum interval of 45.0 seconds.
    max_interval: 45,

    # The multiplier by which to increase the interval between the completion
    # of failed requests, and the initiation of the subsequent retrying request.
    #
    # Customize retry configuration with an interval multiplier per iteration of 1.2.
    multiplier: 1.2
  )

  # Uses the retry configuration set during the client initialization above with 5 retries
  file = storage.service.get_file bucket_name, file_name

  # Maximum retry attempt can be overridden for each operation using options parameter.
  storage.service.delete_file bucket_name, file_name, options: { retries: 4 }
  puts "File #{file.name} deleted with a customized retry strategy."
end