Retry steps

You can define a retry policy to retry steps that have returned a specific error code; for example, a particular HTTP status code. The retry syntax described in this document lets you do the following:

  • Define the maximum number of retry attempts
  • Define a backoff model to increase the likelihood of success

Note that retrying a step counts as an additional step execution for pricing purposes. For more information, see Pricing.

Syntax for try/retry structure

A retry policy consists of a predicate that defines which error codes should be retried, as well as retry configuration values such as the maximum number of retries, and the delay in seconds between retries.

For example, given the following retry configuration values, the step is retried a total of eight times:

YAML

max_retries: 8
backoff:
    initial_delay: 1
    max_delay: 60
    multiplier: 2

JSON

{
  "max_retries": 8,
  "backoff": {
    "initial_delay": 1,
    "max_delay": 60,
    "multiplier": 2
  }
}

The initial delay is 1 second, and the delay is doubled on each attempt, with a maximum delay of 60 seconds. The delays between subsequent attempts are: 1, 2, 4, 8, 16, 32, 60, and 60 (time given in seconds). After eight retry attempts, the step is considered failed, and an exception is raised. Counting the initial execution, the step is executed nine times.

To apply a retry policy, you can configure the try/retry structure in one of three ways:

  • Use a default retry policy
  • Use a default retry predicate with custom retry configuration values
  • Use a custom retry policy

Default retry policy

Workflows has default retry policies available for both idempotent and non-idempotent steps.

Since the default retry policies consist of a default retry predicate and default retry configuration values, you don't need to specify a retry predicate or define the retry configuration values.

YAML

- step_name:
    try:
        steps:
            STEPS_BLOCK
    retry: RETRY_POLICY

JSON

[
  {
    "step_name": {
      "try": {
        "steps": [
          STEPS_BLOCK
        ]
      },
      "retry": "RETRY_POLICY"
    }
  }
]

Replace the following:

  • STEPS_BLOCK: optional. The steps block can contain the following:
  • RETRY_POLICY: the default retry policy to use. One of the following:
    • ${http.default_retry} which has the following default configuration: 
        predicate: ${http.default_retry_predicate}
  max_retries: 5
  backoff:
      initial_delay: 1
      max_delay: 60
      multiplier: 1.25

      This policy retries HTTP status codes [429, 502, 503, 504], connection errors, connection failures, or timeout errors.

    • ${http.default_retry_non_idempotent} which has the following default configuration: 
        predicate: ${http.default_retry_predicate_non_idempotent}
  max_retries: 5
  backoff:
      initial_delay: 1
      max_delay: 60
      multiplier: 1.25

      This policy retries HTTP status codes [429, 503] or connection failures.

Examples:

YAML

- idempotent_step:
    try:
        call: http.get
        args:
            url: https://example.com/api
    retry: ${http.default_retry}

JSON

[
  {
    "idempotent_step": {
      "try": {
        "call": "http.get",
        "args": {
          "url": "https://example.com/api"
        }
      },
      "retry": "${http.default_retry}"
    }
  }
]

YAML

- non_idempotent_step:
    try:
        call: http.get
        args:
            url: https://example.com/api
    retry: ${http.default_retry_non_idempotent}

JSON

[
  {
    "non_idempotent_step": {
      "try": {
        "call": "http.get",
        "args": {
          "url": "https://example.com/api"
        }
      },
      "retry": "${http.default_retry_non_idempotent}"
    }
  }
]

Default retry predicate

You can use a default retry predicate with custom retry configuration values. Select the appropriate predicate for the type of step (idempotent or non-idempotent), then define the retry configuration values.

YAML

- step_name:
    try:
        steps:
            STEPS_BLOCK
    retry:
        predicate: RETRY_PREDICATE
        max_retries: NUMBER_OF_RETRIES
        backoff:
            initial_delay: DELAY_SECONDS
            max_delay: MAX_DELAY_SECONDS
            multiplier: DELAY_MULTIPLIER

JSON

[
  {
    "step_name": {
      "try": {
        "steps": [
          STEPS_BLOCK
        ]
      },
      "retry": {
        "predicate": "RETRY_PREDICATE",
        "max_retries": NUMBER_OF_RETRIES,
        "backoff": {
          "initial_delay": DELAY_SECONDS,
          "max_delay": MAX_DELAY_SECONDS,
          "multiplier": DELAY_MULTIPLIER
        }
      }
    }
  }
]

Replace the following:

  • STEPS_BLOCK: optional. The steps block can contain the following:
  • RETRY_PREDICATE: defines which error codes will be retried. One of the following:
    • ${http.default_retry_predicate}—retries HTTP status codes [429, 502, 503, 504], connection errors, connection failures, or timeout errors
    • ${http.default_retry_predicate_non_idempotent}—retries HTTP status codes [429, 503] or connection failures
  • NUMBER_OF_RETRIES: maximum number of times a step will be retried, not counting the initial step execution attempt.
  • DELAY_SECONDS: delay in seconds between the initial failure and the first retry.
  • MAX_DELAY_SECONDS: maximum delay in seconds between retries.
  • DELAY_MULTIPLIER: multiplier applied to the previous delay to calculate the delay for the subsequent retry.

Example:

YAML

- step_name:
    try:
        steps:
            ...
    retry:
        predicate: ${http.default_retry_predicate_non_idempotent}
        max_retries: 10
        backoff:
            initial_delay: 1
            max_delay: 90
            multiplier: 3

JSON

[
  {
    "step_name": {
      "try": {
        "steps":
        ...
      },
      "retry": {
        "predicate": "${http.default_retry_predicate_non_idempotent}",
        "max_retries": 10,
        "backoff": {
          "initial_delay": 1,
          "max_delay": 90,
          "multiplier": 3
        }
      }
    }
  }
]

Custom retry policy

If the existing default retry policies don't work for your use case, you can use a custom retry policy by creating a subworkflow to define your predicate. Then configure your retry configuration values, including your predicate, in the retry block of the main workflow.

YAML

- step_name:
    try:
        steps:
            STEPS_BLOCK
    retry:
        predicate: CUSTOM_PREDICATE
        max_retries: NUMBER_OF_RETRIES
        backoff:
            initial_delay: DELAY_SECONDS
            max_delay: MAX_DELAY_SECONDS
            multiplier: DELAY_MULTIPLIER

JSON

[
  {
    "step_name": {
      "try": {
        "steps": [
          STEPS_BLOCK
        ]
      },
      "retry": {
        "predicate": "CUSTOM_PREDICATE",
        "max_retries": NUMBER_OF_RETRIES,
        "backoff": {
          "initial_delay": DELAY_SECONDS,
          "max_delay": MAX_DELAY_SECONDS,
          "multiplier": DELAY_MULTIPLIER
        }
      }
    }
  }
]

Replace the following:

  • STEPS_BLOCK: optional. The steps block can contain the following:
  • CUSTOM_PREDICATE: a subworkflow which accepts as its single argument a map representing the exception, and which returns true to trigger a retry and, otherwise, false.
  • NUMBER_OF_RETRIES: maximum number of times a step will be retried, not counting the initial step execution attempt.
  • DELAY_SECONDS: delay in seconds between the initial failure and the first retry.
  • MAX_DELAY_SECONDS: maximum delay in seconds between retries.
  • DELAY_MULTIPLIER: multiplier applied to the previous delay to calculate the delay for the subsequent retry.

Example:

YAML

  main:
      - step_name:
          try:
              steps:
                  ...
          retry:
              predicate: ${retry_predicate}
              max_retries: number_of_retries
              backoff:
                  initial_delay: delay_seconds
                  max_delay: max_delay_seconds
                  multiplier: delay_multiplier

  retry_predicate:
      params: [e]
      steps:
          ...

JSON

  {
    "main": [
      {
        "step_name": {
          "try": {
            "steps":
            ...
          },
          "retry": {
            "predicate": "${retry_predicate}",
            "max_retries": "number_of_retries",
            "backoff": {
              "initial_delay": "delay_seconds",
              "max_delay": "max_delay_seconds",
              "multiplier": "delay_multiplier"
            }
          }
        }
      }
    ],
    "retry_predicate": {
      "params": [
        "e"
      ],
      "steps":
      ...
    }
  }

The predicate is checked against any errors that are raised, and determines if the error triggers a retry. To trigger a retry, return true; otherwise, return false. If an error isn't retried, or the retries are exhausted, the error is propagated, and can cause the execution to fail. You can handle this by using an except block.

Note that when using a custom predicate, errors are only raised for the following HTTP status codes:

  • Client error responses (400-499)
  • Server error responses (500-599)

To trigger the predicate for other HTTP status codes, you must explicitly raise an exception. For an example, see Retry steps using a custom retry policy for other HTTP status codes.

Samples

These samples demonstrate the syntax.

Retry steps using a default retry policy

This sample uses a default retry policy (${http.default_retry}) for HTTP requests.

YAML

- read_item:
    try:
      call: http.get
      args:
        url: https://example.com/someapi
      result: apiResponse
    retry: ${http.default_retry}

JSON

[
  {
    "read_item": {
      "try": {
        "call": "http.get",
        "args": {
          "url": "https://example.com/someapi"
        },
        "result": "apiResponse"
      },
      "retry": "${http.default_retry}"
    }
  }
]

Retry steps using a custom retry policy

This sample implements a custom retry policy that retries HTTP requests that return an HTTP status code 500.

YAML

main:
  steps:
    - read_item:
        try:
          call: http.get
          args:
            url: https://host.com/api
          result: api_response
        retry:
          predicate: ${custom_predicate}
          max_retries: 5
          backoff:
            initial_delay: 2
            max_delay: 60
            multiplier: 2
    - last_step:
        return: "OK"

custom_predicate:
  params: [e]
  steps:
    - what_to_repeat:
        switch:
          - condition: ${e.code == 500}
            return: true
    - otherwise:
        return: false

JSON

{
  "main": {
    "steps": [
      {
        "read_item": {
          "try": {
            "call": "http.get",
            "args": {
              "url": "https://host.com/api"
            },
            "result": "api_response"
          },
          "retry": {
            "predicate": "${custom_predicate}",
            "max_retries": 5,
            "backoff": {
              "initial_delay": 2,
              "max_delay": 60,
              "multiplier": 2
            }
          }
        }
      },
      {
        "last_step": {
          "return": "OK"
        }
      }
    ]
  },
  "custom_predicate": {
    "params": [
      "e"
    ],
    "steps": [
      {
        "what_to_repeat": {
          "switch": [
            {
              "condition": "${e.code == 500}",
              "return": true
            }
          ]
        }
      },
      {
        "otherwise": {
          "return": false
        }
      }
    ]
  }
}

Retry steps using a custom retry policy for other HTTP status codes

This sample implements a custom retry policy that retries HTTP requests that return an HTTP status code 202.

YAML

main:
  steps:
    - read_item:
        try:
          steps:
            - callStep:
                call: http.get
                args:
                  url: https://host.com/api
                result: api_response
            - checkNotOK:
                switch:
                  - condition: ${api_response.code == 202}
                    raise: ${api_response}
        retry:
          predicate: ${custom_predicate}
          max_retries: 5
          backoff:
            initial_delay: 2
            max_delay: 60
            multiplier: 2

custom_predicate:
  params: [e]
  steps:
    - what_to_repeat:
        switch:
          - condition: ${e.code == 202}
            return: true
    - otherwise:
        return: false

JSON

{
  "main": {
    "steps": [
      {
        "read_item": {
          "try": {
            "steps": [
              {
                "callStep": {
                  "call": "http.get",
                  "args": {
                    "url": "https://host.com/api"
                  },
                  "result": "api_response"
                }
              },
              {
                "checkNotOK": {
                  "switch": [
                    {
                      "condition": "${api_response.code == 202}",
                      "raise": "${api_response}"
                    }
                  ]
                }
              }
            ]
          },
          "retry": {
            "predicate": "${custom_predicate}",
            "max_retries": 5,
            "backoff": {
              "initial_delay": 2,
              "max_delay": 60,
              "multiplier": 2
            }
          }
        }
      }
    ]
  },
  "custom_predicate": {
    "params": [
      "e"
    ],
    "steps": [
      {
        "what_to_repeat": {
          "switch": [
            {
              "condition": "${e.code == 202}",
              "return": true
            }
          ]
        }
      },
      {
        "otherwise": {
          "return": false
        }
      }
    ]
  }
}

Retry steps using a custom configuration

This sample implements a default retry predicate (${http.default_retry_predicate}) to determine when to perform a retry, and custom retry configuration values.

YAML

- read_item:
    try:
      call: http.get
      args:
        url: https://example.com/someapi
      result: api_response
    retry:
      predicate: ${http.default_retry_predicate}
      max_retries: 5
      backoff:
        initial_delay: 2
        max_delay: 60
        multiplier: 2

JSON

[
  {
    "read_item": {
      "try": {
        "call": "http.get",
        "args": {
          "url": "https://example.com/someapi"
        },
        "result": "api_response"
      },
      "retry": {
        "predicate": "${http.default_retry_predicate}",
        "max_retries": 5,
        "backoff": {
          "initial_delay": 2,
          "max_delay": 60,
          "multiplier": 2
        }
      }
    }
  }
]

Handle errors using a custom predicate

This sample defines a custom error handler, including a custom predicate, and custom backoff parameters. The custom predicate is defined as a subworkflow which accepts as its single argument a map representing the exception, and which returns true to trigger a retry and, otherwise, false.

YAML

# Define a custom error handler, custom predicate, and custom backoff parameters
# The `my_own_predicate` subworkflow accepts a map as an argument and defines the
# exception; it returns true if a retry; false, otherwise
# Expected outcome: the execution fails and returns an HTTP 404 Not Found error
main:
  steps:
    - read_item:
        try:
          call: http.get
          args:
            url: https://example.com/someapi
          result: api_response
        retry:
          predicate: ${my_own_predicate}
          max_retries: 5
          backoff:
            initial_delay: 2
            max_delay: 60
            multiplier: 2
    - last_step:
        return: "OK"

my_own_predicate:
  params: [e]
  steps:
    - log_error_tags:
        call: sys.log
        args:
          data: ${e.tags}
          severity: "INFO"
    - log_error_message:
        call: sys.log
        args:
          data: ${e.message}
          severity: "INFO"
    - log_error_code:
        call: sys.log
        args:
          data: ${e.code}
          severity: "INFO"
    - what_to_repeat:
        switch:
          - condition: ${e.code == 202}
            return: true
    - otherwise:
        return: false

JSON

{
  "main": {
    "steps": [
      {
        "read_item": {
          "try": {
            "call": "http.get",
            "args": {
              "url": "https://example.com/someapi"
            },
            "result": "api_response"
          },
          "retry": {
            "predicate": "${my_own_predicate}",
            "max_retries": 5,
            "backoff": {
              "initial_delay": 2,
              "max_delay": 60,
              "multiplier": 2
            }
          }
        }
      },
      {
        "last_step": {
          "return": "OK"
        }
      }
    ]
  },
  "my_own_predicate": {
    "params": [
      "e"
    ],
    "steps": [
          {
            "log_error_tags": {
              "call": "sys.log",
              "args": {
                "data": "${e.tags}",
                "severity": "INFO"
              }
            }
          },
          {
            "log_error_message": {
              "call": "sys.log",
              "args": {
                "data": "${e.message}",
                "severity": "INFO"
              }
            }
          },
          {
            "log_error_code": {
              "call": "sys.log",
              "args": {
                "data": "${e.code}",
                "severity": "INFO"
              }
            }
          },
          {
            "what_to_repeat": {
              "switch": [
                {
                  "condition": "${e.code == 202}",
                  "return": true
                }
              ]
            }
          },
          {
            "otherwise": {
              "return": false
            }
          }
        ]
      }
    }

When you run the preceding workflow, the execution fails, and returns an HTTP 404 Not Found error. To demonstrate this, the custom predicate uses the standard library sys.log function and writes elements from the error map to the log. For example, the log output should be similar to the following:

{
  "textPayload": "[\"HttpError\"]",
  ...
  "severity": "INFO",
  ...
}
{
  "textPayload": "HTTP server responded with error code 404",
  ...
  "severity": "INFO",
  ...
}
{
  "textPayload": "404",
  ...
  "severity": "INFO",
  ...
}

What's next