Using Pull Queues in Go

This page provides an overview of pull queues in the App Engine standard environment.

In push queues tasks are delivered to a worker service based on the queue's configuration. In pull queues the worker service must ask the queue for tasks. The queue responds by allowing that worker unique access to process the task for a specified period of time, which is called a lease.

Using pull queues, you can also group related tasks using tags and then configure your worker to pull multiple tasks with a certain tag all at once. This process is called batching.

If a worker cannot process a task before its lease expires, it can either renew the lease or let it expire, at which point another worker can acquire it. Once the work associated with a task is complete, the worker must delete it.

Using pull queues requires your code to handle some functions that are automated in push queues:

Scaling your workers

Your code needs to scale the number of workers based on processing volume. If your code does not handle scaling, you risk wasting computing resources if there are no tasks to process; you also risk latency if you have too many tasks to process.

Deleting the tasks

Your code also needs to explicitly delete tasks after processing. In push queues, App Engine deletes the tasks for you. If your worker does not delete pull queue tasks after processing, another worker will re-process the task. This wastes computing resources and risks errors if tasks are not idempotent.

Pull queues in the App Engine standard environment are created by setting a property in a configuration file called queue.yaml.

Pull queue workflow

Workers that process tasks from pull queues must be defined within a service that runs in the App Engine standard environment.

The workflow is as follows:

1. You create a pull queue, using queue.yaml.
2. You create tasks and add them to the queue.
3. The worker you have created leases the task, using TaskQueue.
4. App Engine sends task data to the worker in the lease response.
5. The worker processes the task. If the task fails to execute before the lease expires, the worker can modify the lease duration. If the lease expires, the task will be available to be leased to another worker.
6. After a task is processed successfully, the worker deletes it.

What's next

• Learn how to create pull queues.