Class BulkWriter (2.20.2) 

BulkWriter(
    client: typing.Optional[BaseClient] = None,
    options: typing.Optional[BulkWriterOptions] = None,
)

Accumulate and efficiently save large amounts of document write operations to the server.

BulkWriter can handle large data migrations or updates, buffering records in memory and submitting them to the server in batches of 20.

The submission of batches is internally parallelized with a ThreadPoolExecutor, meaning end developers do not need to manage an event loop or worry about asyncio to see parallelization speed ups (which can easily 10x throughput). Because of this, there is no companion AsyncBulkWriter class, as is usually seen with other utility classes.

Usage:

# Instantiate the BulkWriter. This works from either `Client` or
# `AsyncClient`.
db = firestore.Client()
bulk_writer = db.bulk_writer()

# Attach an optional success listener to be called once per document.
bulk_writer.on_write_result(
    lambda reference, result, bulk_writer: print(f'Saved {reference._document_path}')
)

# Queue an arbitrary amount of write operations.
# Assume `my_new_records` is a list of (DocumentReference, dict,)
# tuple-pairs that you supply.

reference: DocumentReference
data: dict
for reference, data in my_new_records:
    bulk_writer.create(reference, data)

# Block until all pooled writes are complete.
bulk_writer.flush()

Parameter
Name Description
client Client

The client that created this BulkWriter.

Methods

close

close()

Block until all pooled write operations are complete and then reject any further write operations.

create

create(
    reference: google.cloud.firestore_v1.base_document.BaseDocumentReference,
    document_data: typing.Dict,
    attempts: int = 0,
) -> None

Adds a create pb to the in-progress batch.

If the in-progress batch already contains a write operation involving this document reference, the batch will be sealed and added to the commit queue, and a new batch will be created with this operation as its first entry.

If this create operation results in the in-progress batch reaching full capacity, then the batch will be similarly added to the commit queue, and a new batch will be created for future operations.

Parameters
Name Description
reference BaseDocumentReference

Pointer to the document that should be created.
document_data dict

Raw data to save to the server.

delete

delete(
    reference: google.cloud.firestore_v1.base_document.BaseDocumentReference,
    option: typing.Optional[google.cloud.firestore_v1._helpers.WriteOption] = None,
    attempts: int = 0,
) -> None

Adds a delete pb to the in-progress batch.

If the in-progress batch already contains a write operation involving this document reference, the batch will be sealed and added to the commit queue, and a new batch will be created with this operation as its first entry.

If this delete operation results in the in-progress batch reaching full capacity, then the batch will be similarly added to the commit queue, and a new batch will be created for future operations.

Parameters
Name Description
reference BaseDocumentReference

Pointer to the document that should be created.
option WriteOption

Optional flag to modify the nature of this write.

flush

flush()

Block until all pooled write operations are complete and then resume accepting new write operations.

on_batch_result

on_batch_result(
    callback: typing.Optional[
        typing.Callable[
            [
                google.cloud.firestore_v1.bulk_batch.BulkWriteBatch,
                google.cloud.firestore_v1.types.firestore.BatchWriteResponse,
                google.cloud.firestore_v1.bulk_writer.BulkWriter,
            ],
            None,
        ]
    ],
) -> None

Sets a callback that will be invoked once for every successful batch.

on_write_error

on_write_error(
    callback: typing.Optional[
        typing.Callable[
            [
                google.cloud.firestore_v1.bulk_writer.BulkWriteFailure,
                google.cloud.firestore_v1.bulk_writer.BulkWriter,
            ],
            bool,
        ]
    ],
) -> None

Sets a callback that will be invoked once for every batch that contains an error.

on_write_result

on_write_result(
    callback: typing.Optional[
        typing.Callable[
            [
                google.cloud.firestore_v1.base_document.BaseDocumentReference,
                google.cloud.firestore_v1.types.write.WriteResult,
                google.cloud.firestore_v1.bulk_writer.BulkWriter,
            ],
            None,
        ]
    ],
) -> None

Sets a callback that will be invoked once for every successful operation.

set

set(
    reference: google.cloud.firestore_v1.base_document.BaseDocumentReference,
    document_data: typing.Dict,
    merge: typing.Union[bool, list] = False,
    attempts: int = 0,
) -> None

Adds a set pb to the in-progress batch.

If the in-progress batch already contains a write operation involving this document reference, the batch will be sealed and added to the commit queue, and a new batch will be created with this operation as its first entry.

If this set operation results in the in-progress batch reaching full capacity, then the batch will be similarly added to the commit queue, and a new batch will be created for future operations.

Parameters
Name Description
reference BaseDocumentReference

Pointer to the document that should be created.
document_data dict

Raw data to save to the server.
merge bool

Whether or not to completely overwrite any existing data with the supplied data.

update

update(
    reference: google.cloud.firestore_v1.base_document.BaseDocumentReference,
    field_updates: dict,
    option: typing.Optional[google.cloud.firestore_v1._helpers.WriteOption] = None,
    attempts: int = 0,
) -> None

Adds an update pb to the in-progress batch.

If the in-progress batch already contains a write operation involving this document reference, the batch will be sealed and added to the commit queue, and a new batch will be created with this operation as its first entry.

If this update operation results in the in-progress batch reaching full capacity, then the batch will be similarly added to the commit queue, and a new batch will be created for future operations.

Parameters
Name Description
reference BaseDocumentReference

Pointer to the document that should be created.
field_updates dict

Key paths to specific nested data that should be upated.
option WriteOption

Optional flag to modify the nature of this write.