BigQueryWriteClient(*, credentials: Optional[google.auth.credentials.Credentials] = None, transport: Optional[Union[str, google.cloud.bigquery_storage_v1beta2.services.big_query_write.transports.base.BigQueryWriteTransport]] = None, client_options: Optional[google.api_core.client_options.ClientOptions] = None, client_info: google.api_core.gapic_v1.client_info.ClientInfo = <google.api_core.gapic_v1.client_info.ClientInfo object>)
BigQuery Write API. The Write API can be used to write data to BigQuery.
Properties
transport
Returns the transport used by the client instance.
Type | Description |
BigQueryWriteTransport | The transport used by the client instance. |
Methods
BigQueryWriteClient
BigQueryWriteClient(*, credentials: Optional[google.auth.credentials.Credentials] = None, transport: Optional[Union[str, google.cloud.bigquery_storage_v1beta2.services.big_query_write.transports.base.BigQueryWriteTransport]] = None, client_options: Optional[google.api_core.client_options.ClientOptions] = None, client_info: google.api_core.gapic_v1.client_info.ClientInfo = <google.api_core.gapic_v1.client_info.ClientInfo object>)
Instantiates the big query write client.
Name | Description |
credentials |
Optional[google.auth.credentials.Credentials]
The authorization credentials to attach to requests. These credentials identify the application to the service; if none are specified, the client will attempt to ascertain the credentials from the environment. |
transport |
Union[str, BigQueryWriteTransport]
The transport to use. If set to None, a transport is chosen automatically. |
client_options |
google.api_core.client_options.ClientOptions
Custom options for the client. It won't take effect if a |
client_info |
google.api_core.gapic_v1.client_info.ClientInfo
The client info used to send a user-agent string along with API requests. If |
Type | Description |
google.auth.exceptions.MutualTLSChannelError | If mutual TLS transport creation failed for any reason. |
__exit__
__exit__(type, value, traceback)
Releases underlying transport's resources.
append_rows
append_rows(requests: Optional[Iterator[google.cloud.bigquery_storage_v1beta2.types.storage.AppendRowsRequest]] = None, *, retry: Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault] = <_MethodDefault._DEFAULT_VALUE: <object object>>, timeout: Optional[float] = None, metadata: Sequence[Tuple[str, str]] = ())
Appends data to the given stream.
If offset
is specified, the offset
is checked against
the end of stream. The server returns OUT_OF_RANGE
in
AppendRowsResponse
if an attempt is made to append to an
offset beyond the current end of the stream or
ALREADY_EXISTS
if user provids an offset
that has
already been written to. User can retry with adjusted offset
within the same RPC stream. If offset
is not specified,
append happens at the end of the stream.
The response contains the offset at which the append happened.
Responses are received in the same order in which requests are
sent. There will be one response for each successful request. If
the offset
is not set in response, it means append didn't
happen due to some errors. If one request fails, all the
subsequent requests will also fail until a success request is
made again.
If the stream is of PENDING
type, data will only be
available for read operations after the stream is committed.
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import bigquery_storage_v1beta2
def sample_append_rows():
# Create a client
client = bigquery_storage_v1beta2.BigQueryWriteClient()
# Initialize request argument(s)
request = bigquery_storage_v1beta2.AppendRowsRequest(
write_stream="write_stream_value",
)
# This method expects an iterator which contains
# 'bigquery_storage_v1beta2.AppendRowsRequest' objects
# Here we create a generator that yields a single `request` for
# demonstrative purposes.
requests = [request]
def request_generator():
for request in requests:
yield request
# Make the request
stream = client.append_rows(requests=request_generator())
# Handle the response
for response in stream:
print(response)
Name | Description |
requests |
Iterator[google.cloud.bigquery_storage_v1beta2.types.AppendRowsRequest]
The request object iterator. Request message for |
retry |
google.api_core.retry.Retry
Designation of what errors, if any, should be retried. |
timeout |
float
The timeout for this request. |
metadata |
Sequence[Tuple[str, str]]
Strings which should be sent along with the request as metadata. |
Type | Description |
Iterable[google.cloud.bigquery_storage_v1beta2.types.AppendRowsResponse] | Response message for AppendRows. |
batch_commit_write_streams
batch_commit_write_streams(request: Optional[Union[google.cloud.bigquery_storage_v1beta2.types.storage.BatchCommitWriteStreamsRequest, dict]] = None, *, parent: Optional[str] = None, retry: Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault] = <_MethodDefault._DEFAULT_VALUE: <object object>>, timeout: Optional[float] = None, metadata: Sequence[Tuple[str, str]] = ())
Atomically commits a group of PENDING
streams that belong to
the same parent
table. Streams must be finalized before
commit and cannot be committed multiple times. Once a stream is
committed, data in the stream becomes available for read
operations.
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import bigquery_storage_v1beta2
def sample_batch_commit_write_streams():
# Create a client
client = bigquery_storage_v1beta2.BigQueryWriteClient()
# Initialize request argument(s)
request = bigquery_storage_v1beta2.BatchCommitWriteStreamsRequest(
parent="parent_value",
write_streams=['write_streams_value1', 'write_streams_value2'],
)
# Make the request
response = client.batch_commit_write_streams(request=request)
# Handle the response
print(response)
Name | Description |
request |
Union[google.cloud.bigquery_storage_v1beta2.types.BatchCommitWriteStreamsRequest, dict]
The request object. Request message for |
parent |
str
Required. Parent table that all the streams should belong to, in the form of |
retry |
google.api_core.retry.Retry
Designation of what errors, if any, should be retried. |
timeout |
float
The timeout for this request. |
metadata |
Sequence[Tuple[str, str]]
Strings which should be sent along with the request as metadata. |
Type | Description |
google.cloud.bigquery_storage_v1beta2.types.BatchCommitWriteStreamsResponse | Response message for BatchCommitWriteStreams. |
common_billing_account_path
common_billing_account_path(billing_account: str)
Returns a fully-qualified billing_account string.
common_folder_path
common_folder_path(folder: str)
Returns a fully-qualified folder string.
common_location_path
common_location_path(project: str, location: str)
Returns a fully-qualified location string.
common_organization_path
common_organization_path(organization: str)
Returns a fully-qualified organization string.
common_project_path
common_project_path(project: str)
Returns a fully-qualified project string.
create_write_stream
create_write_stream(request: Optional[Union[google.cloud.bigquery_storage_v1beta2.types.storage.CreateWriteStreamRequest, dict]] = None, *, parent: Optional[str] = None, write_stream: Optional[google.cloud.bigquery_storage_v1beta2.types.stream.WriteStream] = None, retry: Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault] = <_MethodDefault._DEFAULT_VALUE: <object object>>, timeout: Optional[float] = None, metadata: Sequence[Tuple[str, str]] = ())
Creates a write stream to the given table. Additionally, every table has a special COMMITTED stream named '_default' to which data can be written. This stream doesn't need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import bigquery_storage_v1beta2
def sample_create_write_stream():
# Create a client
client = bigquery_storage_v1beta2.BigQueryWriteClient()
# Initialize request argument(s)
request = bigquery_storage_v1beta2.CreateWriteStreamRequest(
parent="parent_value",
)
# Make the request
response = client.create_write_stream(request=request)
# Handle the response
print(response)
Name | Description |
request |
Union[google.cloud.bigquery_storage_v1beta2.types.CreateWriteStreamRequest, dict]
The request object. Request message for |
parent |
str
Required. Reference to the table to which the stream belongs, in the format of |
write_stream |
google.cloud.bigquery_storage_v1beta2.types.WriteStream
Required. Stream to be created. This corresponds to the |
retry |
google.api_core.retry.Retry
Designation of what errors, if any, should be retried. |
timeout |
float
The timeout for this request. |
metadata |
Sequence[Tuple[str, str]]
Strings which should be sent along with the request as metadata. |
Type | Description |
google.cloud.bigquery_storage_v1beta2.types.WriteStream | Information about a single stream that gets data inside the storage system. |
finalize_write_stream
finalize_write_stream(request: Optional[Union[google.cloud.bigquery_storage_v1beta2.types.storage.FinalizeWriteStreamRequest, dict]] = None, *, name: Optional[str] = None, retry: Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault] = <_MethodDefault._DEFAULT_VALUE: <object object>>, timeout: Optional[float] = None, metadata: Sequence[Tuple[str, str]] = ())
Finalize a write stream so that no new data can be appended to the stream. Finalize is not supported on the '_default' stream.
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import bigquery_storage_v1beta2
def sample_finalize_write_stream():
# Create a client
client = bigquery_storage_v1beta2.BigQueryWriteClient()
# Initialize request argument(s)
request = bigquery_storage_v1beta2.FinalizeWriteStreamRequest(
name="name_value",
)
# Make the request
response = client.finalize_write_stream(request=request)
# Handle the response
print(response)
Name | Description |
request |
Union[google.cloud.bigquery_storage_v1beta2.types.FinalizeWriteStreamRequest, dict]
The request object. Request message for invoking |
name |
str
Required. Name of the stream to finalize, in the form of |
retry |
google.api_core.retry.Retry
Designation of what errors, if any, should be retried. |
timeout |
float
The timeout for this request. |
metadata |
Sequence[Tuple[str, str]]
Strings which should be sent along with the request as metadata. |
Type | Description |
google.cloud.bigquery_storage_v1beta2.types.FinalizeWriteStreamResponse | Response message for FinalizeWriteStream. |
flush_rows
flush_rows(request: Optional[Union[google.cloud.bigquery_storage_v1beta2.types.storage.FlushRowsRequest, dict]] = None, *, write_stream: Optional[str] = None, retry: Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault] = <_MethodDefault._DEFAULT_VALUE: <object object>>, timeout: Optional[float] = None, metadata: Sequence[Tuple[str, str]] = ())
Flushes rows to a BUFFERED stream. If users are appending rows to BUFFERED stream, flush operation is required in order for the rows to become available for reading. A Flush operation flushes up to any previously flushed offset in a BUFFERED stream, to the offset specified in the request. Flush is not supported on the _default stream, since it is not BUFFERED.
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import bigquery_storage_v1beta2
def sample_flush_rows():
# Create a client
client = bigquery_storage_v1beta2.BigQueryWriteClient()
# Initialize request argument(s)
request = bigquery_storage_v1beta2.FlushRowsRequest(
write_stream="write_stream_value",
)
# Make the request
response = client.flush_rows(request=request)
# Handle the response
print(response)
Name | Description |
request |
Union[google.cloud.bigquery_storage_v1beta2.types.FlushRowsRequest, dict]
The request object. Request message for |
write_stream |
str
Required. The stream that is the target of the flush operation. This corresponds to the |
retry |
google.api_core.retry.Retry
Designation of what errors, if any, should be retried. |
timeout |
float
The timeout for this request. |
metadata |
Sequence[Tuple[str, str]]
Strings which should be sent along with the request as metadata. |
Type | Description |
google.cloud.bigquery_storage_v1beta2.types.FlushRowsResponse | Respond message for FlushRows. |
from_service_account_file
from_service_account_file(filename: str, *args, **kwargs)
Creates an instance of this client using the provided credentials file.
Name | Description |
filename |
str
The path to the service account private key json file. |
Type | Description |
BigQueryWriteClient | The constructed client. |
from_service_account_info
from_service_account_info(info: dict, *args, **kwargs)
Creates an instance of this client using the provided credentials info.
Name | Description |
info |
dict
The service account private key info. |
Type | Description |
BigQueryWriteClient | The constructed client. |
from_service_account_json
from_service_account_json(filename: str, *args, **kwargs)
Creates an instance of this client using the provided credentials file.
Name | Description |
filename |
str
The path to the service account private key json file. |
Type | Description |
BigQueryWriteClient | The constructed client. |
get_mtls_endpoint_and_cert_source
get_mtls_endpoint_and_cert_source(
client_options: Optional[google.api_core.client_options.ClientOptions] = None,
)
Return the API endpoint and client cert source for mutual TLS.
The client cert source is determined in the following order:
(1) if GOOGLE_API_USE_CLIENT_CERTIFICATE
environment variable is not "true", the
client cert source is None.
(2) if client_options.client_cert_source
is provided, use the provided one; if the
default client cert source exists, use the default one; otherwise the client cert
source is None.
The API endpoint is determined in the following order:
(1) if client_options.api_endpoint
if provided, use the provided one.
(2) if GOOGLE_API_USE_CLIENT_CERTIFICATE
environment variable is "always", use the
default mTLS endpoint; if the environment variabel is "never", use the default API
endpoint; otherwise if client cert source exists, use the default mTLS endpoint, otherwise
use the default API endpoint.
More details can be found at https://google.aip.dev/auth/4114.
Name | Description |
client_options |
google.api_core.client_options.ClientOptions
Custom options for the client. Only the |
Type | Description |
google.auth.exceptions.MutualTLSChannelError | If any errors happen. |
Type | Description |
Tuple[str, Callable[[], Tuple[bytes, bytes]]] | returns the API endpoint and the client cert source to use. |
get_write_stream
get_write_stream(request: Optional[Union[google.cloud.bigquery_storage_v1beta2.types.storage.GetWriteStreamRequest, dict]] = None, *, name: Optional[str] = None, retry: Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault] = <_MethodDefault._DEFAULT_VALUE: <object object>>, timeout: Optional[float] = None, metadata: Sequence[Tuple[str, str]] = ())
Gets a write stream.
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import bigquery_storage_v1beta2
def sample_get_write_stream():
# Create a client
client = bigquery_storage_v1beta2.BigQueryWriteClient()
# Initialize request argument(s)
request = bigquery_storage_v1beta2.GetWriteStreamRequest(
name="name_value",
)
# Make the request
response = client.get_write_stream(request=request)
# Handle the response
print(response)
Name | Description |
request |
Union[google.cloud.bigquery_storage_v1beta2.types.GetWriteStreamRequest, dict]
The request object. Request message for |
name |
str
Required. Name of the stream to get, in the form of |
retry |
google.api_core.retry.Retry
Designation of what errors, if any, should be retried. |
timeout |
float
The timeout for this request. |
metadata |
Sequence[Tuple[str, str]]
Strings which should be sent along with the request as metadata. |
Type | Description |
google.cloud.bigquery_storage_v1beta2.types.WriteStream | Information about a single stream that gets data inside the storage system. |
parse_common_billing_account_path
parse_common_billing_account_path(path: str)
Parse a billing_account path into its component segments.
parse_common_folder_path
parse_common_folder_path(path: str)
Parse a folder path into its component segments.
parse_common_location_path
parse_common_location_path(path: str)
Parse a location path into its component segments.
parse_common_organization_path
parse_common_organization_path(path: str)
Parse a organization path into its component segments.
parse_common_project_path
parse_common_project_path(path: str)
Parse a project path into its component segments.
parse_table_path
parse_table_path(path: str)
Parses a table path into its component segments.
parse_write_stream_path
parse_write_stream_path(path: str)
Parses a write_stream path into its component segments.
table_path
table_path(project: str, dataset: str, table: str)
Returns a fully-qualified table string.
write_stream_path
write_stream_path(project: str, dataset: str, table: str, stream: str)
Returns a fully-qualified write_stream string.