Class StreamingDataFrame (1.27.0)

StreamingDataFrame(df: bigframes.dataframe.DataFrame, *, create_key=0)

Two-dimensional, size-mutable, potentially heterogeneous tabular data.

Data structure also contains labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure.

Methods

__getitem__

__getitem__(*args, **kwargs)

Gets the specified column(s) from the StreamingDataFrame.

Parameter
Name Description
key index

Index or list of indices. It can be a column label, a list of column labels, a Series of booleans or a pandas Index of desired column labels

Returns
Type Description
bigframes.pandas.Series or Any Value(s) at the requested index(es).

__repr__

__repr__(*args, **kwargs)

Converts a StreamingDataFrame to a string. Calls to_pandas.

Only represents the first <xref uid="bigframes.options">bigframes.options</xref>.display.max_rows.

__setitem__

__setitem__(*args, **kwargs)

Modify or insert a column into the StreamingDataFrame.

Parameters
Name Description
key column index

It can be a new column to be inserted, or an existing column to be modified.

value scalar or Series

Value to be assigned to the column

rename

rename(*args, **kwargs)

Rename columns.

Dict values must be unique (1-to-1). Labels not contained in a dict will be left as-is. Extra labels listed don't throw an error.

Parameter
Name Description
columns Mapping

Dict-like from old column labels to new column labels.

Exceptions
Type Description
KeyError If any of the labels is not found.
Returns
Type Description
bigframes.pandas.DataFrame StreamingDataFrame with the renamed axis labels.

to_bigtable

to_bigtable(
    *,
    instance: str,
    table: str,
    service_account_email: typing.Optional[str] = None,
    app_profile: typing.Optional[str] = None,
    truncate: bool = False,
    overwrite: bool = False,
    auto_create_column_families: bool = False,
    bigtable_options: typing.Optional[dict] = None,
    job_id: typing.Optional[str] = None,
    job_id_prefix: typing.Optional[str] = None
) -> google.cloud.bigquery.job.query.QueryJob

Export the StreamingDataFrame as a continue job and returns a QueryJob object for some management functionality.

This method requires an existing bigtable preconfigured to accept the continuous query export statement. For instructions on export to bigtable, see https://cloud.google.com/bigquery/docs/export-to-bigtable.

Parameters
Name Description
instance str

The name of the bigtable instance to export to.

table str

The name of the bigtable table to export to.

service_account_email str

Full name of the service account to run the continuous query. Example: accountname@projectname.gserviceaccounts.com If not provided, the user account will be used, but this limits the lifetime of the continuous query.

app_profile str, default None

The bigtable app profile to export to. If None, no app profile will be used.

truncate bool, default False

The export truncate option, see https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#bigtable_export_option

overwrite bool, default False

The export overwrite option, see https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#bigtable_export_option

auto_create_column_families bool, default False

The auto_create_column_families option, see https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#bigtable_export_option

bigtable_options dict, default None

The bigtable options dict, which will be converted to JSON using json.dumps, see https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#bigtable_export_option If None, no bigtable_options parameter will be passed.

job_id str, default None

If specified, replace the default job id for the query, see job_id parameter of https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_query

job_id_prefix str, default None

If specified, a job id prefix for the query, see job_id_prefix parameter of https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_query

Returns
Type Description
google.cloud.bigquery.QueryJob See https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.job.QueryJob The ongoing query job can be managed using this object. For example, the job can be cancelled or its error status can be examined.

to_pubsub

to_pubsub(
    *,
    topic: str,
    service_account_email: str,
    job_id: typing.Optional[str] = None,
    job_id_prefix: typing.Optional[str] = None
) -> google.cloud.bigquery.job.query.QueryJob

Export the StreamingDataFrame as a continue job and returns a QueryJob object for some management functionality.

This method requires an existing pubsub topic. For instructions on creating a pubsub topic, see https://cloud.google.com/pubsub/docs/samples/pubsub-quickstart-create-topic?hl=en

Note that a service account is a requirement for continuous queries exporting to pubsub.

Parameters
Name Description
topic str

The name of the pubsub topic to export to. For example: "taxi-rides"

service_account_email str

Full name of the service account to run the continuous query. Example: accountname@projectname.gserviceaccounts.com

job_id str, default None

If specified, replace the default job id for the query, see job_id parameter of https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_query

job_id_prefix str, default None

If specified, a job id prefix for the query, see job_id_prefix parameter of https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client#google_cloud_bigquery_client_Client_query

Returns
Type Description
google.cloud.bigquery.QueryJob See https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.job.QueryJob The ongoing query job can be managed using this object. For example, the job can be cancelled or its error status can be examined.