Use online prediction logging

For AutoML tabular models, AutoML image models, and custom-trained models, you can enable or disable prediction logs during model deployment or endpoint creation. This page explains the different types of prediction logs available, and how to enable or disable these logs.

Types of prediction logs

There are several types of prediction logs that you can use to get information from your prediction nodes:

  • Container logging, which logs the stdout and stderr streams from your prediction nodes to Cloud Logging. These logs are required for debugging.

    • On the v1 service endpoint, container logging is enabled by default. You can disable it when you deploy a model. You can also disable or enable logging when you mutate the deployed model.

    • On the v1beta1 service endpoint, container logging is disabled by default. You can enable it when you deploy a model. You can also disable or enable logging when you mutate the deployed model.

  • Access logging, which logs information like timestamp and latency for each request to Cloud Logging.

    On both the v1 and v1beta1 service endpoints, access logging is disabled by default. You can enable access logging when you deploy a model to an endpoint.

  • Request-response logging, which logs a sample of online prediction requests and responses to a BigQuery table.

    You can enable request-response logging by creating or patching the prediction endpoint.

You can enable or disable each type of log independently.

Prediction log settings

You can enable or disable online prediction logs when you create an endpoint, deploy a model to the endpoint, or mutate a deployed model.

To update the settings for access logs, you must undeploy your model, and then redeploy the model with your new settings. You can update the settings for container logs without re-deploying your model.

Online prediction at a high rate of queries per second (QPS) can produce a substantial number of logs, which are subject to Cloud Logging pricing. To estimate the pricing for your online prediction logs, see Estimating your bills for logging. To reduce this cost, you can disable prediction logging.

Enable and disable prediction logs

The following examples highlight where to modify the default log settings:

Console

When you deploy a model to an endpoint or create a new endpoint in the Google Cloud console, you can specify which types of prediction logs to enable in the Logging step. Select the checkboxes to enable Access logging or Container logging, or clear the checkboxes to disable these logs.

Use the REST API to update the settings for container logs.

Use the REST API to enable request-response logging. The Google Cloud console and gcloud CLI don't support request-response logging configuration.

To see more context about how to deploy models, read Deploy a model using the Google Cloud console.

gcloud

To change the default behavior for which logs are enabled in deployed models, add flags to your gcloud command:

v1 service endpoint

Run gcloud ai endpoints deploy-model:

gcloud ai endpoints deploy-model ENDPOINT_ID\
  --region=LOCATION \
  --model=MODEL_ID \
  --display-name=DEPLOYED_MODEL_NAME \
  --machine-type=MACHINE_TYPE \
  --accelerator=count=2,type=nvidia-tesla-t4 \
  --disable-container-logging \
  --enable-access-logging

v1beta1 service endpoint

Run gcloud beta ai endpoints deploy-model:

gcloud beta ai endpoints deploy-model ENDPOINT_ID\
  --region=LOCATION \
  --model=MODEL_ID \
  --display-name=DEPLOYED_MODEL_NAME \
  --machine-type=MACHINE_TYPE \
  --accelerator=count=2,type=nvidia-tesla-t4 \
  --enable-access-logging \
  --enable-container-logging

Use the REST API to update the settings for container logs.

Use the REST API to enable request-response logging. The Google Cloud console and gcloud CLI don't support request-response logging configuration.

To see more context about how to deploy models, read Deploy a model using the Vertex AI API.

REST

To change the default behavior for which logs are enabled in deployed models, set the relevant fields to True:

v1 service endpoint

To disable container logging, set the disableContainerLogging field to True when you call either projects.locations.endpoints.deployModel or projects.locations.endpoints.mutateDeployedModel.

To enable access logging, set enableAccessLogging to True when deploying your model with projects.locations.endpoints.deployModel.

v1beta1 service endpoint

To enable container logging, set the enableContainerLogging field to True when you call either projects.locations.endpoints.deployModel or projects.locations.endpoints.mutateDeployedModel.

To enable access logging, set enableAccessLogging to True when deploying your model with projects.locations.endpoints.deployModel.

To see more context about how to deploy models, read Deploy a model using the Vertex AI API.

Request-response logging

You can only enable request-response logging when you send a create an endpoint using projects.locations.endpoints.create or patch an existing endpoint using projects.locations.endpoints.patch.

Request-response logging is done at the endpoint level, so requests sent to any deployed models under the same endpoint are logged.

When you create or patch an endpoint, populate the predictRequestResponseLoggingConfig field of the Endpoint resource with the following entries:

  • enabled: set as True to enable request-response logging.

  • samplingPercentage: a number between 0 or 1 defining the fraction of requests to log. For example, set this value to 1 in order to log all requests or to 0.1 to log 10% of requests.

  • BigQueryDestination: the BigQuery table to be used for logging. If you only specify a project name, a new dataset is created with the name logging_ENDPOINT_DISPLAY_NAME_ENDPOINT_ID, where ENDPOINT_DISPLAY_NAME follows the BigQuery naming rules. If you don't specify a table name, a new table is created with the name request_response_logging.

    The schema for the BigQuery table should look like the following:

    Field nameTypeMode
    endpointSTRINGNULLABLE
    deployed_model_idSTRINGNULLABLE
    logging_timeTIMESTAMPNULLABLE
    request_idNUMERICNULLABLE
    request_payloadSTRINGREPEATED
    response_payloadSTRINGREPEATED

The following is an example configuration:

{
   "predict_request_response_logging_config": {
     "enabled": true,
     "sampling_rate": 0.5,
     "bigquery_destination": {
       "output_uri": "bq://PROJECT_ID.DATASET_NAME.TABLE_NAME"
     }
   }
}

Request-response logging and Model Monitoring v1

Request-response logging and Model Monitoring v1 use the same BigQuery table on the backend to log incoming requests. To prevent unexpected changes to this BigQuery table, the following limitations are enforced when using both features at the same time:

  • If an endpoint has Model Monitoring enabled, you can't enable request-response logging for the same endpoint.

  • If you enable request-response logging and then Model Monitoring on the same endpoint, you won't be able to change the request-response logging configuration.

What's next