View log messages for Application Load Balancer plugins

This page describes logging for Application Load Balancer plugins. Logging is possible from both the plugin perspective and the load balancer perspective.

For a product overview, see Service Extensions overview.

Log messages

Logs that are pertinent to Service Extensions are in one of the following categories:

Plugin log messages
Generated by a logging call, such as info!(...) for Rust or LOG_INFO for C++. Service Extensions exports these log messages to Cloud Logging. You can log request and response headers and any actions that the plugin has taken. You can view these messages by using the networkservices.googleapis.com service.
Cloud Load Balancing log messages
You can view these messages by using the loadbalancing.googleapis.com service.

Logging from the plugin perspective

This section describes Service Extensions logging from the plugin perspective.

Enable logging for a plugin

Service Extensions supports generating log messages during the execution of your plugin. Recording logs is disabled by default.

To record logs for a plugin, enable it while creating the plugin or updating it.

To enable logging for an existing plugin, use the gcloud service-extensions wasm-plugins update command:

gcloud service-extensions wasm-plugins update WASM_PLUGIN \
    --log-config=[LOG_CONFIG,...]

Replace the following:

  • WASM_PLUGIN: the ID or the fully qualified name of the plugin

  • LOG_CONFIG: logging options for the plugin. To enable logging, set the enable option to true. Then, specify the following details:

    • sample-rate: the sampling rate of activity logs as a value between 0 and 1. The value 0 indicates that log messages are not stored. The default value 1 indicates that all log messages are stored. A floating point value between 0.0 and 1.0 indicates that a percentage of log messages is stored.

    • min-log-level: the minimum severity level of plugin log messages to be exported to Cloud Logging. The default value is INFO.

After you enable logging for the plugin, you can view the messages emitted by logging statements in plugin code in Cloud Logging.

To view logs, in the Google Cloud console, go to the Logs Explorer page.

Log messages for plugins

Plugin log records are annotated with the following contextual information:

  • Standard log annotations, such as timestamp and log level.
  • The identity of the plugin that generated the message.
  • The plugin callback in which the log message was generated.
  • A requestId trace identifier that helps determine the request log that a log message is associated with.

View log messages for plugins

Logs can be viewed by building queries in Logs Explorer.

You can view plugin logs in the following formats:

  • As standalone Service Extensions logs.

    In this view, each plugin log message is recorded in its own log record and isn't automatically associated with request log information.

    These log messages are in the networkservices.googleapis.com/wasm_plugin_activity log with the resource type networkservices.googleapis.com/WasmPluginVersion.

    The system might also add informational log messages to this log. For example, if there's a plugin failure when a plugin invocation exceeds CPU or memory limits, a message of ERROR severity is logged. Such messages can also be seen in Error Reporting.

Plugin log samples

Consider a sample Service Extensions log entry. The value of message is passed to the plugin's LOG_INFO call. The severity value depends on the log level used in the plugin log call. In the labels section, the value of networkservices.googleapis.com/operation is HTTP_REQUEST_HEADER, which indicates that the logged operation is the on_http_request_headers plugin callback.

{
  "insertId": "65224aac-0000-24bd-a0e1-582429bd544c@a1",
  "jsonPayload": {
    "@type": "type.googleapis.com/google.cloud.networkservices.logging.v1.WasmPluginLogEntry",
    "metroIataCode": "ber",
    "proxyRegionCode": "DE",
    "message": "[add_header_plugin.cc:26]::onRequestHeaders() AddHeaderStreamContext::onRequestHeaders called",
    "requestId": "effc0311-6716-431b-9e2a-7586835fdff1"
  },
  "resource": {
    "type": "networkservices.googleapis.com/WasmPluginVersion",
    "labels": {
      "plugin_version": "prod-1",
      "resource_container": "projects/123456789",
      "location": "global",
      "plugin_name": "add-headers-plugin-prod-resource"
    }
  },
  "timestamp": "2023-05-10T03:05:43.317015458Z",
  "severity": "INFO",
  "labels": {
    "networkservices.googleapis.com/operation": "HTTP_REQUEST_HEADERS"
  },
  "logName": "projects/123456789/logs/networkservices.googleapis.com%2Fwasm_plugin_activity",
  "trace": "projects/123456789/traces/effc0311-6716-431b-9e2a-7586835fdff1",
  "receiveTimestamp": "2023-05-10T03:05:44.207265284Z"
}

Logging from the load balancer perspective

This section describes Service Extensions logging from the load balancer perspective.

Enable logging on a backend service

You can enable logging for Application Load Balancer plugins while creating a service by enabling logging on the backend service that's the target of a request.

To enable logging for the target backend service for a regional internal Application Load Balancer, use the gcloud compute backend-services update command.

gcloud compute backend-services update BACKEND_SERVICE \
    --enable-logging \
    --logging-sample-rate=RATE \
    --region=REGION \
    --logging-optional=LOGGING_OPTIONAL_MODE \
    --logging-optional-fields=OPTIONAL_FIELDS

Replace the following:

  • BACKEND_SERVICE: the name of the backend service
  • RATE: a value from 0.0 through 1.0, where 0.0 means no requests are logged and 1.0 means 100% of requests are logged. The default value is 1.0. This setting is effective only when used with the enable-logging parameter. When you omit enable-logging, logging is disabled.
  • REGION: the region of the backend

  • LOGGING_OPTIONAL_MODE: enables logging for optional fields in one of these modes:

    • INCLUDE_ALL_OPTIONAL includes all optional fields.

    • EXCLUDE_ALL_OPTIONAL (default) excludes all optional fields.

    • CUSTOM includes a custom list of optional fields.

  • OPTIONAL_FIELDS: a comma-separated list of optional fields when you select the CUSTOM mode

For more information, see the Monitor and troubleshoot pages of Application Load Balancer documentation, such as Internal Application Load Balancer logging and monitoring.

After you enable logging on the backend service, each HTTP or HTTPS request is logged by using Cloud Logging.

To view logs, in the Google Cloud console, go to the Logs Explorer page.

Log messages for a backend service

In general, Application Load Balancer log entries contain information useful for monitoring and debugging your HTTP or HTTPS traffic. Log entries contain the following types of information:

  • Information shown in most Google Cloud logs, such as severity, project ID, project number, and timestamp as described in the LogEntry.
  • HttpRequest log fields.

Request logs for HTTP and HTTPS load balancers contain a service_extension_info object in the load balancer log entry JSON payload with the following information:

Field Type Description
backend_target_name string Name of the plugin referenced by the extension
backend_target_type string Type of the backend target; this must be WASM_PLUGIN.
chain string Name of the extension chain within the service extension resource that matches the request
extension string Name of the extension within the extension chain
per_processing_request_info array Information about each plugin invocation
per_processing_request_info[].event_type enum The event type of ProcessingRequest. Can be one of these: REQUEST_HEADERS, REQUEST_BODY, RESPONSE_HEADERS, or RESPONSE_BODY.
resource string Name of the extension resource

Limitations for logging

This section describes limitations pertaining to logging for Application Load Balancer plugins.

  • For sampling rate, the option used for plugins is sample-rate and that for callouts is logging-sample-rate. These are independent sampling mechanisms. Setting them both to 0.01 elicits two separate 1% sets of requests.

  • From the plugins perspective, plugins are limited to logging up to 16 KiB of payload data per client HTTP request. This amount is divided across multiple logging calls that are associated with a given HTTP request. The limit applies only to log message text, not to additional metadata added to the log record by Service Extensions.

    For example, if an on_http_request_headers callback makes two logging calls with 4 KiB messages each, and then an on_http_response_headers callback attempts to make three logging calls with 4 KiB messages each for the same HTTP request, the third logging message is dropped. A log message is added to record the number of plugin-generated log messages that were dropped.

What's next