Manually log data to an experiment run

Summary metrics

Summary metrics are single value scalar metrics stored next to time series metrics and represent a final summary of an experiment run.

One example use case is early stopping where a patience configuration allows continued training but the candidate model is restored from an earlier step and the metrics calculated for the model at that step would be represented as a summary metric because the latest time series metric is not representative of the restored model. The log_metrics API for summary metrics is used for this purpose.

from typing import Dict

from google.cloud import aiplatform


def log_metrics_sample(
    experiment_name: str,
    run_name: str,
    metrics: Dict[str, float],
    project: str,
    location: str,
):
    aiplatform.init(experiment=experiment_name, project=project, location=location)

    aiplatform.start_run(run=run_name)

    aiplatform.log_metrics(metrics)

Time series metrics

To log time series metrics, Vertex AI Experiments requires a backing Vertex AI TensorBoard instance.

Assign backing Vertex AI TensorBoard resource for Time Series Metric Logging.

All metrics logged through log_time_series_metrics are stored as time series metrics. Vertex AI TensorBoard is the backing time series metric store.

The experiment_tensorboard can be set at both the experiment and experiment run levels. Setting the experiment_tensorboard at the run level overrides the setting at the experiment level. Once the experiment_tensorboard is set in a run, the run's experiment_tensorboard can't be changed.

• Set experiment_tensorboard at experiment level:

    aiplatform.init(experiment='my-experiment',
                 experiment_tensorboard='projects/.../tensorboard/my-tb-resource')

• Set experiment_tensorboard at run level: Note: Overrides setting at experiment level.

    aiplatform.start_run(run_name='my-other-run',
                      tensorboard='projects/.../.../other-resource')
  aiplatform.log_time_series_metrics(...)

from typing import Dict, Optional

from google.cloud import aiplatform
from google.protobuf import timestamp_pb2


def log_time_series_metrics_sample(
    experiment_name: str,
    run_name: str,
    metrics: Dict[str, float],
    step: Optional[int],
    wall_time: Optional[timestamp_pb2.Timestamp],
    project: str,
    location: str,
):
    aiplatform.init(experiment=experiment_name, project=project, location=location)

    aiplatform.start_run(run=run_name, resume=True)

    aiplatform.log_time_series_metrics(metrics=metrics, step=step, wall_time=wall_time)

Step and walltime

The log_time_series_metrics API optionally accepts step and walltime.

• step: Optional. Step index of this data point within the run. If not provided, an increment over the latest step among all time series metrics already logged is used. If the step exists for any of the provided metric keys, the step is overwritten.
• wall_time: Optional. The seconds after epoch of the logged metric. If this is not provided the default is to Python's time.time.

For example:

aiplatform.log_time_series_metrics({"mse": 2500.00, "rmse": 50.00})

Log to a specific step

aiplatform.log_time_series_metrics({"mse": 2500.00, "rmse": 50.00}, step=8)

Include wall_time

aiplatform.log_time_series_metrics({"mse": 2500.00, "rmse": 50.00}, step=10)

Parameters

Parameters are keyed input values that configure a run, regulate the behavior of the run, and affect the results of the run. Examples include learning rate, dropout rate, and number of training steps. Log parameters using the log_params method.

from typing import Dict, Union

from google.cloud import aiplatform


def log_params_sample(
    experiment_name: str,
    run_name: str,
    params: Dict[str, Union[float, int, str]],
    project: str,
    location: str,
):
    aiplatform.init(experiment=experiment_name, project=project, location=location)

    aiplatform.start_run(run=run_name, resume=True)

    aiplatform.log_params(params)

aiplatform.log_params({"learning_rate": 0.01, "n_estimators": 10})

Classification metrics

In addition to summary metrics and time series metrics, confusion matrices and ROC curves are commonly used metrics. They can be logged to Vertex AI Experiments using the log_classification_metrics API.

from typing import List, Optional

from google.cloud import aiplatform


def log_classification_metrics_sample(
    experiment_name: str,
    run_name: str,
    project: str,
    location: str,
    labels: Optional[List[str]] = None,
    matrix: Optional[List[List[int]]] = None,
    fpr: Optional[List[float]] = None,
    tpr: Optional[List[float]] = None,
    threshold: Optional[List[float]] = None,
    display_name: Optional[str] = None,
) -> None:
    aiplatform.init(experiment=experiment_name, project=project, location=location)

    aiplatform.start_run(run=run_name, resume=True)

    aiplatform.log_classification_metrics(
        labels=labels,
        matrix=matrix,
        fpr=fpr,
        tpr=tpr,
        threshold=threshold,
        display_name=display_name,
    )