This page shows how to configure a user cluster for Google Distributed Cloud so that custom logs and metrics
from user applications are sent to Cloud Logging and Cloud Monitoring.
Enabling Logging and Monitoring for user applications
The configuration for Logging and Monitoring is
held in a Stackdriver object named stackdriver.
To enable the collection of custom metrics from an application, add the
prometheus.io/scrape: "true" annotation to the application's Service or
Deployment manifest.
To avoid garbage collection of metrics, we recommend setting the metrics scrape interval to one minute.
Running an example application
In this section, you create an application that writes custom logs and exposes
a custom metric.
Save the following Service and Deployment manifests to a file named
my-app.yaml. Notice that the Service has the annotation
prometheus.io/scrape: "true":
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-25 UTC."],[[["\u003cp\u003eThis guide demonstrates how to configure a Google Distributed Cloud user cluster to send custom logs and metrics from user applications to Cloud Logging and Cloud Monitoring.\u003c/p\u003e\n"],["\u003cp\u003eEnabling Logging and Monitoring for applications requires setting \u003ccode\u003eenableStackdriverForApplications\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e within the \u003ccode\u003estackdriver\u003c/code\u003e object, located in the \u003ccode\u003ekube-system\u003c/code\u003e namespace, using the provided \u003ccode\u003ekubectl\u003c/code\u003e command and example.\u003c/p\u003e\n"],["\u003cp\u003eTo collect custom metrics, you must add the \u003ccode\u003eprometheus.io/scrape: "true"\u003c/code\u003e annotation to the application's Service or Deployment manifest, with a suggested scrape interval of one minute to avoid garbage collection.\u003c/p\u003e\n"],["\u003cp\u003eAn example application is provided, including Service and Deployment manifests, to illustrate the process of writing custom logs and exposing a custom metric, which can be created with the \u003ccode\u003ekubectl\u003c/code\u003e apply command.\u003c/p\u003e\n"],["\u003cp\u003eYou can view application logs in the Google Cloud console's Logs Explorer by filtering for \u003ccode\u003eKubernetes Container\u003c/code\u003e, and using the provided filters, or via the \u003ccode\u003egcloud\u003c/code\u003e command, and you can view application metrics in the Metrics explorer by selecting the \u003ccode\u003eKubernetes Pod\u003c/code\u003e resource type and \u003ccode\u003eexternal/prometheus/example_monitoring_up\u003c/code\u003e metric.\u003c/p\u003e\n"]]],[],null,["# Application logging and monitoring\n\n\u003cbr /\u003e\n\nThis page shows how to configure a user cluster for Google Distributed Cloud so that custom logs and metrics\nfrom user applications are sent to Cloud Logging and Cloud Monitoring.\n\nEnabling Logging and Monitoring for user applications\n-----------------------------------------------------\n\nThe configuration for Logging and Monitoring is\nheld in a Stackdriver object named `stackdriver`.\n\n1. Open the `stackdriver` object for editing:\n\n ```\n kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG --namespace kube-system edit stackdriver stackdriver\n ```\n\n Replace \u003cvar translate=\"no\"\u003eUSER_CLUSTER_KUBECONFIG\u003c/var\u003e with the path of your user\n cluster kubeconfig file.\n2. Under `spec`, set `enableStackdriverForApplications` to `true`:\n\n ```\n apiVersion: addons.sigs.k8s.io/v1alpha1\n kind: Stackdriver\n metadata:\n name: stackdriver\n namespace: kube-system\n spec:\n projectID: ...\n clusterName: ...\n clusterLocation: ...\n proxyConfigSecretName: ...\n enableStackdriverForApplications: true\n enableVPC: ...\n optimizedMetrics: true\n ```\n3. Close the edited file.\n\nAnnotating workloads\n--------------------\n\nTo enable the collection of custom metrics from an application, add the\n`prometheus.io/scrape: \"true\"` annotation to the application's Service or\nDeployment manifest.\n\nTo avoid garbage collection of metrics, we recommend setting the metrics scrape interval to one minute.\n\nRunning an example application\n------------------------------\n\nIn this section, you create an application that writes custom logs and exposes\na custom metric.\n\n1. Save the following Service and Deployment manifests to a file named\n `my-app.yaml`. Notice that the Service has the annotation\n `prometheus.io/scrape: \"true\"`:\n\n ```\n kind: Service\n apiVersion: v1\n metadata:\n name: \"monitoring-example\"\n namespace: \"default\"\n annotations:\n prometheus.io/scrape: \"true\"\n spec:\n selector:\n app: \"monitoring-example\"\n ports:\n - name: http\n port: 9090\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: \"monitoring-example\"\n namespace: \"default\"\n labels:\n app: \"monitoring-example\"\n spec:\n replicas: 1\n selector:\n matchLabels:\n app: \"monitoring-example\"\n template:\n metadata:\n labels:\n app: \"monitoring-example\"\n spec:\n containers:\n - image: gcr.io/google-samples/prometheus-dummy-exporter:latest\n name: prometheus-example-exporter\n imagePullPolicy: Always\n command:\n - /bin/sh\n - -c\n - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090\n resources:\n requests:\n cpu: 100m\n ```\n2. Create the Deployment and the Service:\n\n ```\n kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml\n ```\n\nViewing application logs\n------------------------\n\n### Console\n\n1. Go to the Logs explorer in the Google Cloud console.\n\n [Go to the Logs explorer](https://console.cloud.google.com/logs/query)\n2. Click **Resource** . Under **ALL_RESOURCE_TYPES** , select `Kubernetes\n Container`.\n\n3. Under **CLUSTER_NAME**, select the name of your user cluster.\n\n4. Under **NAMESPACE_NAME** , select `default`.\n\n5. Click **Add** and then click **Run Query**.\n\n6. Under **Query results** , you can see log entries from the\n `monitoring-example` Deployment. For example:\n\n ```\n {\n \"textPayload\": \"2020/11/14 01:24:24 Starting to listen on :9090\\n\",\n \"insertId\": \"1oa4vhg3qfxidt\",\n \"resource\": {\n \"type\": \"k8s_container\",\n \"labels\": {\n \"pod_name\": \"monitoring-example-7685d96496-xqfsf\",\n \"cluster_name\": ...,\n \"namespace_name\": \"default\",\n \"project_id\": ...,\n \"location\": \"us-west1\",\n \"container_name\": \"prometheus-example-exporter\"\n }\n },\n \"timestamp\": \"2020-11-14T01:24:24.358600252Z\",\n \"labels\": {\n \"k8s-pod/pod-template-hash\": \"7685d96496\",\n \"k8s-pod/app\": \"monitoring-example\"\n },\n \"logName\": \"projects/.../logs/stdout\",\n \"receiveTimestamp\": \"2020-11-14T01:24:39.562864735Z\"\n }\n ```\n\n### gcloud\n\n1. Run this command:\n\n ```\n gcloud logging read 'resource.labels.project_id=\"PROJECT_ID\" AND \\\n resource.type=\"k8s_container\" AND resource.labels.namespace_name=\"default\"'\n ```\n\n Replace \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e with the ID of your\n [logging-monitoring project](/anthos/clusters/docs/on-prem/1.9/how-to/multiple-gcp-projects#logging_monitoring_project).\n2. In the output, you can see log entries from the `monitoring-example`\n Deployment. For example:\n\n ```\n insertId: 1oa4vhg3qfxidt\n labels:\n k8s-pod/app: monitoring-example\n k8s- pod/pod-template-hash: 7685d96496\n logName: projects/.../logs/stdout\n receiveTimestamp: '2020-11-14T01:24:39.562864735Z'\n resource:\n labels:\n cluster_name: ...\n container_name: prometheus-example-exporter\n location: us-west1\n namespace_name: default\n pod_name: monitoring-example-7685d96496-xqfsf\n project_id: ...\n type: k8s_container\n textPayload: |\n 2020/11/14 01:24:24 Starting to listen on :9090\n timestamp: '2020-11-14T01:24:24.358600252Z'\n ```\n\nViewing application metrics in the Google Cloud console\n-------------------------------------------------------\n\nYour example application exposes a custom metric named `example_monitoring_up`.\nYou can view the values of that metric in the Google Cloud console.\n\n1. Go to the Metrics explorer in the Google Cloud console.\n\n [Go to the Metrics explorer](https://console.cloud.google.com/monitoring/metrics-explorer)\n2. For **Resource type** , select `Kubernetes Pod`.\n\n3. For **metric** , select `external/prometheus/example_monitoring_up`.\n\n4. In the chart, you can see that `example_monitoring_up` has a repeated value\n of 1."]]