Application logging and monitoring

Version 1.11 keyboard_arrow_down

• Latest version
• Version 1.16
• Earlier versions (unsupported)

This page shows how to configure a user cluster for Google Distributed Cloud so that custom logs and metrics from user applications are sent to Cloud Logging and Cloud Monitoring.

Enabling Logging and Monitoring for user applications

The configuration for Logging and Monitoring is held in a Stackdriver object named stackdriver.

1. Open the stackdriver object for editing:

   kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG --namespace kube-system edit stackdriver stackdriver
   

   Replace USER_CLUSTER_KUBECONFIG with the path of your user cluster kubeconfig file.

2. Under spec, set enableStackdriverForApplications to true:

     apiVersion: addons.gke.io/v1alpha1
     kind: Stackdriver
     metadata:
       name: stackdriver
       namespace: kube-system
     spec:
       projectID: ...
       clusterName: ...
       clusterLocation: ...
       proxyConfigSecretName: ...
       enableStackdriverForApplications: true
       enableVPC: ...
       optimizedMetrics: true
   

3. Close the edited file.

Annotating workloads

To enable the collection of custom metrics from an application, add the prometheus.io/scrape: "true" annotation to the application's Service or Pod manifest, or add the same annotation to the spec.template section in Deployment or DaemonSet manifest in order to be passed to their Pods.

To avoid garbage collection of metrics, we recommend setting the metrics scrape interval to one minute.

Running an example application

In this section, you create an application that writes custom logs and exposes a custom metric.

1. Save the following Service and Deployment manifests to a file named my-app.yaml. Notice that the Service has the annotation prometheus.io/scrape: "true":

   kind: Service
   apiVersion: v1
   metadata:
     name: "monitoring-example"
     namespace: "default"
     annotations:
       prometheus.io/scrape: "true"
   spec:
     selector:
       app: "monitoring-example"
     ports:
       - name: http
         port: 9090
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: "monitoring-example"
     namespace: "default"
     labels:
       app: "monitoring-example"
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: "monitoring-example"
     template:
       metadata:
         labels:
           app: "monitoring-example"
       spec:
         containers:
         - image: gcr.io/google-samples/prometheus-dummy-exporter:latest
           name: prometheus-example-exporter
           imagePullPolicy: Always
           command:
           - /bin/sh
           - -c
           - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090
           resources:
             requests:
               cpu: 100m
   

2. Create the Deployment and the Service:

   kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml
   

Viewing application logs

Viewing application metrics in the Google Cloud console

Your example application exposes a custom metric named example_monitoring_up. You can view the values of that metric in the Google Cloud console.

1. Go to the Metrics explorer in the Google Cloud console.

   Go to the Metrics explorer

2. For Resource type, select Kubernetes Pod or Kubernetes Container.

3. For metric, select external.googleapis.com/prometheus/example_monitoring_up.

4. In the chart, you can see that example_monitoring_up has a repeated value of 1.