This page explains how to use metrics to monitor Policy Controller.
Policy Controller includes multiple metrics related to policy usage. For example, there are metrics recording the number of constraints and constraint templates, and the number of audit violations detected. To create and record these metrics, Policy Controller uses OpenTelemetry. You can configure Policy Controller to export these metrics to Prometheus or Cloud Monitoring. The default setting for exporting metrics exports the metrics to both Prometheus and Cloud Monitoring.
Configure metrics export
You can configure how Policy Controller exports its metrics. You can choose Prometheus, Cloud Monitoring, both, or neither when installing Policy Controller. By default, Policy Controller attempts to export metrics to both Prometheus and Cloud Monitoring.
Export metrics to Cloud Monitoring
If Policy Controller is running inside a Google Cloud environment that has a default service account, Policy Controller automatically exports metrics to Cloud Monitoring.
If Workload Identity Federation for GKE or fleet Workload Identity Federation are enabled, allow Policy Controller to send metrics by running this command:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member="serviceAccount:PROJECT_ID.svc.id.goog[gatekeeper-system/gatekeeper-admin]" \
--role=roles/monitoring.metricWriter
Replace PROJECT_ID with the cluster's Google Cloud project ID.
You can view these metrics with Metrics Explorer or by using the Cloud Monitoring API.
Export metrics to Prometheus
Policy Controller exports metrics for Prometheus on port 8888 of the
gatekeeper-controller-manager-* Pod under the gatekeeper-system namespace.
If Policy Controller is running on a cluster with Google Cloud Managed Service for Prometheus configured, these metrics will automatically be collected and stored in Cloud Monitoring. This also works for clusters that enable Google Cloud Managed Service for Prometheus after Policy Controller is installed. You might also need to grant additional permissions to the default Compute Engine service account that Google Cloud Managed Service for Prometheus uses, depending on your policies. For details on granting permissions to Google Cloud Managed Service for Prometheus, see Enable Managed Collection: GKE. For more information on configuring Google Cloud Managed Service for Prometheus, see Get started with managed collection.
Because Policy Controller is available with GKE Enterprise, there is no additional cost for Google Cloud Observability, including Google Cloud Managed Service for Prometheus. If you had Policy Controller installed prior to requiring a GKE Enterprise license, and have Google Cloud Managed Service for Prometheus enabled, you may see charges for the metrics collected. For more information on Cloud Monitoring pricing, see Pricing for Google Cloud Managed Service for Prometheus.
For examples on how to view metrics with the Google Cloud Managed Service for Prometheus solution, see Query using Cloud Monitoring.
View metrics
Policy Controller metrics are exported to your Cloud Monitoring project in Prometheus format. As a result, you can query metrics by using the Cloud Monitoring API and a dashboard in the Google Cloud console. You can edit this dashboard to meet your business and operational needs.
To query the Cloud Monitoring API, use either Prometheus Query Language (PromQL) (the de-facto query language for Kubernetes metrics) or Monitoring Query Language (MQL) (Google's proprietary metrics query language).
To create the Policy Controller dashboard:
In the Google Cloud console, go to the Dashboards page.
On the Dashboards overview page, click the Sample library tab. This tab shows you all of the dashboards that you can import.
In the Categories column, select Anthos Config Management.
In the Anthos Config Management samples table, select the Policy Controller checkbox.
Click download Import.
To create the dashboard, click Confirm in the confirmation window.
To view and edit the Policy Controller dashboard:
- On the Dashboards overview page, select the Dashboard list tab.
- Select the Policy Controller dashboard.
- To customize the dashboard, click Edit dashboard.
- Make the required changes and click Save. To learn more about customizing dashboards, see Filter a dashboard in the Cloud Monitoring documentation.
Create alerts
To receive notifications when your metrics meet certain thresholds, create alerting policies in Cloud Monitoring.
Third party integration
By using the Cloud Monitoring API, any third party observability tool can ingest Policy Controller metrics.
For example, if you're using Grafana dashboards, add the Cloud Monitoring API as the data source in Grafana. To learn more, see Google Cloud Monitoring in the Grafana documentation.
Available metrics
If Policy Controller is enabled on your cluster and is configured to export to Cloud Monitoring, you can query the following
metrics (all prefixed with OpenCensus/):
| Name | Type | Labels | Description |
|---|---|---|---|
OpenCensus/audit_duration_seconds |
Cumulative | Audit cycle duration distribution | |
OpenCensus/audit_last_run_time |
Gauge | The epoch timestamp since the last audit runtime, given as seconds in floating-point | |
OpenCensus/constraint_template_ingestion_count |
Cumulative | status | Total number of constraint template ingestion actions |
OpenCensus/constraint_template_ingestion_duration_seconds |
Cumulative | status | Constraint Template ingestion duration distribution |
OpenCensus/constraint_templates |
Gauge | status | Current number of constraint templates |
OpenCensus/validation_request_count |
Counter | admission_status | Count of admission requests from the API server |
OpenCensus/validation_request_duration_seconds |
Cumulative | admission_status | Admission request duration distribution |
OpenCensus/violations |
Gauge | enforcement_action | Number of audit violations detected in the last audit cycle |
OpenCensus/watch_manager_intended_watch_gvk |
Gauge | How many unique GroupVersionKinds Policy Controller is meant to be watching. This metric is a combination of synced resources and constraints. | |
OpenCensus/watch_manager_watched_gvk |
Gauge | How many unique GroupVersionKinds Policy Controller is actually watching. This metric is meant to converge on being equal to OpenCensus/watch_manager_intended_watch_gvk. |
If Policy Controller is configured to export to Prometheus, you can query the following metrics (all prefixed with Prometheus/):
| Name | Type | Labels | Description |
|---|---|---|---|
Prometheus/gatekeeper_audit_duration_seconds/histogram |
Cumulative | Audit cycle duration distribution | |
Prometheus/gatekeeper_audit_last_run_end_time/gauge |
Gauge | The epoch timestamp of the end of the last audit run, given as seconds in floating-point | |
Prometheus/gatekeeper_audit_last_run_time/gauge |
Gauge | The epoch timestamp of the start of the last audit run, given as seconds in floating-point | |
Prometheus/gatekeeper_constraint_template_ingestion_count/counter |
Cumulative | status | Total number of constraint template ingestion actions |
Prometheus/gatekeeper_constraint_template_ingestion_duration_seconds/histogram |
Cumulative | status | Constraint Template ingestion duration distribution |
Prometheus/gatekeeper_constraint_templates/gauge |
Gauge | status | Current number of constraint templates |
Prometheus/gatekeeper_validation_request_count/counter |
Cumulative | admission_status, admission_dryrun | Count of admission requests from the API server |
Prometheus/gatekeeper_validation_request_duration_seconds/histogram |
Cumulative | admission_status | Admission request duration distribution |
Prometheus/gatekeeper_violations/gauge |
Gauge | enforcement_action | Number of audit violations detected in the last audit cycle |
Prometheus/gatekeeper_watch_manager_intended_watch_gvk/gauge |
Gauge | How many unique GroupVersionKinds Policy Controller is meant to be watching. This metric is a combination of synced resources and constraints. | |
Prometheus/gatekeeper_watch_manager_watched_gvk/gauge |
Gauge | How many unique GroupVersionKinds Policy Controller is actually watching. This metric is meant to converge on being equal to Prometheus/gatekeeper_watch_manager_intended_watch_gvk/gauge. |