Monitoring (MON)

Audit log source

Audited operations

Run PromQL queries using the user interface of the monitoring instance
Run PromQL queries using the HTTP API
Perform dashboard CRUD operations

Perform alert CRUD operations

Run PromQL queries using the user interface of the monitoring instance

Audit log source	Proxy server
Log type	Data plane

Fields in the log entry that contain audit information
Audit metadata	Audit field name	Value
User or service identity	`user`	For example, "user":{ "identity":"fop-cluster-admin@example.com", "issuer":"https://ais-core.org-1.zone1.google.gdch.test" }
Target (Fields and values that call the API)	`resource`	For example, `"resource": "/infra-obs/grafana/api/ds/query"`
Action (Fields containing the performed operation)	`action`	Possible values: `"action": "QUERY"` `"action": "CREATE"` `"action": "READ"` `"action": "UPDATE"` `"action": "DELETE"` `"action": "CREATE/UPDATE"`
Event timestamp	`time`	For example, `"time": "2022-12-05T14:39:15.713354008Z"`
Source of action	`sourceIPs` `_gdch_service_name`	For example, "sourceIPs":[ "10.253.166.214", "127.0.0.6" ], "_gdch_service_name":"grafana"
Outcome	`response`	For example, `"response": "Successful: 200 OK"`
Other fields	`description`	The `description` value contains the complete query. For more information. see the Example log.

Example log

{
  "resource":"/infra-obs/grafana/api/ds/query",
  "response":"Successful: 200 OK",
  "_gdch_service_tenant":"infra-obs",
  "sourceIPs":[
    "10.253.166.214",
    "127.0.0.6"
  ],
  "_gdch_namespace":"infra-obs-obs-system",
  "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0",
  "time":"2022-12-05T14:39:15.713354008Z",
  "auditID":"6bba5ff1-97d9-4bf8-92af-4f63049448cf",
  "numBytesSent":1821,
  "action":"QUERY",
  "_gdch_service_name":"grafana",
  "numBytesReceived":2827,
  "description":"{
      \"queries\":[{
      \"refId\":\"A\",
      \"expr\":\"{container=\\\"grafana-proxy-server\\\"},
      \"queryType\":\"range\",
      \"datasource\":{\"uid\":\"P982945308D3682D1\",\"type\":\"loki\"},
      \"key\":\"Q-c63373da-dec2-49c3-aa6c-4e5ba07ec8de-0\",
      \"editorMode\":\"builder\",
      \"maxLines\":1000,
      \"legendFormat\":\"\",
      \"datasourceId\":2,
      \"intervalMs\":1000,
      \"maxDataPoints\":2493
    }],
    \"range\":{
      \"from\":\"2022-12-05T13:39:15.461Z\",
      \"to\":\"2022-12-05T14:39:15.461Z\",
      \"raw\":{\"from\":\"now-1h\",\"to\":\"now\"}
    },
    \"from\":\"1670247555461\",
    \"to\":\"1670251155461\"
  }",
  "user":{
    "identity":"fop-cluster-admin@example.com",
    "issuer":"https://ais-core.org-1.zone1.google.gdch.test"
  },
  "_gdch_cluster":"org-1-admin",
  "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-whltm"
}

Run PromQL queries using the HTTP API

Audit log source	Proxy server
Log type	Data plane

Fields in the log entry that contain audit information
Audit metadata	Audit field name	Value
User or service identity	`user`	For example, "user":{ "issuer":"https://ais-core.org-1.zone1.google.gdch.test", "identity":"fop-cluster-admin@example.com" }
Target (Fields and values that call the API)	`resource`	For example, `"resource":"/alertmanager/api/v2/alerts/groups?silenced=false&inhibited=false&active=true"`
Action (Fields containing the performed operation)	`action`	Possible values: `"action": "QUERY"` `"action": "CREATE"` `"action": "READ"` `"action": "UPDATE"` `"action": "DELETE"` `"action": "CREATE/UPDATE"`
Event timestamp	`time`	For example, `"time": "2022-12-05T18:20:50.616925009Z"`
Source of action	`sourceIPs` `_gdch_service_name`	For example, "sourceIPs":[ "10.200.0.1", "127.0.0.6" ], "_gdch_service_name":"cortex"
Outcome	`response`	For example, `"response": "Successful: 200 OK"`
Other fields	Not applicable	Not applicable

Example log

{
  "user":{
    "issuer":"https://ais-core.org-1.zone1.google.gdch.test",
    "identity":"fop-cluster-admin@example.com"
  },
  "_gdch_service_tenant":"infra-obs",
  "_gdch_service_name":"cortex",
  "resource":"/alertmanager/api/v2/alerts/groups?silenced=false&inhibited=false&active=true",
  "time":"2022-12-05T18:20:50.616925009Z",
  "action":"READ",
  "numBytesReceived":2376,
  "sourceIPs":[
    "10.200.0.1",
    "127.0.0.6"
  ],
  "_gdch_namespace":"obs-system",
  "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0",
  "numBytesSent":173,
  "auditID":"8451a7b3-77f9-4878-9308-641b55a83865",
  "response":"Successful: 200 OK",
  "_gdch_cluster":"org-1-admin",
  "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-2wqxp"
}

Perform dashboard CRUD operations

Audit log source	Kubernetes audit logs
Log type	Control plane

Fields in the log entry that contain audit information
Audit metadata	Audit field name	Value
User or service identity	`user`	For example, "user":{ "extra":{ "authentication.kubernetes.io/pod-name":["fleet-admin-controller-875778d98-dnkj2"], "authentication.kubernetes.io/pod-uid":["caa4df7a-ae04-458e-a616-1c6893ce6e46"] }, "username":"system:serviceaccount:gpc-system:fleet-admin-controller", "groups":[ "system:serviceaccounts", "system:serviceaccounts:gpc-system", "system:authenticated" ], "uid":"0b93d757-e3be-440a-b18a-4a2b524de156" }
Target (Fields and values that call the API)	`requestURI` `objectRef`	For example, "requestURI":"/apis/observability.gdc.goog/v1/namespaces/alice-obs-system/dashboards", "objectRef":{ "apiVersion":"v1", "apiGroup":"observability.gdc.goog", "resource":"dashboards", "namespace":"alice-obs-system" }
Action (Fields containing the performed operation)	`verb`	Possible values: `"verb": "create"` `"verb": "delete"` `"verb": "get"` `"verb": "list"` `"verb": "patch"` `"verb": "update"` `"verb": "watch"`
Event timestamp	`requestReceivedTimestamp`	For example, `"requestReceivedTimestamp": "2022-12-05T15:36:24.980257Z"`
Source of action	`sourceIPs` `_gdch_service_name`	For example, "sourceIPs":["10.253.166.100"], "_gdch_service_name":"apiserver"
Outcome	`responseStatus`	For example, "responseStatus":{ "code":201, "metadata":{} }
Other fields	Not applicable	Not applicable

Example log

{
  "user":{
    "extra":{
      "authentication.kubernetes.io/pod-name":["fleet-admin-controller-875778d98-dnkj2"],
      "authentication.kubernetes.io/pod-uid":["caa4df7a-ae04-458e-a616-1c6893ce6e46"]
    },
    "username":"system:serviceaccount:gpc-system:fleet-admin-controller",
    "groups":[
      "system:serviceaccounts",
      "system:serviceaccounts:gpc-system",
      "system:authenticated"
    ],
    "uid":"0b93d757-e3be-440a-b18a-4a2b524de156"
  },
  "kind":"Event",
  "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-2wqxp",
  "apiVersion":"audit.k8s.io/v1",
  "_gdch_cluster":"org-1-admin",
  "level":"Metadata",
  "stageTimestamp":"2022-12-05T15:36:24.980257Z",
  "auditID":"a060d80a-4a47-4490-a859-5d3ccff36d3d",
  "requestReceivedTimestamp":"2022-12-05T15:36:24.980257Z",
  "userAgent":"fleet-admin-cm/v0.0.0 (linux/amd64) kubernetes/$Format",
  "stage":"RequestReceived",
  "requestURI":"/apis/observability.gdc.goog/v1/namespaces/alice-obs-system/dashboards",
  "objectRef":{
    "apiVersion":"v1",
    "apiGroup":"observability.gdc.goog",
    "resource":"dashboards",
    "namespace":"alice-obs-system"
  },
  "verb":"create",
  "sourceIPs":["10.253.166.100"],
  "_gdch_service_name":"apiserver"
}

Perform alert CRUD operations

Audit log source	Kubernetes audit logs
Log type	Control plane

Fields in the log entry that contain audit information
Audit metadata	Audit field name	Value
User or service identity	`user`	For example, "user":{ "username":"kubernetes-admin", "groups":[ "system:masters", "system:authenticated" ] }
Target (Fields and values that call the API)	`requestURI` `objectRef`	For example, "requestURI":"/apis/monitoring.gdc.goog/v1/namespaces/alice/monitoringrules?fieldManager=kubectl-client-side-apply&fieldValidation=Strict", "objectRef":{ "apiVersion":"v1", "apiGroup":"monitoring.gdc.goog", "name":"obs-test-alert-sequel", "namespace":"alice", "resource":"monitoringrules" }
Action (Fields containing the performed operation)	`verb`	Possible values: `"verb": "create"` `"verb": "delete"` `"verb": "get"` `"verb": "list"` `"verb": "patch"` `"verb": "update"` `"verb": "watch"`
Event timestamp	`requestReceivedTimestamp`	For example, `"requestReceivedTimestamp": "2022-12-05T16:28:50.619659Z"`
Source of action	`sourceIPs` `_gdch_service_name`	For example, "sourceIPs":["10.200.0.6"], "_gdch_service_name":"apiserver"
Outcome	`responseStatus`	For example, "responseStatus":{ "code":201, "metadata":{} }
Other fields	Not applicable	Not applicable

Example log

{
  "level":"Metadata",
  "sourceIPs":[
    "10.200.0.6"
  ],
  "auditID":"753c3370-d3a5-4717-b84e-00fd56883fc4",
  "requestURI":"/apis/monitoring.gdc.goog/v1/namespaces/alice/monitoringrules?fieldManager=kubectl-client-side-apply&fieldValidation=Strict",
  "apiVersion":"audit.k8s.io/v1",
  "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-fgkth",
  "user":{
    "username":"kubernetes-admin",
    "groups":[
      "system:masters",
      "system:authenticated"
    ]
  },
  "userAgent":"kubectl/v1.25.4 (linux/amd64) kubernetes/872a965",
  "verb":"create",
  "stage":"ResponseComplete",
  "stageTimestamp":"2022-12-05T16:28:50.636050Z",
  "_gdch_cluster":"org-1-admin",
  "objectRef":{
    "apiVersion":"v1",
    "apiGroup":"monitoring.gdc.goog",
    "name":"obs-test-alert-sequel",
    "namespace":"alice",
    "resource":"monitoringrules"
  },
  "responseStatus":{
    "code":201,
    "metadata":{}
  },
  "kind":"Event",
  "annotations":{
    "authorization.k8s.io/reason":"",
    "authorization.k8s.io/decision":"allow"
  },
  "requestReceivedTimestamp":"2022-12-05T16:28:50.619659Z",
  "_gdch_service_name":"apiserver"
}