Manage billing alerts

This page covers creating and silencing Billing alerts and alert rules.

Before you begin

Before continuing, ensure you have the following required role:

  • MonitoringRule PA Creator: create and manage MonitoringRule custom resources. Ask your Organization IAM Admin to grant you the monitoringrule-pa-creator role.

Create alert rules

You can create alert rules based on invoice metrics and your spend thresholds using the MonitoringRule resource in the Observability API.

See Create alert rules for instructions on creating alerts using the GDC console.

To create an alert rule, work through the following steps:

  1. Create a YAML file, and add in the MonitoringRule resource and the following contents:

    apiVersion: monitoring.gdc.goog/v1
kind: MonitoringRule
metadata:
  name: MONITORING_RULE_NAME
  namespace: PROJECT
spec:
  interval: INTERVAL
  limit: LIMIT
  alertRules:
  - alert: ALERT
    expr: EXPRESSION
    labels:
      severity: SEVERITY
      code: BILLING_CODE
      resource: BILLING
    annotations:
      message: MESSAGE

    Replace the following variables:

    Variable Description
    MONITORING_RULE_NAME The name of the monitoring rule.

    For example: invoice-total-cost.
    PROJECT The project namespace of where the invoice resides.

    For example: platform-obs.
    INTERVAL The length of time the alert re-runs in a given period.

    For example: 60s defines that the alert runs every sixty seconds.
    LIMIT The amount of times an alert can run.

    For example: 0 for no limit.
    ALERT The alert name.

    For example: invoice_total_cost_approaching_limit.
    EXPRESSION The PromQL expression to describe the alert rule. To retrieve a PromQL expression, see Query alerts from the HTTP API.

    For example, to set an alert rule with a spend threshold at 80% of your $500,000 budgeted amount, use the following expression:

    sum(metering_invoice_line_item_total_cost{task_order_number="TASK_ORDER", clin_number="CLIN"}) / 500000 > 0.8

    Replace TASK_ORDER with your task order and CLIN with your contract line item number.
    SEVERITY The severity of the alert rule. Severity values include error, critical, warning, and info.
    BILLING_CODE The billing code you define for the alert rule.

    For example: "BILLING-100".
    MESSAGE A description of the alert that appears when the alert rule runs.

    For example: "Spending exceeded limit".

  2. Apply the contents of the YAML file and set the alert rule:

    kubectl apply -f FILENAME.yaml

Create spend threshold alerts

Your Infrastructure Operator (IO) creates spend threshold alert policies in the GDC console. Spend threshold alerts are custom billing alerts that indicate when the billing costs exceed the threshold you set. For example, your IO creates billing total cost alerts and configures the alerts based on a fixed dollar amount, or as a percentage of the initial budgeted amount.

To create spend threshold alerts and other organization-level billing alerts, submit a request to your IO. Include the following parameters in your request:

  • Task order (TO) number.
  • CLIN
  • The total budget amount for the specified TO and CLIN.
  • The budget threshold.

Silence alerts

You can silence and unsilence alert rules through the GDC console. Follow the instructions in Silence an alert.

Configure billing notifications

To configure billing notifications, follow the instructions in Configure notification channels.