Implement a Job queuing system with quota sharing between namespaces on GKE

Create the ResourceFlavors

A ResourceFlavor represents resource variations in your cluster nodes, such as different VMs (for example spot versus on-demand), architectures (for example, x86 vs ARM CPUs), brands and models (for example, Nvidia A100 versus T4 GPUs).

ResourceFlavors use node labels and taints to match with a set of nodes in the cluster.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: on-demand # This ResourceFlavor will be used for the CPU resource
spec:
  nodeLabels:
    cloud.google.com/gke-provisioning: standard # This label was applied automatically by GKE
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: spot # This ResourceFlavor will be used as added resource for the CPU resource
spec:
  nodeLabels:  
    cloud.google.com/gke-provisioning: spot # This label was applied automatically by GKE

In this manifest:

• The ResourceFlavor on-demand has its label set to cloud.google.com/gke-provisioning: standard.
• The ResourceFlavor spot has its label set to cloud.google.com/gke-provisioning: spot.

When a workload is assigned a ResourceFlavor, Kueue assigns the Pods of the workload to nodes that match the node labels defined for the ResourceFlavor.

Deploy the ResourceFlavor:

kubectl apply -f flavors.yaml

Create the ClusterQueue and LocalQueue

Create two ClusterQueues cq-team-a and cq-team-b, and their corresponding LocalQueues lq-team-a and lq-team-b respectively namespaced to team-a and team-b.

ClusterQueues are cluster-scoped object that governs a pool of resources such as CPU, memory, and hardware accelerators. Batch administrators can restrict the visibility of these objects to batch users.

LocalQueues are namespaced objects that batch users can list. They point to CluterQueues, from which resources are allocated to run the LocalQueue workloads.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: cq-team-a
spec:
  cohort: all-teams # cq-team-a and cq-team-b share the same cohort
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: team-a #Only team-a can submit jobs direclty to this queue, but will be able to share it through the cohort
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: on-demand
      resources:
      - name: "cpu"
        nominalQuota: 10
        borrowingLimit: 5
      - name: "memory"
        nominalQuota: 10Gi
        borrowingLimit: 15Gi
    - name: spot # This ClusterQueue doesn't have nominalQuota for spot, but it can borrow from others
      resources:
      - name: "cpu"
        nominalQuota: 0
      - name: "memory"
        nominalQuota: 0
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: team-a # LocalQueue under team-a namespace
  name: lq-team-a
spec:
  clusterQueue: cq-team-a # Point to the ClusterQueue team-a-cq

ClusterQueues allows resources to have multiple flavors. In this case, both ClusterQueues have two flavors, on-demand and spot, each providing cpu resources. The quota of the ResourceFlavor spot is set to 0, and will not be used for now.

Both ClusterQueues share the same cohort called all-teams, defined in .spec.cohort. When two or more ClusterQueues share the same cohort, they can borrow unused quota from each other.

You can learn more about how cohorts work and the borrowing semantics in the Kueue documentation

Deploy the ClusterQueues and LocalQueues:

kubectl apply -f cq-team-a.yaml
kubectl apply -f cq-team-b.yaml

(Optional) Deploy kube-prometheus and monitor Workloads using Prometheus

You can use Prometheus to monitor Kueue pending workloads and active workloads. In order to monitor workloads being brought up and observe load on each ClusterQueue, set up Prometheus to the cluster under the namespace monitoring.

1. Download the source code for prometheus operator for monitoring:

   cd
   git clone https://github.com/prometheus-operator/kube-prometheus.git
   

2. Create the CustomResourceDefinitions(CRDs):

   kubectl create -f kube-prometheus/manifests/setup
   

3. Create the monitoring components:

   kubectl create -f kube-prometheus/manifests
   

4. Allow prometheus-operator to scrape metrics from Kueue components:

   kubectl apply -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/prometheus.yaml
   

5. Change to the working directory:

   cd kubernetes-engine-samples/batch/kueue-cohort
   

6. Start a new terminal to access Prometheus by port forwarding the service:

   kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
   

7. Open Prometheus on localhost:9090 in the browser

8. If using Cloud Shell, click Web Preview, select change port, set the port number to 9090, and select Change and Preview.

9. Enter the query for the first panel that monitors the active ClusterQueue cq-team-a:

   kueue_pending_workloads{cluster_queue="cq-team-a", status="active"} or kueue_admitted_active_workloads{cluster_queue="cq-team-a"}
   

10. Add another panel and enter the query that monitors the active ClusterQueue cq-team-b:

    kueue_pending_workloads{cluster_queue="cq-team-b", status="active"} or kueue_admitted_active_workloads{cluster_queue="cq-team-b"}
    

11. Add another panel and enter the query that monitors the number of nodes in the cluster:

    count(kube_node_info)
    

Create Jobs and observe the admitted workloads

Generate Jobs to both ClusterQueues that will sleep for 10 seconds, with three paralleled Jobs and will be completed with three completions. It will then be cleaned up after 60 seconds.

apiVersion: batch/v1
kind: Job
metadata:
  namespace: team-a # Job under team-a namespace
  generateName: sample-job-team-a-
  labels:
    kueue.x-k8s.io/queue-name: lq-team-a # Point to the LocalQueue
spec:
  ttlSecondsAfterFinished: 60 # Job will be deleted after 60 seconds
  parallelism: 3 # This Job will have 3 replicas running at the same time
  completions: 3 # This Job requires 3 completions
  suspend: true # Set to true to allow Kueue to control the Job when it starts
  template:
    spec:
      containers:
      - name: dummy-job
        image: gcr.io/k8s-staging-perf-tests/sleep:latest
        args: ["10s"] # Sleep for 10 seconds
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
      restartPolicy: Never

job-team-a.yaml creates Jobs under the namespace team-a and points to the LocalQueue lq-team-a and the ClusterQueue cq-team-a.

Similarly, job-team-b.yaml creates Jobs under team-b namespace, and points to the LocalQueue lq-team-b and the ClusterQueue cq-team-b.

1. Start a new terminal and run this script to generate a Job every second:

   ./create_jobs.sh job-team-a.yaml 1
   

2. Start another terminal and create Jobs for the team-b namespace:

   ./create_jobs.sh job-team-b.yaml 1
   

3. Observe the Jobs being queued up in Prometheus. Or with this command:

   watch -n 2 kubectl get clusterqueues -o wide
   

The output should be similar to the following:

    NAME        COHORT      STRATEGY         PENDING WORKLOADS   ADMITTED WORKLOADS
    cq-team-a   all-teams   BestEffortFIFO   0                   5
    cq-team-b   all-teams   BestEffortFIFO   0                   4

Borrow unused quota with cohorts

ClusterQueues might not be at full capacity at all times. Quotas usage is not maximized when workloads are not evenly spread out among ClusterQueues. If ClusterQueues share the same cohort between each other, ClusterQueues can borrow quotas from other ClusterQueues to maximize the quota utilization.

1. Once there are Jobs queued up for both ClusterQueues cq-team-a and cq-team-b, stop the script for the team-b namespace by pressing CTRL+c on the corresponding terminal.

2. Once all the pending Jobs from the namespace team-b are processed, the jobs from the namespace team-a can borrow the available resources in cq-team-b:

   kubectl describe clusterqueue cq-team-a
   

   Because cq-team-a and cq-team-b share the same cohort called all-teams, these ClusterQueues are able to share resources that are not utilized.

     Flavors Usage:
       Name:  on-demand
       Resources:
         Borrowed:  5
         Name:      cpu
         Total:     15
         Borrowed:  5Gi
         Name:      memory
         Total:     15Gi
   

3. Resume the script for the team-b namespace.

   ./create_jobs.sh job-team-b.yaml 3
   

   Observe how the borrowed resources from cq-team-a go back to 0, while the resources from cq-team-b are used for its own workloads:

   kubectl describe clusterqueue cq-team-a
   

     Flavors Usage:
       Name:  on-demand
       Resources:
         Borrowed:  0
         Name:      cpu
         Total:     9
         Borrowed:  0
         Name:      memory
         Total:     9Gi
   

Increase quota with Spot VMs

When quota needs to be temporarily increased, for example to meet high demand in pending workloads, you can configure Kueue to accommodate the demand by adding more ClusterQueues to the cohort. ClusterQueues with unused resources can share those resources with other ClusterQueues that belong to the same cohort.

At the beginning of the tutorial, you created a node pool named spot using Spot VMs and a ResourceFlavor named spot with the label set to cloud.google.com/gke-provisioning: spot. Create a ClusterQueue to use this node pool and the ResourceFlavor that represents it:

1. Create a new ClusterQueue called cq-spot with cohort set to all-teams:

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ClusterQueue
   metadata:
     name: spot-cq
   spec:
     cohort: all-teams # Same cohort as cq-team-a and cq-team-b
     resourceGroups:
     - coveredResources: ["cpu", "memory"]
       flavors:
       - name: spot
         resources:
         - name: "cpu"
           nominalQuota: 40
         - name: "memory"
           nominalQuota: 144Gi

   Because this ClusterQueue shares the same cohort with cq-team-a and cq-team-b, both ClusterQueue cq-team-a and cq-team-b can borrow resources up to 15 CPU requests, and 15 Gi of memory.

   kubectl apply -f cq-spot.yaml
   

2. In Prometheus, observe how the admitted workloads spike for both cq-team-a and cq-team-b thanks to the added quota by cq-spot who shares the same cohort. Or with this command:

   watch -n 2 kubectl get clusterqueues -o wide
   

3. In Prometheus, observe the number of nodes in the cluster. Or with this command:

   watch -n 2 kubectl get nodes -o wide
   

4. Stop both scripts by pressing CTRL+c for team-a and team-b namespace.