Manage process ID limits

Stay organized with collections Save and categorize content based on your preferences.

Overview

A Process ID (PID) limit is a Kubernetes resource constraint on nodes and pods to prevent excessive process creation, which can impact node stability. Apigee Hybrid supports the Kubernetes feature to set process ID limits. This document provides instructions on how to set these limits, with recommendations for values for Apigee services on specific platforms.

When Apigee hybrid users manage their own clusters, setting PID limits in Kubernetes can improve system stability, security, and resource management. This is also consistent with Kubernetes best practices.

Definitions of process ID limits

Process ID limits include Node PID limits and Pod PID limits.

Node PID limits include Kube-reserved PIDs and system-reserved PIDs. The total number of allocatable PIDs is the Kernel max minus the kube-reserved PIDs minus the system-reserved PIDs minus the eviction threshold PIDs:

• Kernel max ID limits: Determined by the operating system and its kernel settings. Since Apigee Hybrid runs exclusively on a Linux kernel, this guide discusses Linus-based limits on Kubernetes nodes. The max number of process ID limits for a Linux kernel is 4194304.
• Kube-reserved and system-reserved: For resource reservation for Kubernetes or OS system daemons.
• Eviction threshold: The limit that signals pressure on the node. The node will be evicted when the threshold is met. See PID based eviction for more details.
• Allocatable: The number of PIDs available. See Kubernetes: Node Allocatable for more information. Kube-reserved and system-reserved can be configured with Node PID limit settings.

The pod PID limit can be configured for the node and shared across all pods within the node.

Prepare to manage the process ID limits

These procedures uses the following environment variables:

export PROJECT_ID=MY_PROJECT_ID
export CLUSTER_NAME=MY_CLUSTER_NAME
export LOCATION=MY_CLUSTER_LOCATION
export APIGEE_NAMESPACE=MY_APIGEE_NAMESPACE # Default: apigee

Verify access

Before configuring the process ID limit, ensure that you have access to edit Kubernetes clusters.

The following steps are for installations on GKE. For other platforms, see the documentation for your platform.

1. Check if you have roles/container.clusterAdmin in the IAM policy:

   gcloud projects get-iam-policy ${PROJECT_ID}  \
    --flatten="bindings[].members" \
    --format='table(bindings.role)' \
    --filter="bindings.members:your_account_email"
   

2. If you do not have the access, add the role to your account:

   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member user:your_account_email \
    --role roles/container.clusterAdmin

Verify Existing PID Limits

Verify whether the node has existing PID limits before configuring new limits.

1. Get a node from the cluster to check values. You will need to check nodes from both the apigee-data and apigee-runtime node pools:

   kubectl get nodes -n ${APIGEE_NAMESPACE}
   

   Your output should look similar to the following:

   NAME                                      STATUS   ROLES    AGE    VERSION
   gke-my-hybrid-apigee-data-0a1b2c3d-efgh      Ready       2d8h   v1.31.5-gke.1169000
   gke-my-hybrid-apigee-runtime-1b2c3d4e-fghi   Ready       2d8h   v1.31.5-gke.1169000
   

2. Export the node name from the output of the previous step. Run this the next step first for an apigee-data node and again for an apigee-runtime node:

   Code

   export NODE_NAME=MY_NODE_NAME
   

   Example

   export NODE_NAME="gke-my-hybrid-apigee-data-0a1b2c3d-efgh"
   

3. Check the node PID limits. Use the following commands to check the reserved values. If the value is null, the value is not configured:

   kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.kubeReserved'
   kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.systemReserved'
   kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.evictionHard'
   

4. Check the pod PID limit. Use the following command to check the existing pod PID limit. If the returned value is -1 or empty, the limit is not set:

   kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.podPidsLimit'
   

Manage the process ID limits

Manage the Node PID limits

For installations on GKE, the infrastructure resources of Kubernetes nodes are managed internally and you do not need to configure them. The current capacity and allocatable resources can be viewed following Node allocatable resources in the Google Kubernetes Engine documentation.

For non-GKE platforms, refer to corresponding Kubernetes documentation for the platform. If the cluster/node is user-managed (as opposed to Fully-managed), the kube-reserved PID limit and system-reserved PID limit can be configured with Kubelet. See Node PID limits in the Kubernetes documentation.

Tooling

This procedure uses Kubelet to manage the process ID limit. Kubelet is the agent that runs on pods and containers that ensures they are running according to their PodSpec. If you need to install Kubelet, follow the instructions in the Kubernetes documentation: Installing kubeadm, kubelet and kubectl.

Procedure

1. Create a Kubelet configuration file called kubelet-config.yaml.

   apiVersion: kubelet.config.k8s.io/v1beta1
   kind: KubeletConfiguration
   kubeReserved:
    pid: PID_VALUE # Example: 1000
   

   See more info on the configurations: Kube Reserved in the Kubernetes documentation.

2. Apply the Kubelet configuration:

   kubelet --config PATH_TO_KUBELET_CONFIG_YAML
   

Manage the Pod PID limits

Choosing the Limit

If the PID limit is set too low, it can prevent a pod from starting up. If it is set too high, it can fail to detect resource misbehaviors. When selecting the appropriate limit, it is important to consider the previous behavior of the node and any service-specific requirements.

GKE has a required range for the value: [1024, 4194304]. On GKE Platforms you can see the status of your Kubernetes service accounts in the Google Cloud console Metrics Explorer. Select Kubernetes Node - PID usage metrics and apply filters. This metrics shows the recent usage of process IDs and can be referenced when choosing a PID limit.

Metrics Explorer

On non-GKE platforms, different monitoring options may be available. Please refer to the corresponding platform's Kubernetes documentation to check the metrics.

Process ID requirements for Apigee pods

Apigee Hybrid uses 2 node pools, apigee-data and apigee-runtime. Since some of the Apigee components are deployed in both node pools, the Pod PID limit shall be the same for the two node pools. The Pod PID limit should also be higher than the maximum number of PIDs required across all Apigee pods. The required Apigee Pod PID limit is 1000, which is below the minimum required number for GKE platforms.

Recommended Pod PIDs Limit

Some platforms enforce a minimum value requirement for the number of the Pod PIDs limit, in which case the min value requirement is selected.

Procedure

The procedures for managing the Pod PID limits are different for GKE and non-GKE platforms.

Troubleshooting process ID limits

Pod stuck in Pending status with a FailedScheduling error

When the Node or Pod PIDs limit evicts or restricts a pod from starting up, the pod will be stuck in Pending status and fail with FailedScheduling error.

1. Get the Node column:

   kubectl get pods -n ${APIGEE_NAMESPACE} ${POD_NAME} -o wide
   

2. Check if there's a PIDPressure condition:

   kubectl describe node -n apigee ${NODE_NAME} | grep PIDPressure
   

3. Alternatively, check ApigeeDeployment of the corresponding pod. Get the ApigeeDeployment from the result that has the same prefix as the erroring pod.

   kubectl get ApigeeDeployment -n ${APIGEE_NAMESPACE}
   

4. Check if the recent Events has PID-related error messages:

   kubectl describe ApigeeDeployment -n ${APIGEE_NAMESPACE} ${APIGEE_DEPLOYMENT_NAME}
   

5. If the cause is confirmed to be PID limit, follow the steps in Manage the Node PID limits to update the PID limit to a higher value.

Invalid podPidsLimit

When setting the limit for GKE, if the podPidsLimit is beyond limit, error will prompt:

ERROR: (gcloud.container.node-pools.update) ResponseError: code=400, message=Invalid podPidsLimit : value must be 1024 <= podPidsLimit <= 4194304.

Update the podPidsLimit value to within the required range.