This page describes how to deploy GKE Inference Gateway.
This page is intended for Networking specialists responsible for managing GKE infrastructure, and platform administrators who manage AI workloads.
Before reading this page, ensure that you're familiar with the following:
- About GKE Inference Gateway
- AI/ML orchestration on GKE.
- Generative AI glossary.
- Load balancing in Google Cloud, especially how load balancers interact with GKE.
- GKE Service Extensions. For more information, read the GKE Gateway controller documentation.
- Customize GKE Gateway traffic using Service Extensions.
GKE Inference Gateway enhances Google Kubernetes Engine (GKE) Gateway to optimize the serving of generative AI applications. GKE Inference Gateway lets you optimize the serving of generative AI workloads on GKE. It provides efficient management and scaling of AI workloads, enables workload-specific performance objectives such as latency, and enhances resource utilization, observability, and AI safety.
Before you begin
Before you start, make sure that you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update.
Enable the Compute Engine API, the Network Services API, and the Model Armor API if needed.
Go to Enable access to APIs and follow the instructions.
GKE Gateway controller requirements
- GKE version 1.32.3.
- Google Cloud CLI version 407.0.0 or later.
- Gateway API is supported on VPC-native clusters only.
- You must enable a proxy-only subnet.
- Your cluster must have the
HttpLoadBalancingadd-on enabled. - If you are using Istio, you must upgrade Istio to one of the following
versions:
- 1.15.2 or later
- 1.14.5 or later
- 1.13.9 or later
- If you are using Shared VPC, then in the host project, you need to
assign the
Compute Network Userrole to the GKE Service account for the service project.
Restrictions and limitations
The following restrictions and limitations apply:
- Multi-cluster Gateways are not supported.
- GKE Inference Gateway is only supported on the
gke-l7-regional-external-managedandgke-l7-rilbGatewayClass resources. - Cross-regional internal Application Load Balancers are not supported.
Configure GKE Inference Gateway
To configure GKE Inference Gateway, consider this example. A
team runs vLLM and
Llama3 models and actively experiments with two
distinct LoRA fine-tuned adapters: "food-review" and "cad-fabricator".
The high-level workflow for configuring GKE Inference Gateway is as follows:
- Prepare your environment: set up the necessary infrastructure and components.
- Create an inference pool: define a pool of model
servers using the
InferencePoolCustom Resource. - Specify model serving objectives: specify model
objectives using the
InferenceModelCustom Resource. - Create the Gateway: expose the inference service using Gateway API.
- Create the
HTTPRoute: define how HTTP traffic is routed to the inference service. - Send inference requests: make requests to the deployed model.
Prepare your environment
Install Helm.
Create a GKE cluster:
- Create a GKE Autopilot or Standard cluster with version 1.31 or later. For instructions, see Create a GKE cluster.
- Configure the nodes with your preferred compute family and accelerator.
- Use GKE Inference Quickstart for pre-configured and tested deployment manifests, based on your selected accelerator, model, and performance needs.
To install the
InferencePoolandInferenceModelCustom Resource Definition (CRDs) in your GKE cluster, run the following command:kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yamlReplace
VERSIONwith the version of the CRDs you want to install (for example,v0.3.0).If you are using GKE version earlier than v1.32.2-gke.1182001 and you want to use Model Armor with GKE Inference Gateway, you must install the traffic and routing extension CRDs:
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-gateway-api/refs/heads/main/config/crd/networking.gke.io_gcptrafficextensions.yaml kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-gateway-api/refs/heads/main/config/crd/networking.gke.io_gcproutingextensions.yamlTo set up authorization to scrape metrics, create the
inference-gateway-sa-metrics-reader-secretsecret:kubectl apply -f - <<EOF --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: inference-gateway-metrics-reader rules: - nonResourceURLs: - /metrics verbs: - get --- apiVersion: v1 kind: ServiceAccount metadata: name: inference-gateway-sa-metrics-reader namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: inference-gateway-sa-metrics-reader-role-binding namespace: default subjects: - kind: ServiceAccount name: inference-gateway-sa-metrics-reader namespace: default roleRef: kind: ClusterRole name: inference-gateway-metrics-reader apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: Secret metadata: name: inference-gateway-sa-metrics-reader-secret namespace: default annotations: kubernetes.io/service-account.name: inference-gateway-sa-metrics-reader type: kubernetes.io/service-account-token --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: inference-gateway-sa-metrics-reader-secret-read rules: - resources: - secrets apiGroups: [""] verbs: ["get", "list", "watch"] resourceNames: ["inference-gateway-sa-metrics-reader-secret"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: gmp-system:collector:inference-gateway-sa-metrics-reader-secret-read namespace: default roleRef: name: inference-gateway-sa-metrics-reader-secret-read kind: ClusterRole apiGroup: rbac.authorization.k8s.io subjects: - name: collector namespace: gmp-system kind: ServiceAccount EOF
Create a model server and model deployment
This section shows how to deploy a model server and model. The example uses a
vLLM model server with a Llama3 model. The deployment is labeled as
app:vllm-llama3-8b-instruct. This deployment also uses two LoRA adapters
named food-review and cad-fabricator from Hugging Face.
You can adapt this example with your own model server container and model, serving port, and deployment name. You can also configure LoRA adapters in the deployment, or deploy the base model. The following steps describe how to create the necessary Kubernetes resources.
Create a Kubernetes Secret to store your Hugging Face token. This token is used to access the LoRA adapters:
kubectl create secret generic hf-token --from-literal=token=HF_TOKENReplace
HF_TOKENwith your Hugging Face token.To deploy on a
nvidia-h100-80gbaccelerator type, save the following manifest asvllm-llama3-8b-instruct.yaml. This manifest defines a Kubernetes Deployment with your model and model server:apiVersion: apps/v1 kind: Deployment metadata: name: vllm-llama3-8b-instruct spec: replicas: 3 selector: matchLabels: app: vllm-llama3-8b-instruct template: metadata: labels: app: vllm-llama3-8b-instruct spec: containers: - name: vllm image: "vllm/vllm-openai:latest" imagePullPolicy: Always command: ["python3", "-m", "vllm.entrypoints.openai.api_server"] args: - "--model" - "meta-llama/Llama-3.1-8B-Instruct" - "--tensor-parallel-size" - "1" - "--port" - "8000" - "--enable-lora" - "--max-loras" - "2" - "--max-cpu-loras" - "12" env: # Enabling LoRA support temporarily disables automatic v1, we want to force it on # until 0.8.3 vLLM is released. - name: PORT value: "8000" - name: HUGGING_FACE_HUB_TOKEN valueFrom: secretKeyRef: name: hf-token key: token - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING value: "true" ports: - containerPort: 8000 name: http protocol: TCP lifecycle: preStop: # vLLM stops accepting connections when it receives SIGTERM, so we need to sleep # to give upstream gateways a chance to take us out of rotation. The time we wait # is dependent on the time it takes for all upstreams to completely remove us from # rotation. Older or simpler load balancers might take upwards of 30s, but we expect # our deployment to run behind a modern gateway like Envoy which is designed to # probe for readiness aggressively. sleep: # Upstream gateway probers for health should be set on a low period, such as 5s, # and the shorter we can tighten that bound the faster that we release # accelerators during controlled shutdowns. However, we should expect variance, # as load balancers may have internal delays, and we don't want to drop requests # normally, so we're often aiming to set this value to a p99 propagation latency # of readiness -> load balancer taking backend out of rotation, not the average. # # This value is generally stable and must often be experimentally determined on # for a given load balancer and health check period. We set the value here to # the highest value we observe on a supported load balancer, and we recommend # tuning this value down and verifying no requests are dropped. # # If this value is updated, be sure to update terminationGracePeriodSeconds. # seconds: 30 # # IMPORTANT: preStop.sleep is beta as of Kubernetes 1.30 - for older versions # replace with this exec action. #exec: # command: # - /usr/bin/sleep # - 30 livenessProbe: httpGet: path: /health port: http scheme: HTTP # vLLM's health check is simple, so we can more aggressively probe it. Liveness # check endpoints should always be suitable for aggressive probing. periodSeconds: 1 successThreshold: 1 # vLLM has a very simple health implementation, which means that any failure is # likely significant. However, any liveness triggered restart requires the very # large core model to be reloaded, and so we should bias towards ensuring the # server is definitely unhealthy vs immediately restarting. Use 5 attempts as # evidence of a serious problem. failureThreshold: 5 timeoutSeconds: 1 readinessProbe: httpGet: path: /health port: http scheme: HTTP # vLLM's health check is simple, so we can more aggressively probe it. Readiness # check endpoints should always be suitable for aggressive probing, but may be # slightly more expensive than readiness probes. periodSeconds: 1 successThreshold: 1 # vLLM has a very simple health implementation, which means that any failure is # likely significant, failureThreshold: 1 timeoutSeconds: 1 # We set a startup probe so that we don't begin directing traffic or checking # liveness to this instance until the model is loaded. startupProbe: # Failure threshold is when we believe startup will not happen at all, and is set # to the maximum possible time we believe loading a model will take. In our # default configuration we are downloading a model from HuggingFace, which may # take a long time, then the model must load into the accelerator. We choose # 10 minutes as a reasonable maximum startup time before giving up and attempting # to restart the pod. # # IMPORTANT: If the core model takes more than 10 minutes to load, pods will crash # loop forever. Be sure to set this appropriately. failureThreshold: 3600 # Set delay to start low so that if the base model changes to something smaller # or an optimization is deployed, we don't wait unnecessarily. initialDelaySeconds: 2 # As a startup probe, this stops running and so we can more aggressively probe # even a moderately complex startup - this is a very important workload. periodSeconds: 1 httpGet: # vLLM does not start the OpenAI server (and hence make /health available) # until models are loaded. This may not be true for all model servers. path: /health port: http scheme: HTTP resources: limits: nvidia.com/gpu: 1 requests: nvidia.com/gpu: 1 volumeMounts: - mountPath: /data name: data - mountPath: /dev/shm name: shm - name: adapters mountPath: "/adapters" initContainers: - name: lora-adapter-syncer tty: true stdin: true image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/lora-syncer:main restartPolicy: Always imagePullPolicy: Always env: - name: DYNAMIC_LORA_ROLLOUT_CONFIG value: "/config/configmap.yaml" volumeMounts: # DO NOT USE subPath, dynamic configmap updates don't work on subPaths - name: config-volume mountPath: /config restartPolicy: Always # vLLM allows VLLM_PORT to be specified as an environment variable, but a user might # create a 'vllm' service in their namespace. That auto-injects VLLM_PORT in docker # compatible form as `tcp://<IP>:<PORT>` instead of the numeric value vLLM accepts # causing CrashLoopBackoff. Set service environment injection off by default. enableServiceLinks: false # Generally, the termination grace period needs to last longer than the slowest request # we expect to serve plus any extra time spent waiting for load balancers to take the # model server out of rotation. # # An easy starting point is the p99 or max request latency measured for your workload, # although LLM request latencies vary significantly if clients send longer inputs or # trigger longer outputs. Since steady state p99 will be higher than the latency # to drain a server, you may wish to slightly this value either experimentally or # via the calculation below. # # For most models you can derive an upper bound for the maximum drain latency as # follows: # # 1. Identify the maximum context length the model was trained on, or the maximum # allowed length of output tokens configured on vLLM (llama2-7b was trained to # 4k context length, while llama3-8b was trained to 128k). # 2. Output tokens are the more compute intensive to calculate and the accelerator # will have a maximum concurrency (batch size) - the time per output token at # maximum batch with no prompt tokens being processed is the slowest an output # token can be generated (for this model it would be about 100ms TPOT at a max # batch size around 50) # 3. Calculate the worst case request duration if a request starts immediately # before the server stops accepting new connections - generally when it receives # SIGTERM (for this model that is about 4096 / 10 ~ 40s) # 4. If there are any requests generating prompt tokens that will delay when those # output tokens start, and prompt token generation is roughly 6x faster than # compute-bound output token generation, so add 20% to the time from above (40s + # 16s ~ 55s) # # Thus we think it will take us at worst about 55s to complete the longest possible # request the model is likely to receive at maximum concurrency (highest latency) # once requests stop being sent. # # NOTE: This number will be lower than steady state p99 latency since we stop receiving # new requests which require continuous prompt token computation. # NOTE: The max timeout for backend connections from gateway to model servers should # be configured based on steady state p99 latency, not drain p99 latency # # 5. Add the time the pod takes in its preStop hook to allow the load balancers have # stopped sending us new requests (55s + 30s ~ 85s) # # Because termination grace period controls when the Kubelet forcibly terminates a # stuck or hung process (a possibility due to a GPU crash), there is operational safety # in keeping the value roughly proportional to the time to finish serving. There is also # value in adding a bit of extra time to deal with unexpectedly long workloads. # # 6. Add a 50% safety buffer to this time since the operational impact should be low # (85s * 1.5 ~ 130s) # # One additional source of drain latency is that some workloads may run close to # saturation and have queued requests on each server. Since traffic in excess of the # max sustainable QPS will result in timeouts as the queues grow, we assume that failure # to drain in time due to excess queues at the time of shutdown is an expected failure # mode of server overload. If your workload occasionally experiences high queue depths # due to periodic traffic, consider increasing the safety margin above to account for # time to drain queued requests. terminationGracePeriodSeconds: 130 nodeSelector: cloud.google.com/gke-accelerator: "nvidia-h100-80gb" volumes: - name: data emptyDir: {} - name: shm emptyDir: medium: Memory - name: adapters emptyDir: {} - name: config-volume configMap: name: vllm-llama3-8b-adapters --- apiVersion: v1 kind: ConfigMap metadata: name: vllm-llama3-8b-adapters data: configmap.yaml: | vLLMLoRAConfig: name: vllm-llama3.1-8b-instruct port: 8000 defaultBaseModel: meta-llama/Llama-3.1-8B-Instruct ensureExist: models: - id: food-review source: Kawon/llama3.1-food-finetune_v14_r8 - id: cad-fabricator source: redcathode/fabricatorApply the sample manifest to your cluster:
kubectl apply -f vllm-llama3-8b-instruct.yaml
After you apply the manifest, consider the following key fields and parameters:
replicas: specifies the number of Pods for the Deployment.image: specifies the Docker image for the model server.command: specifies the command to run when the container starts.args: specifies the arguments to pass to the command.env: specifies environment variables for the container.ports: specifies the ports exposed by the container.resources: specifies the resource requests and limits for the container, such as GPU.volumeMounts: specifies how volumes are mounted into the container.initContainers: specifies containers that run prior to the application container.restartPolicy: specifies the restart policy for the Pods.terminationGracePeriodSeconds: specifies the grace period for Pod termination.volumes: specifies the volumes used by the Pods.
You can modify these fields to match your specific requirements.
Create an inference pool
The InferencePool Kubernetes custom resource defines a group of Pods with a
common base large language model (LLM) and compute configuration. The
selector field specifies which Pods belong to this pool. The labels in this
selector must exactly match the labels applied to your model server Pods. The
targetPort field defines the ports that the model server uses within the Pods.
The extensionRef field references an extension service that provides
additional capability for the inference pool. The InferencePool enables
GKE Inference Gateway to route traffic to your model server
Pods.
Before you create the InferencePool, ensure that the Pods that the
InferencePool selects are already running.
To create an InferencePool using Helm, perform the following steps:
helm install vllm-llama3-8b-instruct \
--set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
--set provider.name=gke \
--version v0.3.0 \
oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool
Change the following field to match your Deployment:
inferencePool.modelServers.matchLabels.app: the key of the label used to select your model server Pods.
The Helm install automatically installs the necessary timeout policy, endpoint-picker and the Pods needed for observability.
This creates an InferencePool object: vllm-llama3-8b-instruct referencing the model endpoint services within the Pods. It also creates a deployment of the Endpoint Picker named app:vllm-llama3-8b-instruct-epp for this created InferencePool.
Specify model serving objectives
The InferenceModel custom resource defines a specific model to be served that
includes support for LoRA-tuned models and its serving criticality. You must
define which models are served on an InferencePool by creating
InferenceModel resources. These InferenceModel resources can reference base models, or LoRA adapters supported by the model servers in the InferencePool.
The modelName field specifies the name of the base model or LoRA adapter. The
Criticality field specifies the serving criticality of the model. The poolRef
field specifies the InferencePool on which this model is served.
To create an InferenceModel, perform the following steps:
Save the following sample manifest as
inferencemodel.yaml:apiVersion: inference.networking.x-k8s.io/v1alpha2 kind: InferenceModel metadata: name: inferencemodel-sample spec: modelName: MODEL_NAME criticality: VALUE poolRef: name: INFERENCE_POOL_NAMEReplace the following:
MODEL_NAME: the name of your base model or LoRA adapter. For example,food-review.VALUE: the chosen serving criticality. Choose fromCritical,Standard, orSheddable. For example,Standard.INFERENCE_POOL_NAME: the name of theInferencePoolyou created in the previous step. For example,vllm-llama3-8b-instruct.
Apply the sample manifest to your cluster:
kubectl apply -f inferencemodel.yaml
The following example creates an InferenceModel object that configures the
food-review LoRA model on the vllm-llama3-8b-instruct InferencePool with a
Standard serving criticality. The InferenceModel object also configures the
base model to be served with a Critical priority level.
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
name: food-review
spec:
modelName: food-review
criticality: Standard
poolRef:
name: vllm-llama3-8b-instruct
targetModels:
- name: food-review
weight: 100
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
name: llama3-base-model
spec:
modelName: meta-llama/Llama-3.1-8B-Instruct
criticality: Critical
poolRef:
name: vllm-llama3-8b-instruct
Create the Gateway
The Gateway resource is the entry point for external traffic into your Kubernetes cluster. It defines the listeners that accept incoming connections.
The GKE Inference Gateway works with the following Gateway Classes:
gke-l7-rilb: for regional internal Application Load Balancers.gke-l7-regional-external-managed
For more information, see Gateway Classes documentation.
To create a Gateway, perform the following steps:
Save the following sample manifest as
gateway.yaml:apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: GATEWAY_NAME spec: gatewayClassName: GATEWAY_CLASS listeners: - protocol: HTTP port: 80 name: httpReplace
GATEWAY_NAMEwith a unique name for your Gateway resource (for exampleinference-gateway), andGATEWAY_CLASSwith the Gateway Class you want to use (for example,gke-l7-regional-external-managed).Apply the manifest to your cluster:
kubectl apply -f gateway.yaml
Note: For more information about configuring TLS to secure your Gateway with HTTPS, see the GKE documentation on TLS configuration.
Create the HTTPRoute
The HTTPRoute resource defines how the GKE Gateway routes
incoming HTTP requests to backend services, which in this context would be your
InferencePool. The HTTPRoute resource specifies matching rules (for example, headers or paths) and the
backend to which traffic should be forwarded.
To create an
HTTPRoute, save the following sample manifest ashttproute.yaml:apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: HTTPROUTE_NAME spec: parentRefs: - name: GATEWAY_NAME rules: - matches: - path: type: PathPrefix value: PATH_PREFIX backendRefs: - name: INFERENCE_POOL_NAME group: inference.networking.x-k8s.io kind: InferencePoolReplace the following:
HTTPROUTE_NAME: a unique name for yourHTTPRouteresource. For example,my-route.GATEWAY_NAME: the name of theGatewayresource that you created. For example,inference-gateway.PATH_PREFIX: the path prefix that you use to match incoming requests. For example,/to match all.INFERENCE_POOL_NAME: the name of theInferencePoolresource that you want to route traffic to. For example,vllm-llama3-8b-instruct.
Apply the manifest to your cluster:
kubectl apply -f httproute.yaml
Send inference request
After you have configured GKE Inference Gateway, you can send inference requests to your deployed model. This lets you generate text based on your input prompt and specified parameters.
To send inference requests, perform the following steps:
To get the Gateway endpoint, run the following command:
IP=$(kubectl get gateway/GATEWAY_NAME -o jsonpath='{.status.addresses[0].value}') PORT=PORT_NUMBER # Use 80 for HTTPReplace the following:
GATEWAY_NAME: the name of your Gateway resource.PORT_NUMBER: the port number you configured in the Gateway.
To send a request to the
/v1/completionsendpoint usingcurl, run the following command:curl -i -X POST ${IP}:${PORT}/v1/completions \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer $(gcloud auth print-access-token)' \ -d '{ "model": "MODEL_NAME", "prompt": "PROMPT_TEXT", "max_tokens": MAX_TOKENS, "temperature": "TEMPERATURE" }'Replace the following:
MODEL_NAME: the name of the model or LoRA adapter to use.PROMPT_TEXT: the input prompt for the model.MAX_TOKENS: the maximum number of tokens to generate in the response.TEMPERATURE: controls the randomness of the output. Use the value0for deterministic output, or a higher number for more creative output.
The following example shows you how to send a sample request to GKE Inference Gateway:
curl -i -X POST ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -H 'Authorization: Bearer $(gcloud auth print-access-token)' -d '{
"model": "food-review",
"prompt": "What is the best pizza in the world?",
"max_tokens": 2048,
"temperature": "0"
}'
Be aware of the following behaviours:
- Request body: the request body can include additional parameters like
stopandtop_p. Refer to the OpenAI API specification for a complete list of options. - Error handling: implement proper error handling in your client code to
handle potential errors in the response. For example, check the HTTP status
code in the
curlresponse. A non-200status code generally indicates an error. - Authentication and authorization: for production deployments, secure your
API endpoint with authentication and authorization mechanisms. Include the
appropriate headers (for example,
Authorization) in your requests.