Deploy and serve Gemma 3 27B inference with vLLM on GKE

Access Gemma 3 by using Hugging Face

To use Hugging Face to access Gemma 3, follow these steps:

1. Sign the consent agreement to use Gemma 3 27B.

2. Create a read access token.

Prepare your environment

To prepare your environment, set the default environment variables:

gcloud config set project PROJECT_ID
gcloud config set billing/quota_project PROJECT_ID
export PROJECT_ID=$(gcloud config get project)
export RESERVATION_URL=RESERVATION_URL
export REGION=REGION
export CLUSTER_NAME=CLUSTER_NAME
export HUGGING_FACE_TOKEN=HUGGING_FACE_TOKEN
export NETWORK=NETWORK_NAME
export SUBNETWORK=SUBNETWORK_NAME

Replace the following:

• PROJECT_ID: the ID of the Google Cloud project where you want to create the GKE cluster.

• RESERVATION_URL: the URL of the reservation that you want to use to create your GKE cluster. Based on the project in which the reservation exists, specify one of the following values:

  • The reservation exists in your project: RESERVATION_NAME

  • The reservation exists in a different project, and your project can use the reservation: projects/RESERVATION_PROJECT_ID/reservations/RESERVATION_NAME

• REGION: the region where you want to create your GKE cluster. You can only create the cluster in the region where you reservation exists.

• CLUSTER_NAME: the name of the GKE cluster to create.

• HUGGING_FACE_TOKEN: the Hugging Face access token that you created in the previous section.

• NETWORK_NAME: the network that the GKE cluster uses. Specify one of the following values:

  • If you created a custom network, then specify the name of your network.

  • Otherwise, specify default.

• SUBNETWORK_NAME: the subnetwork that the GKE cluster uses. Specify one of the following values:

  • If you created a custom subnetwork, then specify the name of your subnetwork. You can only specify a subnetwork that exists in the same region as the reservation.

  • Otherwise, specify default.

Create a GKE cluster in Autopilot mode

To create a GKE cluster in Autopilot mode, run the following command:

gcloud container clusters create-auto $CLUSTER_NAME \
    --project=$PROJECT_ID \
    --region=$REGION \
    --release-channel=rapid \
    --network=$NETWORK \
    --subnetwork=$SUBNETWORK

Creating the GKE cluster might take some time to complete. To verify when Google Cloud finishes creating your cluster, go to Kubernetes clusters on the Google Cloud console.

Create a Kubernetes secret for Hugging Face credentials

To create a Kubernetes secret for Hugging Face credentials, follow these steps:

1. Configure kubectl to communicate with your GKE cluster:

   gcloud container clusters get-credentials $CLUSTER_NAME \
       --location=$REGION
   

2. Create a Kubernetes secret to store your Hugging Face token:

   kubectl create secret generic hf-secret \
       --from-literal=hf_api_token=${HUGGING_FACE_TOKEN} \
       --dry-run=client -o yaml | kubectl apply -f -
   

Deploy a vLLM container to your GKE cluster

To use Kubernetes Deployments to deploy the vLLM container to serve the Gemma 3 27B model, follow these steps:

1. Create a vllm-3-27b-it.yaml file with your chosen vLLM deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-3-27b-it
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250722_0916_RC01
           resources:
             requests:
               cpu: "10"
               memory: "128Gi"
               ephemeral-storage: "120Gi"
               nvidia.com/gpu : "8"
             limits:
               cpu: "10"
               memory: "128Gi"
               ephemeral-storage: "120Gi"
               nvidia.com/gpu : "8"
           command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=8
           - --host=0.0.0.0
           - --port=8000
           - --max-model-len=4096
           - --max-num-seqs=4
           env:
           - name: MODEL_ID
             value: google/gemma-3-27b-it
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
           livenessProbe:
             httpGet:
               path: /health
               port: 8000
             initialDelaySeconds: 600
             periodSeconds: 10
           readinessProbe:
             httpGet:
               path: /health
               port: 8000
             initialDelaySeconds: 600
             periodSeconds: 5
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-b200
           cloud.google.com/reservation-name: RESERVATION_URL
           cloud.google.com/reservation-affinity: "specific"
           cloud.google.com/gke-gpu-driver-version: latest
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000
   

2. Apply the vllm-3-27b-it.yaml file to your GKE cluster:

   kubectl apply -f vllm-3-27b-it.yaml
   

   During the deployment process, the container must download Gemma 3 from Hugging Face. For this reason, deployment of the container might take up to 30 minutes to complete.

3. Wait for the deployment to complete:

   kubectl wait \
       --for=condition=Available \
       --timeout=1800s deployment/vllm-gemma-deployment
   

Interact with Gemma 3 by using curl

To verify your deployed Gemma 3 27B instruction-tuned models, follow these steps:

1. Set up port forwarding to Gemma 3:

   kubectl port-forward service/llm-service 8000:8000
   

2. Open a new terminal window. You can then chat with your model by using curl:

   curl http://127.0.0.1:8000/v1/chat/completions \
   -X POST \
   -H "Content-Type: application/json" \
   -d '{
     "model": "google/gemma-3-27b-it",
     "messages": [
       {
         "role": "user",
         "content": "Why is the sky blue?"
       }
     ]
   }'
   

   The output is similar to the following:

   {
     "id": "chatcmpl-e4a2e624bea849d9b09f838a571c4d9e",
     "object": "chat.completion",
     "created": 1741763029,
     "model": "google/gemma-3-27b-it",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "reasoning_content": null,
           "content": "Okay, let's break down why the sky appears blue! It's a fascinating phenomenon rooted in physics, specifically something called **Rayleigh scattering**. Here's the explanation: ...",
           "tool_calls": []
         },
         "logprobs": null,
         "finish_reason": "stop",
         "stop_reason": 106
       }
     ],
     "usage": {
       "prompt_tokens": 15,
       "total_tokens": 668,
       "completion_tokens": 653,
       "prompt_tokens_details": null
     },
     "prompt_logprobs": null
   }
   

If you want to observe your model's performance, then you can use the vLLM dashboard integration in Cloud Monitoring. This dashboard helps you view critical performance metrics for your model like token throughput, network latency, and error rates. For information, see vLLM in the Monitoring documentation.