vLLM과 함께 GKE에서 GPU를 사용하여 Gemma 개방형 모델 제공

Autopilot

Cloud Shell에서 다음 명령어를 실행합니다.

gcloud container clusters create-auto ${CLUSTER_NAME} \
  --project=${PROJECT_ID} \
  --region=${REGION} \
  --release-channel=rapid

GKE는 배포된 워크로드의 요청에 따라 CPU 및 GPU 노드를 사용하여 Autopilot 클러스터를 만듭니다.

표준

1. Cloud Shell에서 다음 명령어를 실행하여 Standard 클러스터를 만듭니다.

   gcloud container clusters create ${CLUSTER_NAME} \
     --project=${PROJECT_ID} \
     --region=${REGION} \
     --workload-pool=${PROJECT_ID}.svc.id.goog \
     --release-channel=rapid \
     --num-nodes=1
   

   클러스터 만들기는 몇 분 정도 걸릴 수 있습니다.

2. 다음 명령어를 실행하여 클러스터에 대해 노드 풀을 만듭니다.

   Gemma 2 2B

   gcloud container node-pools create gpupool \
     --accelerator type=nvidia-l4,count=1,gpu-driver-version=latest \
     --project=${PROJECT_ID} \
     --location=${REGION} \
     --node-locations=${REGION}-a \
     --cluster=${CLUSTER_NAME} \
     --machine-type=g2-standard-8 \
     --num-nodes=1
   

   GKE가 각 노드에 L4 GPU가 포함된 단일 노드 풀을 만듭니다.

   Gemma 2 9B

   gcloud container node-pools create gpupool \
     --accelerator type=nvidia-l4,count=2,gpu-driver-version=latest \
     --project=${PROJECT_ID} \
     --location=${REGION} \
     --node-locations=${REGION}-a \
     --cluster=${CLUSTER_NAME} \
     --machine-type=g2-standard-24 \
     --num-nodes=1
   

   GKE는 각 노드에 2개의 L4 GPU를 포함하는 단일 노드 풀을 만듭니다.

   Gemma 2 27B

   gcloud container node-pools create gpupool \
     --accelerator type=nvidia-l4,count=4,gpu-driver-version=latest \
     --project=${PROJECT_ID} \
     --location=${REGION} \
     --node-locations=${REGION}-a \
     --cluster=${CLUSTER_NAME} \
     --machine-type=g2-standard-48 \
     --num-nodes=1
   

   GKE가 각 노드에 4개의 L4 GPU가 포함된 단일 노드 풀을 만듭니다.

Gemma 2 2B-it

다음 안내에 따라 Gemma 2 2B 명령 조정 모델을 배포합니다.

1. 다음 vllm-2-2b-it.yaml 매니페스트를 만듭니다.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2-2b-it
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image:  vllm/vllm-openai:v0.5.5
           resources:
             requests:
               cpu: "2"
               memory: "10Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: "1"
             limits:
               cpu: "2"
               memory: "10Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: "1"
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=1
           env:
           - name: MODEL_ID
             value: google/gemma-2-2b-it
           - name: VLLM_ATTENTION_BACKEND
             value: FLASHINFER
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
           cloud.google.com/gke-gpu-driver-version: latest
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. 매니페스트를 적용합니다.

   kubectl apply -f vllm-2-2b-it.yaml
   

Gemma 2 9B-it

다음 안내에 따라 Gemma 2 9B 명령 조정 모델을 배포합니다.

1. 다음 vllm-2-9b-it.yaml 매니페스트를 만듭니다.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2-9b-it
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image:  vllm/vllm-openai:v0.5.5
           resources:
             requests:
               cpu: "4"
               memory: "30Gi"
               ephemeral-storage: "30Gi"
               nvidia.com/gpu: "2"
             limits:
               cpu: "4"
               memory: "30Gi"
               ephemeral-storage: "30Gi"
               nvidia.com/gpu: "2"
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=2
           env:
           - name: MODEL_ID
             value: google/gemma-2-9b-it
           - name: VLLM_ATTENTION_BACKEND
             value: FLASHINFER
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
           cloud.google.com/gke-gpu-driver-version: latest
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. 매니페스트를 적용합니다.

   kubectl apply -f vllm-2-9b-it.yaml
   

Gemma 2 27B-it

다음 안내에 따라 Gemma 2 27B 명령 조정 모델을 배포합니다.

1. 다음 vllm-2-27b-it.yaml 매니페스트를 만듭니다.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2-27b-it
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image:  vllm/vllm-openai:v0.5.5
           resources:
             requests:
               cpu: "10"
               memory: "80Gi"
               ephemeral-storage: "80Gi"
               nvidia.com/gpu: "4"
             limits:
               cpu: "10"
               memory: "80Gi"
               ephemeral-storage: "80Gi"
               nvidia.com/gpu: "4"
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=4
           env:
           - name: MODEL_ID
             value: google/gemma-2-27b-it
           - name: VLLM_ATTENTION_BACKEND
             value: FLASHINFER
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
           cloud.google.com/gke-gpu-driver-version: latest
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. 매니페스트를 적용합니다.

   kubectl apply -f vllm-2-27b-it.yaml