Disponibilizar modelos abertos do Gemma usando GPUs no GKE com vLLM

Autopilot

No Cloud Shell, execute este comando:

gcloud container clusters create-auto ${CLUSTER_NAME} \
  --project=${PROJECT_ID} \
  --region=${REGION} \
  --release-channel=rapid \
  --cluster-version=1.28

O GKE cria um cluster do Autopilot com nós de CPU e GPU conforme solicitado pelas cargas de trabalho implantadas.

Standard

1. No Cloud Shell, execute o seguinte comando para criar um cluster Standard:

   gcloud container clusters create ${CLUSTER_NAME} \
     --project=${PROJECT_ID} \
     --region=${REGION} \
     --workload-pool=${PROJECT_ID}.svc.id.goog \
     --release-channel=rapid \
     --num-nodes=1
   

   A criação do cluster pode levar vários minutos.

2. Execute o seguinte comando para criar um pool de nós para o cluster:

   gcloud container node-pools create gpupool \
     --accelerator type=nvidia-l4,count=2,gpu-driver-version=latest \
     --project=${PROJECT_ID} \
     --location=${REGION} \
     --node-locations=${REGION}-a \
     --cluster=${CLUSTER_NAME} \
     --machine-type=g2-standard-24 \
     --num-nodes=1
   

   O GKE cria um único pool de nós contendo duas GPUs L4 para cada nó.

Gemma 2B-it

Siga estas instruções para implantar o modelo ajustado por instruções do Gemma 2B.

1. Crie o seguinte manifesto vllm-2b-it.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2b-it
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240220_0936_RC01
           resources:
             requests:
               cpu: "2"
               memory: "7Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: 1
             limits:
               cpu: "2"
               memory: "7Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: 1
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=1
           env:
           - name: MODEL_ID
             value: google/gemma-2b-it
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f vllm-2b-it.yaml
   

Gemma 7B-it

Siga estas instruções para implantar o modelo ajustado por instruções do Gemma 7B.

1. Crie o seguinte manifesto vllm-7b-it.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-7b-it
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240220_0936_RC01
           resources:
             requests:
               cpu: "2"
               memory: "25Gi"
               ephemeral-storage: "25Gi"
               nvidia.com/gpu: 2
             limits:
               cpu: "2"
               memory: "25Gi"
               ephemeral-storage: "25Gi"
               nvidia.com/gpu: 2
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=2
           env:
           - name: MODEL_ID
             value: google/gemma-7b-it
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f vllm-7b-it.yaml
   

Gemma 2B

Siga estas instruções para implantar o modelo pré-treinado do Gemma 2B.

1. Crie o seguinte manifesto vllm-2b.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2b
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240220_0936_RC01
           resources:
             requests:
               cpu: "2"
               memory: "7Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: 1
             limits:
               cpu: "2"
               memory: "7Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: 1
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=1
           env:
           - name: MODEL_ID
             value: google/gemma-2b
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f vllm-2b.yaml
   

Gemma 7B

Siga estas instruções para implantar o modelo pré-treinado do Gemma 7B.

1. Crie o seguinte manifesto vllm-7b.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-7b
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240220_0936_RC01
           resources:
             requests:
               cpu: "2"
               memory: "25Gi"
               ephemeral-storage: "25Gi"
               nvidia.com/gpu: 2
             limits:
               cpu: "2"
               memory: "25Gi"
               ephemeral-storage: "25Gi"
               nvidia.com/gpu: 2
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --tensor-parallel-size=2
           env:
           - name: MODEL_ID
             value: google/gemma-7b
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f vllm-7b.yaml
   

Pré-treinado (2B)

Em uma nova sessão do terminal, use curl para conversar com seu modelo:

USER_PROMPT="Java is a"

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
    "prompt": "${USER_PROMPT}",
    "temperature": 0.90,
    "top_p": 1.0,
    "max_tokens": 128
}
EOF

A saída a seguir mostra um exemplo da resposta do modelo:

{"predictions":["Prompt:\nJava is a\nOutput:\n<strong>programming language</strong> that is primarily aimed at developers. It was originally created by creators of the Java Virtual Machine (JVM). Java is multi-paradigm, which means it supports object-oriented, procedural, and functional programming paradigms. Java is object-oriented, which means that it is designed to support classes and objects. Java is a dynamically typed language, which means that the type of variables are not determined at compile time. Java is also a multi-paradigm language, which means it supports more than one programming paradigm. Java is also a very lightweight language, which means that it is a very low level language compared to other popular"]}

Instruções ajustadas (2B-it)

Em uma nova sessão do terminal, use curl para conversar com seu modelo:

USER_PROMPT="I'm new to coding. If you could only recommend one programming language to start with, what would it be and why?"

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
    "prompt": "<start_of_turn>user\n${USER_PROMPT}<end_of_turn>\n",
    "temperature": 0.90,
    "top_p": 1.0,
    "max_tokens": 128
}
EOF

A saída a seguir mostra um exemplo da resposta do modelo:

{"predictions":["Prompt:\n<start_of_turn>user\nI'm new to coding. If you could only recommend one programming language to start with, what would it be and why?<end_of_turn>\nOutput:\n**Python** is an excellent choice for beginners due to the following reasons:\n\n* **Clear and simple syntax:** Python boasts a simple and straightforward syntax that makes it easy to learn the fundamentals of programming.\n* **Extensive libraries and modules:** Python comes with a vast collection of libraries and modules that address various programming tasks, including data manipulation, machine learning, and web development.\n* **Large and supportive community:** Python has a vibrant and active community that offers resources, tutorials, and support to help you along your journey.\n* **Cross-platform compatibility:** Python can be run on various platforms, including Windows, macOS, and"]}