Disponibilizar modelos abertos do Gemma usando GPUs no GKE com o TGI Hugging Face

Piloto automático

No Cloud Shell, execute este comando:

gcloud container clusters create-auto ${CLUSTER_NAME} \
  --project=${PROJECT_ID} \
  --region=${REGION} \
  --release-channel=rapid

O GKE cria um cluster do Autopilot com nós de CPU e GPU conforme solicitado pelas cargas de trabalho implantadas.

Standard

1. No Cloud Shell, execute o seguinte comando para criar um cluster Standard:

   gcloud container clusters create ${CLUSTER_NAME} \
     --project=${PROJECT_ID} \
     --region=${REGION} \
     --workload-pool=${PROJECT_ID}.svc.id.goog \
     --release-channel=rapid \
     --num-nodes=1
   

   A criação do cluster pode levar vários minutos.

2. Execute o seguinte comando para criar um pool de nós para o cluster:

   Gemma 2 2B

   gcloud container node-pools create gpupool \
     --accelerator type=nvidia-l4,count=1,gpu-driver-version=latest \
     --project=${PROJECT_ID} \
     --location=${REGION} \
     --node-locations=${REGION}-a \
     --cluster=${CLUSTER_NAME} \
     --machine-type=g2-standard-8 \
     --num-nodes=1
   

   O GKE cria um único pool de nós contendo uma GPU L4 para cada nó.

   Gemma 2 9B

   gcloud container node-pools create gpupool \
     --accelerator type=nvidia-l4,count=2,gpu-driver-version=latest \
     --project=${PROJECT_ID} \
     --location=${REGION} \
     --node-locations=${REGION}-a \
     --cluster=${CLUSTER_NAME} \
     --machine-type=g2-standard-24 \
     --num-nodes=1
   

   O GKE cria um único pool de nós contendo duas GPUs L4 para cada nó.

   Gemma 2 27B

   gcloud container node-pools create gpupool \
     --accelerator type=nvidia-l4,count=4,gpu-driver-version=latest \
     --project=${PROJECT_ID} \
     --location=${REGION} \
     --node-locations=${REGION}-a \
     --cluster=${CLUSTER_NAME} \
     --machine-type=g2-standard-48 \
     --num-nodes=1
   

   O GKE cria um único pool de nós contendo quatro GPUs L4 para cada nó.

Gemma 2 2B-it

Siga estas instruções para implantar o modelo ajustado por instruções do Gemma 2B.

1. Crie o seguinte manifesto tgi-2-2b-it.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: tgi-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2-2b-it
           ai.gke.io/inference-server: text-generation-inference
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu121.2-1.ubuntu2204.py310
           resources:
             requests:
               cpu: "2"
               memory: "10Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: "1"
             limits:
               cpu: "2"
               memory: "10Gi"
               ephemeral-storage: "10Gi"
               nvidia.com/gpu: "1"
           env:
           - name: AIP_HTTP_PORT
             value: '8000'
           - name: NUM_SHARD
             value: '1'
           - name: MAX_INPUT_LENGTH
             value: '1562'
           - name: MAX_TOTAL_TOKENS
             value: '2048'
           - name: MAX_BATCH_PREFILL_TOKENS
             value: '2048'
           - name: CUDA_MEMORY_FRACTION
             value: '0.93'
           - name: MODEL_ID
             value: google/gemma-2-2b-it
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
             medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
     - protocol: TCP
       port: 8000
       targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f tgi-2-2b-it.yaml
   

Gemma 2 9B-it

Siga estas instruções para implantar o modelo ajustado por instruções do Gemma 9B.

1. Crie o seguinte manifesto tgi-2-9b-it.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: tgi-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2-9b-it
           ai.gke.io/inference-server: text-generation-inference
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu121.2-1.ubuntu2204.py310
           resources:
             requests:
               cpu: "4"
               memory: "30Gi"
               ephemeral-storage: "30Gi"
               nvidia.com/gpu: "2"
             limits:
               cpu: "4"
               memory: "30Gi"
               ephemeral-storage: "30Gi"
               nvidia.com/gpu: "2"
           env:
           - name: AIP_HTTP_PORT
             value: '8000'
           - name: NUM_SHARD
             value: '2'
           - name: MAX_INPUT_LENGTH
             value: '1562'
           - name: MAX_TOTAL_TOKENS
             value: '2048'
           - name: MAX_BATCH_PREFILL_TOKENS
             value: '2048'
           - name: CUDA_MEMORY_FRACTION
             value: '0.93'
           - name: MODEL_ID
             value: google/gemma-2-9b-it
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
             medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
     - protocol: TCP
       port: 8000
       targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f tgi-2-9b-it.yaml
   

Gemma 2 27B-it

Siga estas instruções para implantar o modelo ajustado por instruções do Gemma 2 27B.

1. Crie o seguinte manifesto tgi-2-27b-it.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: tgi-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: gemma-2-27b-it
           ai.gke.io/inference-server: text-generation-inference
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu121.2-1.ubuntu2204.py310
           resources:
             requests:
               cpu: "10"
               memory: "80Gi"
               ephemeral-storage: "80Gi"
               nvidia.com/gpu: "4"
             limits:
               cpu: "10"
               memory: "80Gi"
               ephemeral-storage: "80Gi"
               nvidia.com/gpu: "4"
           env:
           - name: AIP_HTTP_PORT
             value: '8000'
           - name: NUM_SHARD
             value: '4'
           - name: MAX_INPUT_LENGTH
             value: '1562'
           - name: MAX_TOTAL_TOKENS
             value: '2048'
           - name: MAX_BATCH_PREFILL_TOKENS
             value: '2048'
           - name: CUDA_MEMORY_FRACTION
             value: '0.93'
           - name: MODEL_ID
             value: google/gemma-2-27b-it
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
             medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
     - protocol: TCP
       port: 8000
       targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f tgi-2-27b-it.yaml
   

CodeGemma 7B-it

Siga estas instruções para implantar o modelo ajustado por instruções do CodeGemma 7B.

1. Crie o seguinte manifesto tgi-codegemma-1.1-7b-it.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: tgi-gemma-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: gemma-server
     template:
       metadata:
         labels:
           app: gemma-server
           ai.gke.io/model: codegemma-1.1-7b-it
           ai.gke.io/inference-server: text-generation-inference
           examples.ai.gke.io/source: user-guide
       spec:
         containers:
         - name: inference-server
           image: ghcr.io/huggingface/text-generation-inference:2.0.2
           resources:
             requests:
               cpu: "2"
               memory: "25Gi"
               ephemeral-storage: "40Gi"
               nvidia.com/gpu: 2
             limits:
               cpu: "10"
               memory: "25Gi"
               ephemeral-storage: "40Gi"
               nvidia.com/gpu: 2
           args:
           - --model-id=$(MODEL_ID)
           - --num-shard=2
           env:
           - name: MODEL_ID
             value: google/codegemma-1.1-7b-it
           - name: PORT
             value: "8000"
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - name: dshm
           emptyDir:
             medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-l4
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-service
   spec:
     selector:
       app: gemma-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000

2. Aplique o manifesto:

   kubectl apply -f tgi-codegemma-1.1-7b-it.yaml
   

Gemma 2

Em uma nova sessão do terminal, use curl para conversar com seu modelo:

USER_PROMPT="I'm new to coding. If you could only recommend one programming language to start with, what would it be and why?"

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
    "inputs": "<start_of_turn>user\n${USER_PROMPT}<end_of_turn>\n",
    "parameters": {
        "temperature": 0.90,
        "top_p": 0.95,
        "max_new_tokens": 128
    }
}
EOF

A saída a seguir mostra um exemplo da resposta do modelo:

{"generated_text":"**Python**\n\n**Reasons why Python is a great choice for beginners:**\n\n* **Simple syntax:** Python uses clear and concise syntax, making it easy for beginners to pick up.\n* **Easy to learn:** Python's syntax is based on English, making it easier to learn than other languages.\n* **Large and supportive community:** Python has a massive and active community of developers who are constantly willing to help.\n* **Numerous libraries and tools:** Python comes with a vast collection of libraries and tools that make it easy to perform various tasks, such as data manipulation, web development, and machine learning.\n* **"}

CodeGemma

Em uma nova sessão do terminal, use curl para conversar com seu modelo:

USER_PROMPT="Generate a python code example of a adding two numbers from a function called addNumbers"

curl -s -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d @- <<EOF | jq -r .generated_text
{
    "inputs": "<start_of_turn>user\n${USER_PROMPT}<end_of_turn>\n",
    "parameters": {
        "temperature": 0.90,
        "top_p": 0.95,
        "max_new_tokens": 2000
    }
}
EOF

A saída a seguir mostra um exemplo da resposta do modelo:

def addNumbers(num1, num2):
  sum = num1 + num2
  return sum

# Get the input from the user
num1 = float(input("Enter the first number: "))
num2 = float(input("Enter the second number: "))

# Call the addNumbers function
sum = addNumbers(num1, num2)

# Print the result
print("The sum of", num1, "and", num2, "is", sum)