Disponibilizar LLMs como DeepSeek-R1 671B ou Llama 3.1 405B no GKE

Criar um cluster do GKE

É possível exibir modelos usando o vLLM em vários nós de GPU em um cluster do GKE Autopilot ou Standard. Recomendamos que você use um cluster do Autopilot para ter uma experiência totalmente gerenciada do Kubernetes. Para escolher o modo de operação do GKE mais adequado para suas cargas de trabalho, consulte Escolher um modo de operação do GKE.

  gcloud container clusters create-auto ${CLUSTER_NAME} \
    --project=${PROJECT_ID} \
    --region=${REGION} \
    --cluster-version=${CLUSTER_VERSION}

Configure kubectl para se comunicar com o cluster

Configure kubectl para se comunicar com o cluster com o seguinte comando:

gcloud container clusters get-credentials CLUSTER_NAME --location=ZONE

Criar um Secret do Kubernetes para as credenciais do Hugging Face

Crie um secret do Kubernetes que contenha o token do Hugging Face usando o seguinte comando:

kubectl create secret generic hf-secret \
  --from-literal=hf_api_token=${HF_TOKEN} \
  --dry-run=client -o yaml | kubectl apply -f -

Instalar o LeaderWorkerSet

Para instalar o LWS, execute este comando:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/lws/releases/latest/download/manifests.yaml

Valide se o controlador LeaderWorkerSet está em execução no namespace lws-system usando o seguinte comando:

kubectl get pod -n lws-system

O resultado será assim:

NAME                                     READY   STATUS    RESTARTS   AGE
lws-controller-manager-546585777-crkpt   1/1     Running   0          4d21h
lws-controller-manager-546585777-zbt2l   1/1     Running   0          4d21h

Implantar o servidor de modelos do vLLM

Para implantar o servidor de modelos vLLM, siga estas etapas:

1. Aplique o manifesto, dependendo do LLM que você quer implantar.

   DeepSeek-R1

   1. Inspecione o manifesto vllm-deepseek-r1-A3.yaml.

      
      apiVersion: leaderworkerset.x-k8s.io/v1
      kind: LeaderWorkerSet
      metadata:
        name: vllm
      spec:
        replicas: 1
        leaderWorkerTemplate:
          size: 2
          restartPolicy: RecreateGroupOnPodRestart
          leaderTemplate:
            metadata:
              labels:
                role: leader
            spec:
              nodeSelector:
                cloud.google.com/gke-accelerator: nvidia-h100-80gb
              containers:
                - name: vllm-leader
                  image: docker.io/vllm/vllm-openai:latest
                  env:
                    - name: HUGGING_FACE_HUB_TOKEN
                      valueFrom:
                        secretKeyRef:
                          name: hf-secret
                          key: hf_api_token
                  command:
                    - sh
                    - -c
                    - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
                      python3 -m vllm.entrypoints.openai.api_server --port 8080 --model deepseek-ai/DeepSeek-R1 --tensor-parallel-size 8 --pipeline-parallel-size 2 --trust-remote-code --max-model-len 4096"
                  resources:
                    limits:
                      nvidia.com/gpu: "8"
                  ports:
                    - containerPort: 8080
                  readinessProbe:
                    tcpSocket:
                      port: 8080
                    initialDelaySeconds: 15
                    periodSeconds: 10
                  volumeMounts:
                    - mountPath: /dev/shm
                      name: dshm
              volumes:
              - name: dshm
                emptyDir:
                  medium: Memory
                  sizeLimit: 15Gi
          workerTemplate:
            spec:
              nodeSelector:
                cloud.google.com/gke-accelerator: nvidia-h100-80gb
              containers:
                - name: vllm-worker
                  image: docker.io/vllm/vllm-openai:latest
                  command:
                    - sh
                    - -c
                    - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
                  resources:
                    limits:
                      nvidia.com/gpu: "8"
                  env:
                    - name: HUGGING_FACE_HUB_TOKEN
                      valueFrom:
                        secretKeyRef:
                          name: hf-secret
                          key: hf_api_token
                  volumeMounts:
                    - mountPath: /dev/shm
                      name: dshm   
              volumes:
              - name: dshm
                emptyDir:
                  medium: Memory
                  sizeLimit: 15Gi
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: vllm-leader
      spec:
        ports:
          - name: http
            port: 8080
            protocol: TCP
            targetPort: 8080
        selector:
          leaderworkerset.sigs.k8s.io/name: vllm
          role: leader
        type: ClusterIP
      

   2. Execute este comando para aplicar o manifesto:

      kubectl apply -f vllm-deepseek-r1-A3.yaml
      

   Llama 3.1 405B

   1. Inspecione o manifesto vllm-llama3-405b-A3.yaml.

      
      apiVersion: leaderworkerset.x-k8s.io/v1
      kind: LeaderWorkerSet
      metadata:
        name: vllm
      spec:
        replicas: 1
        leaderWorkerTemplate:
          size: 2
          restartPolicy: RecreateGroupOnPodRestart
          leaderTemplate:
            metadata:
              labels:
                role: leader
            spec:
              nodeSelector:
                cloud.google.com/gke-accelerator: nvidia-h100-80gb
              containers:
                - name: vllm-leader
                  image: docker.io/vllm/vllm-openai:latest
                  env:
                    - name: HUGGING_FACE_HUB_TOKEN
                      valueFrom:
                        secretKeyRef:
                          name: hf-secret
                          key: hf_api_token
                  command:
                    - sh
                    - -c
                    - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
                      python3 -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2"
                  resources:
                    limits:
                      nvidia.com/gpu: "8"
                  ports:
                    - containerPort: 8080
                  readinessProbe:
                    tcpSocket:
                      port: 8080
                    initialDelaySeconds: 15
                    periodSeconds: 10
                  volumeMounts:
                    - mountPath: /dev/shm
                      name: dshm
              volumes:
              - name: dshm
                emptyDir:
                  medium: Memory
                  sizeLimit: 15Gi
          workerTemplate:
            spec:
              nodeSelector:
                cloud.google.com/gke-accelerator: nvidia-h100-80gb
              containers:
                - name: vllm-worker
                  image: docker.io/vllm/vllm-openai:latest
                  command:
                    - sh
                    - -c
                    - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
                  resources:
                    limits:
                      nvidia.com/gpu: "8"
                  env:
                    - name: HUGGING_FACE_HUB_TOKEN
                      valueFrom:
                        secretKeyRef:
                          name: hf-secret
                          key: hf_api_token
                  volumeMounts:
                    - mountPath: /dev/shm
                      name: dshm   
              volumes:
              - name: dshm
                emptyDir:
                  medium: Memory
                  sizeLimit: 15Gi
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: vllm-leader
      spec:
        ports:
          - name: http
            port: 8080
            protocol: TCP
            targetPort: 8080
        selector:
          leaderworkerset.sigs.k8s.io/name: vllm
          role: leader
        type: ClusterIP
      

   2. Execute este comando para aplicar o manifesto:

      kubectl apply -f vllm-llama3-405b-A3.yaml
      

2. Confira os registros do servidor de modelos em execução com o seguinte comando:

   kubectl logs vllm-0 -c vllm-leader
   

   A saída será parecida com esta:

   INFO 08-09 21:01:34 api_server.py:297] Route: /detokenize, Methods: POST
   INFO 08-09 21:01:34 api_server.py:297] Route: /v1/models, Methods: GET
   INFO 08-09 21:01:34 api_server.py:297] Route: /version, Methods: GET
   INFO 08-09 21:01:34 api_server.py:297] Route: /v1/chat/completions, Methods: POST
   INFO 08-09 21:01:34 api_server.py:297] Route: /v1/completions, Methods: POST
   INFO 08-09 21:01:34 api_server.py:297] Route: /v1/embeddings, Methods: POST
   INFO:     Started server process [7428]
   INFO:     Waiting for application startup.
   INFO:     Application startup complete.
   INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
   

Disponibilizar o modelo

Configure o encaminhamento de portas para o modelo executando o seguinte comando:

kubectl port-forward svc/vllm-leader 8080:8080

Interagir com o modelo usando curl

Para interagir com o modelo usando curl, siga estas instruções:

curl http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "deepseek-ai/DeepSeek-R1",
    "prompt": "I have four boxes. I put the red box on the bottom and put the blue box on top. Then I put the yellow box on top the blue. Then I take the blue box out and put it on top. And finally I put the green box on the top. Give me the final order of the boxes from bottom to top. Show your reasoning but be brief",
    "max_tokens": 1024,
    "temperature": 0
}'

{
"id": "cmpl-f2222b5589d947419f59f6e9fe24c5bd",
"object": "text_completion",
"created": 1738269669,
"model": "deepseek-ai/DeepSeek-R1",
"choices": [
  {
    "index": 0,
    "text": ".\n\nOkay, let's see. The user has four boxes and is moving them around. Let me try to visualize each step. \n\nFirst, the red box is placed on the bottom. So the stack starts with red. Then the blue box is put on top of red. Now the order is red (bottom), blue. Next, the yellow box is added on top of blue. So now it's red, blue, yellow. \n\nThen the user takes the blue box out. Wait, blue is in the middle. If they remove blue, the stack would be red and yellow. But where do they put the blue box? The instruction says to put it on top. So after removing blue, the stack is red, yellow. Then blue is placed on top, making it red, yellow, blue. \n\nFinally, the green box is added on the top. So the final order should be red (bottom), yellow, blue, green. Let me double-check each step to make sure I didn't mix up any steps. Starting with red, then blue, then yellow. Remove blue from the middle, so yellow is now on top of red. Then place blue on top of that, so red, yellow, blue. Then green on top. Yes, that seems right. The key step is removing the blue box from the middle, which leaves yellow on red, then blue goes back on top, followed by green. So the final order from bottom to top is red, yellow, blue, green.\n\n**Final Answer**\nThe final order from bottom to top is \\boxed{red}, \\boxed{yellow}, \\boxed{blue}, \\boxed{green}.\n</think>\n\n1. Start with the red box at the bottom.\n2. Place the blue box on top of the red box. Order: red (bottom), blue.\n3. Place the yellow box on top of the blue box. Order: red, blue, yellow.\n4. Remove the blue box (from the middle) and place it on top. Order: red, yellow, blue.\n5. Place the green box on top. Final order: red, yellow, blue, green.\n\n\\boxed{red}, \\boxed{yellow}, \\boxed{blue}, \\boxed{green}",
    "logprobs": null,
    "finish_reason": "stop",
    "stop_reason": null,
    "prompt_logprobs": null
  }
],
"usage": {
  "prompt_tokens": 76,
  "total_tokens": 544,
  "completion_tokens": 468,
  "prompt_tokens_details": null
}
}

curl http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",
    "prompt": "San Francisco is a",
    "max_tokens": 7,
    "temperature": 0
}'

{"id":"cmpl-0a2310f30ac3454aa7f2c5bb6a292e6c",
"object":"text_completion","created":1723238375,"model":"meta-llama/Llama-3.1-405B-Instruct","choices":[{"index":0,"text":" top destination for foodies, with","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":5,"total_tokens":12,"completion_tokens":7}}

Configurar o escalonador automático personalizado

Nesta seção, você configura o escalonamento automático horizontal de pods para usar métricas personalizadas do Prometheus. Você usa as métricas do Google Cloud Managed Service para Prometheus no servidor vLLM.

Para saber mais, consulte Google Cloud Managed Service para Prometheus. Essa opção precisa estar ativada por padrão no cluster do GKE.

1. Configure o adaptador de métricas personalizadas do Stackdriver no cluster:

   kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
   

2. Adicione a função "Visualizador de monitoramento" à conta de serviço usada pelo adaptador de métricas personalizadas do Stackdriver:

   gcloud projects add-iam-policy-binding projects/PROJECT_ID \
       --role roles/monitoring.viewer \
       --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/custom-metrics/sa/custom-metrics-stackdriver-adapter
   

3. Salve o seguinte manifesto como vllm_pod_monitor.yaml:

   
   apiVersion: monitoring.googleapis.com/v1
   kind: PodMonitoring
   metadata:
    name: vllm-pod-monitoring
   spec:
    selector:
      matchLabels:
       leaderworkerset.sigs.k8s.io/name: vllm
       role: leader
    endpoints:
    - path: /metrics
      port: 8080
      interval: 15s
   

4. Aplique o manifesto ao cluster:

   kubectl apply -f vllm_pod_monitor.yaml
   

Criar carga no endpoint vLLM

Crie carga no servidor do vLLM para testar como o GKE é escalonado automaticamente com uma métrica personalizada do vLLM.

1. Configure o encaminhamento de portas para o modelo:

   kubectl port-forward svc/vllm-leader 8080:8080
   

2. Execute um script bash (load.sh) para enviar N solicitações paralelas ao endpoint do vLLM:

   #!/bin/bash
   N=PARALLEL_PROCESSES
   export vllm_service=$(kubectl get service vllm-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   for i in $(seq 1 $N); do
     while true; do
       curl http://$vllm_service:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "meta-llama/Llama-3.1-70B", "prompt": "Write a story about san francisco", "max_tokens": 100, "temperature": 0}'
     done &  # Run in the background
   done
   wait
   

   Substitua PARALLEL_PROCESSES pelo número de processos em paralelo que você quer executar.

3. Execute o script bash:

   nohup ./load.sh &
   

Verificar se o Google Cloud Managed Service para Prometheus processa as métricas

Depois que o Google Cloud Managed Service para Prometheus extrair as métricas e você adicionar carga ao endpoint do vLLM, será possível conferir as métricas no Cloud Monitoring.

1. No console Google Cloud , acesse a página Metrics Explorer.

   Acessar o Metrics Explorer

2. Clique em < > PromQL.

3. Insira a seguinte consulta para observar as métricas de tráfego:

   vllm:gpu_cache_usage_perc{cluster='CLUSTER_NAME'}
   

A imagem a seguir é um exemplo de gráfico após a execução do script de carga. Este gráfico mostra que o Google Cloud Managed Service para Prometheus está ingerindo as métricas de tráfego em resposta à carga adicionada ao endpoint do vLLM:

Implantar a configuração do escalonador automático horizontal de pods

Ao decidir qual métrica usar para o escalonamento automático, recomendamos as seguintes métricas para o vLLM:

• num_requests_waiting: essa métrica se relaciona ao número de solicitações aguardando na fila do servidor do modelo. Esse número começa a crescer significativamente quando o cache kv está cheio.

• gpu_cache_usage_perc: essa métrica se relaciona à utilização do cache KV, que está diretamente relacionada ao número de solicitações processadas para um determinado ciclo de inferência no servidor do modelo.

Recomendamos o uso de num_requests_waiting ao otimizar para capacidade de processamento e custo e quando as metas de latência forem alcançáveis com a capacidade de processamento máxima do servidor de modelo.

Recomendamos o uso de gpu_cache_usage_perc quando você tiver cargas de trabalho sensíveis à latência em que o escalonamento baseado em fila não é rápido o suficiente para atender aos requisitos.

Para mais explicações, consulte Práticas recomendadas para o escalonamento automático de cargas de trabalho de inferência de modelos de linguagem grandes (LLM) com GPUs.

Ao selecionar uma meta de averageValue para a configuração do HPA, você precisa determinar experimentalmente qual métrica será escalonada automaticamente. Para mais ideias sobre como otimizar seus experimentos, consulte a postagem do blog Economizar em GPUs: escalonamento automático mais inteligente para suas cargas de trabalho de inferência do GKE. O profile-generator usado nesta postagem do blog também funciona para vLLM.

Para implantar a configuração do escalonador automático horizontal de pods usando num_requests_waiting, siga estas etapas:

1. Salve o seguinte manifesto como vllm-hpa.yaml:

   
   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: lws-hpa
   spec:
     minReplicas: 1
     maxReplicas: 2
     metrics:
     - type: Pods
       pods:
         metric:
           name: prometheus.googleapis.com|vllm:num_requests_waiting|gauge
         target:
           type: AverageValue
           averageValue: 5
     scaleTargetRef:
       apiVersion: leaderworkerset.x-k8s.io/v1
       kind: LeaderWorkerSet
       name: vllm
   

   As métricas do vLLM no Google Cloud Managed Service para Prometheus seguem o formato vllm:metric_name.

   Prática recomendada:

   Use num_requests_waiting para dimensionar a taxa de transferência. Use gpu_cache_usage_perc para casos de uso de GPU sensíveis à latência.

2. Implante a configuração do escalonador automático horizontal de pods:

   kubectl apply -f vllm-hpa.yaml
   

   O GKE programa outro pod para implantação, o que aciona o escalonador automático do pool de nós para adicionar um segundo nó antes de implantar a segunda réplica do vLLM.

3. Acompanhe o progresso do escalonamento automático de pods:

   kubectl get hpa --watch
   

   O resultado será assim:

   NAME      REFERENCE              TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
   lws-hpa   LeaderWorkerSet/vllm   0/1       1         2         1          6d1h
   lws-hpa   LeaderWorkerSet/vllm   1/1       1         2         1          6d1h
   lws-hpa   LeaderWorkerSet/vllm   0/1       1         2         1          6d1h
   lws-hpa   LeaderWorkerSet/vllm   4/1       1         2         1          6d1h
   lws-hpa   LeaderWorkerSet/vllm   0/1       1         2         2          6d1h
   

Acelerar os tempos de carregamento do modelo com o Hyperdisk ML

Com esses tipos de LLMs, o vLLM pode levar muito tempo para fazer o download, carregar e ativar cada nova réplica. Por exemplo, esse processo pode levar cerca de 90 minutos com o Llama 3.1 405B. É possível reduzir esse tempo (para 20 minutos com o Llama 3.1 405B) fazendo o download do modelo diretamente em um volume do Hyperdisk ML e montando esse volume em cada pod. Para concluir essa operação, este tutorial usa um volume do Hyperdisk ML e um job do Kubernetes. Um controlador de job no Kubernetes cria um ou mais pods e garante que eles executem uma tarefa específica.

Para acelerar o carregamento do modelo, siga estas etapas:

1. Salve o seguinte manifesto de exemplo como producer-pvc.yaml:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: producer-pvc
   spec:
     storageClassName: hyperdisk-ml
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 800Gi
   

2. Salve o seguinte manifesto de exemplo como producer-job.yaml:

   DeepSeek-R1

   
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: producer-job
   spec:
     template:  # Template for the Pods the Job will create
       spec:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
               - matchExpressions:
                 - key: cloud.google.com/machine-family
                   operator: In
                   values:
                   - "c3"
               - matchExpressions:
                 - key: topology.kubernetes.io/zone
                   operator: In
                   values:
                   - "ZONE"
         containers:
         - name: copy
           resources:
             requests:
               cpu: "32"
             limits:
               cpu: "32"
           image: python:3.11-alpine
           command:
           - sh
           - -c
           - "pip install 'huggingface_hub==0.24.6' && \
             huggingface-cli download deepseek-ai/DeepSeek-R1 --local-dir-use-symlinks=False --local-dir=/data/DeepSeek-R1 --include *.safetensors *.json *.py"
           env:
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
             - mountPath: "/data"
               name: volume
         restartPolicy: Never
         volumes:
           - name: volume
             persistentVolumeClaim:
               claimName: producer-pvc
     parallelism: 1         # Run 1 Pods concurrently
     completions: 1         # Once 1 Pods complete successfully, the Job is done
     backoffLimit: 4        # Max retries on failure
   
   

   Llama 3.1 405B

   
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: producer-job
   spec:
     template:  # Template for the Pods the Job will create
       spec:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
               - matchExpressions:
                 - key: cloud.google.com/machine-family
                   operator: In
                   values:
                   - "c3"
               - matchExpressions:
                 - key: topology.kubernetes.io/zone
                   operator: In
                   values:
                   - "ZONE"
         containers:
         - name: copy
           resources:
             requests:
               cpu: "32"
             limits:
               cpu: "32"
           image: python:3.11-alpine
           command:
           - sh
           - -c
           - "pip install 'huggingface_hub==0.24.6' && \
             huggingface-cli download meta-llama/Meta-Llama-3.1-405B-Instruct --local-dir-use-symlinks=False --local-dir=/data/Meta-Llama-3.1-405B-Instruct --include *.safetensors *.json"
           env:
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           volumeMounts:
             - mountPath: "/data"
               name: volume
         restartPolicy: Never
         volumes:
           - name: volume
             persistentVolumeClaim:
               claimName: producer-pvc
     parallelism: 1         # Run 1 Pods concurrently
     completions: 1         # Once 1 Pods complete successfully, the Job is done
     backoffLimit: 4        # Max retries on failure
   
   

3. Siga as instruções em Acelerar o carregamento de dados de IA/ML com o Hyperdisk ML, usando os dois arquivos que você criou nas etapas anteriores.

   Depois desta etapa, você criou e preencheu o volume de ML do Hyperdisk com os dados do modelo.

4. Implante a implantação do servidor de GPU de vários nós do vLLM, que vai usar o volume de ML do Hyperdisk recém-criado para dados de modelos.

   DeepSeek-R1

   
   
   apiVersion: leaderworkerset.x-k8s.io/v1
   kind: LeaderWorkerSet
   metadata:
     name: vllm
   spec:
     replicas: 1
     leaderWorkerTemplate:
       size: 2
       restartPolicy: RecreateGroupOnPodRestart
       leaderTemplate:
         metadata:
           labels:
             role: leader
         spec:
           containers:
             - name: vllm-leader
               image: docker.io/vllm/vllm-openai:latest
               env:
                 - name: HUGGING_FACE_HUB_TOKEN
                   valueFrom:
                     secretKeyRef:
                       name: hf-secret
                       key: hf_api_token
               command:
                 - sh
                 - -c
                 - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
                   python3 -m vllm.entrypoints.openai.api_server --port 8080 --model /models/DeepSeek-R1 --tensor-parallel-size 8 --pipeline-parallel-size 2 --trust-remote-code --max-model-len 4096"
               resources:
                 limits:
                   nvidia.com/gpu: "8"
               ports:
                 - containerPort: 8080
               readinessProbe:
                 tcpSocket:
                   port: 8080
                 initialDelaySeconds: 15
                 periodSeconds: 10
               volumeMounts:
                 - mountPath: /dev/shm
                   name: dshm
                 - mountPath: /models
                   name: deepseek-r1
           volumes:
           - name: dshm
             emptyDir:
               medium: Memory
           - name: deepseek-r1
             persistentVolumeClaim:
               claimName: hdml-static-pvc
       workerTemplate:
         spec:
           containers:
             - name: vllm-worker
               image: docker.io/vllm/vllm-openai:latest
               command:
                 - sh
                 - -c
                 - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
               resources:
                 limits:
                   nvidia.com/gpu: "8"
               env:
                 - name: HUGGING_FACE_HUB_TOKEN
                   valueFrom:
                     secretKeyRef:
                       name: hf-secret
                       key: hf_api_token
               volumeMounts:
                 - mountPath: /dev/shm
                   name: dshm
                 - mountPath: /models
                   name: deepseek-r1
           volumes:
           - name: dshm
             emptyDir:
               medium: Memory
           - name: deepseek-r1
             persistentVolumeClaim:
               claimName: hdml-static-pvc
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: vllm-leader
   spec:
     ports:
       - name: http
         port: 8080
         protocol: TCP
         targetPort: 8080
     selector:
       leaderworkerset.sigs.k8s.io/name: vllm
       role: leader
     type: ClusterIP
   

   Llama 3.1 405B

   
   
   apiVersion: leaderworkerset.x-k8s.io/v1
   kind: LeaderWorkerSet
   metadata:
     name: vllm
   spec:
     replicas: 1
     leaderWorkerTemplate:
       size: 2
       restartPolicy: RecreateGroupOnPodRestart
       leaderTemplate:
         metadata:
           labels:
             role: leader
         spec:
           containers:
             - name: vllm-leader
               image: docker.io/vllm/vllm-openai:latest
               env:
                 - name: HUGGING_FACE_HUB_TOKEN
                   valueFrom:
                     secretKeyRef:
                       name: hf-secret
                       key: hf_api_token
               command:
                 - sh
                 - -c
                 - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
                   python3 -m vllm.entrypoints.openai.api_server --port 8080 --model /models/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2"
               resources:
                 limits:
                   nvidia.com/gpu: "8"
               ports:
                 - containerPort: 8080
               readinessProbe:
                 tcpSocket:
                   port: 8080
                 initialDelaySeconds: 15
                 periodSeconds: 10
               volumeMounts:
                 - mountPath: /dev/shm
                   name: dshm
                 - mountPath: /models
                   name: llama3-405b
           volumes:
           - name: dshm
             emptyDir:
               medium: Memory
           - name: llama3-405b
             persistentVolumeClaim:
               claimName: hdml-static-pvc
       workerTemplate:
         spec:
           containers:
             - name: vllm-worker
               image: docker.io/vllm/vllm-openai:latest
               command:
                 - sh
                 - -c
                 - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
               resources:
                 limits:
                   nvidia.com/gpu: "8"
               env:
                 - name: HUGGING_FACE_HUB_TOKEN
                   valueFrom:
                     secretKeyRef:
                       name: hf-secret
                       key: hf_api_token
               volumeMounts:
                 - mountPath: /dev/shm
                   name: dshm
                 - mountPath: /models
                   name: llama3-405b
           volumes:
           - name: dshm
             emptyDir:
               medium: Memory
           - name: llama3-405b
             persistentVolumeClaim:
               claimName: hdml-static-pvc
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: vllm-leader
   spec:
     ports:
       - name: http
         port: 8080
         protocol: TCP
         targetPort: 8080
     selector:
       leaderworkerset.sigs.k8s.io/name: vllm
       role: leader
     type: ClusterIP