本页介绍了如何自定义 GKE 推理网关部署。
本页面适用于负责管理 GKE 基础架构的网络专家，以及负责管理 AI 工作负载的平台管理员。
如需管理和优化推理工作负载，您可以配置 GKE 推理网关的高级功能。
了解并配置以下高级功能：
- 如需使用 Model Armor 集成，请配置 AI 安全和安全性检查。
- 如需查看 GKE Inference Gateway 和模型服务器指标和信息中心，以及启用 HTTP 访问日志记录以获取详细的请求和响应信息，请配置可观测性
- 如需自动扩缩 GKE 推理网关部署，请配置自动扩缩。
配置 AI 安全和安全性检查
GKE 推理网关与 Model Armor 集成，可针对使用大语言模型 (LLM) 的应用的提示和回答执行安全检查。此集成在基础架构级别提供了额外的安全强制执行层，可与应用级安全措施相辅相成。这样，您就可以集中对所有 LLM 流量应用政策。
下图展示了在 GKE 集群上将 Model Armor 与 GKE 推理网关集成：
如需配置 AI 安全检查，请执行以下步骤：
确保满足以下前提条件：
- 在 Google Cloud 项目中启用 Model Armor 服务。
- 使用 Model Armor 控制台、Google Cloud CLI 或 API 创建 Model Armor 模板。
确保您已创建名为
my-model-armor-template-name-id的 Model Armor 模板。如需配置
GCPTrafficExtension，请执行以下步骤：将以下示例清单保存为
gcp-traffic-extension.yaml：kind: GCPTrafficExtension apiVersion: networking.gke.io/v1 metadata: name: my-model-armor-extension spec: targetRefs: - group: "gateway.networking.k8s.io" kind: Gateway name: GATEWAY_NAME extensionChains: - name: my-model-armor-chain1 matchCondition: celExpressions: - celMatcher: request.path.startsWith("/") extensions: - name: my-model-armor-service supportedEvents: - RequestHeaders timeout: 1s googleAPIServiceName: "modelarmor.us-central1.rep.googleapis.com" metadata: 'extensionPolicy': MODEL_ARMOR_TEMPLATE_NAME 'sanitizeUserPrompt': 'true' 'sanitizeUserResponse': 'true'替换以下内容：
GATEWAY_NAME：网关的名称。MODEL_ARMOR_TEMPLATE_NAME：模型装甲模板的名称。
gcp-traffic-extension.yaml文件包含以下设置：targetRefs：指定此扩展适用的网关。extensionChains：定义要应用于流量的扩展程序链。matchCondition：定义应用扩展的条件。extensions：定义要应用的扩展。supportedEvents：指定调用扩展程序的事件。timeout：指定扩展程序的超时时间。googleAPIServiceName：指定扩展程序的服务名称。metadata：指定扩展程序的元数据，包括extensionPolicy以及提示或回答的净化设置。
将示例清单应用到您的集群：
kubectl apply -f `gcp-traffic-extension.yaml`
配置 AI 安全检查并将其与网关集成后，Model Armor 会根据定义的规则自动过滤提示和回答。
配置可观测性
GKE 推理网关可深入了解推理工作负载的运行状况、性能和行为。这有助于您发现和解决问题、优化资源利用率，并确保应用的可靠性。
Google Cloud 提供了以下 Cloud Monitoring 信息中心，可为 GKE 推理网关提供推理可观测性：
- GKE 推理网关信息中心：提供 LLM 服务的黄金指标，例如
InferencePool的请求和令牌吞吐量、延迟时间、错误和缓存利用率。如需查看可用 GKE 推理网关指标的完整列表，请参阅公开的指标。 - 模型服务器信息中心：提供模型服务器黄金信号的信息中心。这样，您就可以监控模型服务器（例如
KVCache Utilization和Queue length）的负载和性能。这样，您就可以监控模型服务器的负载和性能。 - 负载均衡器信息中心：报告负载均衡器的指标，例如每秒请求数、端到端请求处理延迟时间和请求-响应状态代码。这些指标可帮助您了解端到端请求服务的性能并发现错误。
- 数据中心 GPU 管理器 (DCGM) 指标：提供来自 NVIDIA GPU 的指标，例如 NVIDIA GPU 的性能和利用率。您可以在 Cloud Monitoring 中配置 NVIDIA 数据中心 GPU 管理器 (DCGM) 指标。如需了解详情，请参阅收集和查看 DCGM 指标。
查看 GKE 推理网关信息中心
如需查看 GKE 推理网关信息中心，请执行以下操作：
在 Google Cloud 控制台中，前往 Monitoring 页面。
在导航窗格中，选择信息中心。
在集成部分中，选择 GMP。
在 Cloud Monitoring 信息中心模板页面中，搜索“网关”。
查看 GKE 推理网关信息中心。
或者，您也可以按照监控信息中心中的说明操作。
配置模型服务器可观测性信息中心
如需从每个模型服务器收集黄金信号，并了解有哪些因素会影响 GKE 推理网关的性能，您可以为模型服务器配置自动监控。这包括以下模型服务器：
如需查看集成信息中心，请执行以下步骤：
- 从模型服务器收集指标。
在 Google Cloud 控制台中，前往 Monitoring 页面。
在导航窗格中，选择信息中心。
在集成下，选择 GMP。 系统会显示相应的集成信息中心。
图： 集成信息中心
如需了解详情，请参阅自定义应用监控。
配置负载均衡器可观测性信息中心
如需将应用负载平衡器与 GKE 推理网关搭配使用，请按以下步骤导入信息中心：
如需创建负载均衡器信息中心，请创建以下文件并将其另存为
dashboard.json：{ "displayName": "GKE Inference Gateway (Load Balancer) Prometheus Overview", "dashboardFilters": [ { "filterType": "RESOURCE_LABEL", "labelKey": "cluster", "templateVariable": "", "valueType": "STRING" }, { "filterType": "RESOURCE_LABEL", "labelKey": "location", "templateVariable": "", "valueType": "STRING" }, { "filterType": "RESOURCE_LABEL", "labelKey": "namespace", "templateVariable": "", "valueType": "STRING" }, { "filterType": "RESOURCE_LABEL", "labelKey": "forwarding_rule_name", "templateVariable": "", "valueType": "STRING" } ], "labels": {}, "mosaicLayout": { "columns": 48, "tiles": [ { "height": 8, "width": 48, "widget": { "title": "", "id": "", "text": { "content": "### Inferece Gateway Metrics\n\nPlease refer to the [official documentation](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/site-src/guides/metrics.md) for more details of underlying metrics used in the dashboard.\n\n\n### External Application Load Balancer Metrics\n\nPlease refer to the [pubic page](/load-balancing/docs/metrics) for complete list of External Application Load Balancer metrics.\n\n### Model Server Metrics\n\nYou can redirect to the detail dashboard for model servers under the integration tab", "format": "MARKDOWN", "style": { "backgroundColor": "#FFFFFF", "fontSize": "FS_EXTRA_LARGE", "horizontalAlignment": "H_LEFT", "padding": "P_EXTRA_SMALL", "pointerLocation": "POINTER_LOCATION_UNSPECIFIED", "textColor": "#212121", "verticalAlignment": "V_TOP" } } } }, { "yPos": 8, "height": 4, "width": 48, "widget": { "title": "External Application Load Balancer", "id": "", "sectionHeader": { "dividerBelow": false, "subtitle": "" } } }, { "yPos": 12, "height": 15, "width": 24, "widget": { "title": "E2E Request Latency p99 (by code)", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.99, sum by(le, response_code) (rate(loadbalancing_googleapis_com:https_external_regional_total_latencies_bucket{monitored_resource=\"http_external_regional_lb_rule\",forwarding_rule_name=~\".*inference-gateway.*\"}[1m])))", "unitOverride": "ms" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 12, "height": 43, "width": 48, "widget": { "title": "Regional", "collapsibleGroup": { "collapsed": false }, "id": "" } }, { "yPos": 12, "xPos": 24, "height": 15, "width": 24, "widget": { "title": "E2E Request Latency p95 (by code)", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le, response_code) (rate(loadbalancing_googleapis_com:https_external_regional_total_latencies_bucket{monitored_resource=\"http_external_regional_lb_rule\",forwarding_rule_name=~\".*inference-gateway.*\"}[1m])))", "unitOverride": "ms" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 27, "height": 15, "width": 24, "widget": { "title": "E2E Request Latency p90 (by code)", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.90, sum by(le, response_code) (rate(loadbalancing_googleapis_com:https_external_regional_total_latencies_bucket{monitored_resource=\"http_external_regional_lb_rule\",forwarding_rule_name=~\".*inference-gateway.*\"}[1m])))", "unitOverride": "ms" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 27, "xPos": 24, "height": 15, "width": 24, "widget": { "title": "E2E Request Latency p50 (by code)", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.50, sum by(le, response_code) (rate(loadbalancing_googleapis_com:https_external_regional_total_latencies_bucket{monitored_resource=\"http_external_regional_lb_rule\",forwarding_rule_name=~\".*inference-gateway.*\"}[1m])))", "unitOverride": "ms" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 42, "height": 13, "width": 48, "widget": { "title": "Request /s (by code)", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "sum by (response_code)(rate(loadbalancing_googleapis_com:https_external_regional_request_count{monitored_resource=\"http_external_regional_lb_rule\", forwarding_rule_name=~\".*inference-gateway.*\"}[1m]))", "unitOverride": "" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 55, "height": 4, "width": 48, "widget": { "title": "Inference Optimized Gateway", "id": "", "sectionHeader": { "dividerBelow": false, "subtitle": "" } } }, { "yPos": 59, "height": 17, "width": 48, "widget": { "title": "Request Latency", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(inference_model_request_duration_seconds_bucket{}[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(inference_model_request_duration_seconds_bucket{}[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(inference_model_request_duration_seconds_bucket{}[${__interval}])))", "unitOverride": "s" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 59, "height": 65, "width": 48, "widget": { "title": "Inference Model", "collapsibleGroup": { "collapsed": false }, "id": "" } }, { "yPos": 76, "height": 16, "width": 24, "widget": { "title": "Request / s", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "sum by(model_name, target_model_name) (rate(inference_model_request_total{}[${__interval}]))", "unitOverride": "" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 76, "xPos": 24, "height": 16, "width": 24, "widget": { "title": "Request Error / s", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "sum by (error_code,model_name,target_model_name) (rate(inference_model_request_error_total[${__interval}]))", "unitOverride": "" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 92, "height": 16, "width": 24, "widget": { "title": "Request Size", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(inference_model_request_sizes_bucket{}[${__interval}])))", "unitOverride": "By" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(inference_model_request_sizes_bucket{}[${__interval}])))", "unitOverride": "By" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(inference_model_request_sizes_bucket{}[${__interval}])))", "unitOverride": "By" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 92, "xPos": 24, "height": 16, "width": 24, "widget": { "title": "Response Size", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(inference_model_response_sizes_bucket{}[${__interval}])))", "unitOverride": "By" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(inference_model_response_sizes_bucket{}[${__interval}])))", "unitOverride": "By" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(inference_model_response_sizes_bucket{}[${__interval}])))", "unitOverride": "By" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 108, "height": 16, "width": 24, "widget": { "title": "Input Token Count", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(inference_model_input_tokens_bucket{}[${__interval}])))", "unitOverride": "" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(inference_model_input_tokens_bucket{}[${__interval}])))", "unitOverride": "" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(inference_model_input_tokens_bucket{}[${__interval}])))", "unitOverride": "" } } ], "thresholds": [] } } }, { "yPos": 108, "xPos": 24, "height": 16, "width": 24, "widget": { "title": "Output Token Count", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(inference_model_output_tokens_bucket{}[${__interval}])))", "unitOverride": "" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(inference_model_output_tokens_bucket{}[${__interval}])))", "unitOverride": "" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(inference_model_output_tokens_bucket{}[${__interval}])))", "unitOverride": "" } } ], "thresholds": [] } } }, { "yPos": 124, "height": 16, "width": 24, "widget": { "title": "Average KV Cache Utilization", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "sum by (name)(avg_over_time(inference_pool_average_kv_cache_utilization[${__interval}]))*100", "unitOverride": "%" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 124, "height": 16, "width": 48, "widget": { "title": "Inference Pool", "collapsibleGroup": { "collapsed": false }, "id": "" } }, { "yPos": 124, "xPos": 24, "height": 16, "width": 24, "widget": { "title": "Average Queue Size", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "sum by (name) (avg_over_time(inference_pool_average_queue_size[${__interval}]))", "unitOverride": "" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 140, "height": 4, "width": 48, "widget": { "title": "Model Server", "id": "", "sectionHeader": { "dividerBelow": true, "subtitle": "The following charts will only be populated if model server is exporting metrics." } } }, { "yPos": 144, "height": 32, "width": 48, "widget": { "title": "vLLM", "collapsibleGroup": { "collapsed": false }, "id": "" } }, { "yPos": 144, "xPos": 1, "height": 16, "width": 24, "widget": { "title": "Token Throughput", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "Prompt Tokens/Sec", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "sum by(model_name) (rate(vllm:prompt_tokens_total[${__interval}]))", "unitOverride": "" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "Generation Tokens/Sec", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "sum by(model_name) (rate(vllm:generation_tokens_total[${__interval}]))", "unitOverride": "" } } ], "thresholds": [] } } }, { "yPos": 144, "xPos": 25, "height": 16, "width": 23, "widget": { "title": "Request Latency", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(vllm:e2e_request_latency_seconds_bucket[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(vllm:e2e_request_latency_seconds_bucket[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(vllm:e2e_request_latency_seconds_bucket[${__interval}])))", "unitOverride": "s" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 160, "xPos": 1, "height": 16, "width": 24, "widget": { "title": "Time Per Output Token Latency", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[${__interval}])))", "unitOverride": "s" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } }, { "yPos": 160, "xPos": 25, "height": 16, "width": 23, "widget": { "title": "Time To First Token Latency", "id": "", "xyChart": { "chartOptions": { "displayHorizontal": false, "mode": "COLOR", "showLegend": false }, "dataSets": [ { "breakdowns": [], "dimensions": [], "legendTemplate": "p95", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.95, sum by(le) (rate(vllm:time_to_first_token_seconds_bucket[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p90", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.9, sum by(le) (rate(vllm:time_to_first_token_seconds_bucket[${__interval}])))", "unitOverride": "s" } }, { "breakdowns": [], "dimensions": [], "legendTemplate": "p50", "measures": [], "plotType": "LINE", "targetAxis": "Y1", "timeSeriesQuery": { "outputFullDuration": false, "prometheusQuery": "histogram_quantile(0.5, sum by(le) (rate(vllm:time_to_first_token_seconds_bucket[${__interval}])))", "unitOverride": "s" } } ], "thresholds": [], "yAxis": { "label": "", "scale": "LINEAR" } } } } ] } }如需将信息中心安装到 Google Cloud Armor，请运行以下命令：
gcloud monitoring dashboards create --project $PROJECT_ID --config-from-file=dashboard.json在 Google Cloud 控制台中打开 Monitoring 页面。
在导航菜单中，选择信息中心。
从自定义信息中心列表中选择 Inference Optimized Gateway (with L7LB) Prometheus Overview（启用了 L7LB 的推理优化网关 Prometheus 概览）信息中心。
外部应用负载平衡器部分会显示以下负载均衡指标：
- 端到端请求延迟时间第 99 百分位数（按代码）：显示负载均衡器处理的请求的端到端请求延迟时间第 99 百分位数，按返回的状态代码汇总。
- 请求数（按代码）：显示负载均衡器处理的请求数量，按返回的状态代码汇总。
为 GKE 推理网关配置日志记录
为 GKE 推理网关配置日志记录可提供有关请求和响应的详细信息，这对问题排查、审核和性能分析非常有用。HTTP 访问日志会记录每个请求和响应，包括标头、状态代码和时间戳。此级别的详细信息可帮助您发现问题、查找错误并了解推理工作负载的行为。
如需为 GKE 推理网关配置日志记录，请为每个 InferencePool 对象启用 HTTP 访问日志记录。
将以下示例清单保存为
logging-backend-policy.yaml：apiVersion: networking.gke.io/v1 kind: GCPBackendPolicy metadata: name: logging-backend-policy namespace: NAMESPACE_NAME spec: default: logging: enabled: true sampleRate: 500000 targetRef: group: inference.networking.x-k8s.io kind: InferencePool name: INFERENCE_POOL_NAME替换以下内容：
NAMESPACE_NAME：部署InferencePool的命名空间的名称。INFERENCE_POOL_NAME：InferencePool的名称。
将示例清单应用到您的集群：
kubectl apply -f logging-backend-policy.yaml
应用此清单后，GKE 推理网关会为指定的 InferencePool 启用 HTTP 访问日志。您可以在 Cloud Logging 中查看这些日志。日志包含有关每个请求和响应的详细信息，例如请求网址、标头、响应状态代码和延迟时间。
配置自动扩缩
自动扩缩功能会根据负载变化调整资源分配，通过根据需求动态添加或移除 Pod 来维持性能和资源效率。对于 GKE 推理网关，这涉及对每个 InferencePool 中的 Pod 进行横向自动扩缩。GKE Pod 横向自动扩缩器 (HPA) 根据 KVCache Utilization 等模型服务器指标自动扩缩 Pod。这可确保推理服务在高效管理资源使用情况的同时，处理不同的工作负载和查询量。
如需配置 InferencePool 实例，使其根据 GKE Inference Gateway 生成的指标自动扩缩，请按以下步骤操作：
在集群中部署
PodMonitoring对象，以收集 GKE 推理网关生成的指标。如需了解详情，请参阅配置可观测性。部署自定义指标 Stackdriver 适配器，以向 HPA 授予对指标的访问权限：
将以下示例清单保存为
adapter_new_resource_model.yaml：apiVersion: v1 kind: Namespace metadata: name: custom-metrics --- apiVersion: v1 kind: ServiceAccount metadata: name: custom-metrics-stackdriver-adapter namespace: custom-metrics --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: custom-metrics:system:auth-delegator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: custom-metrics-stackdriver-adapter namespace: custom-metrics --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: custom-metrics-auth-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects: - kind: ServiceAccount name: custom-metrics-stackdriver-adapter namespace: custom-metrics --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: custom-metrics-resource-reader namespace: custom-metrics rules: - apiGroups: - "" resources: - pods - nodes - nodes/stats verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: custom-metrics-resource-reader roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: custom-metrics-resource-reader subjects: - kind: ServiceAccount name: custom-metrics-stackdriver-adapter namespace: custom-metrics --- apiVersion: apps/v1 kind: Deployment metadata: run: custom-metrics-stackdriver-adapter k8s-app: custom-metrics-stackdriver-adapter spec: replicas: 1 selector: matchLabels: run: custom-metrics-stackdriver-adapter k8s-app: custom-metrics-stackdriver-adapter template: metadata: labels: run: custom-metrics-stackdriver-adapter k8s-app: custom-metrics-stackdriver-adapter kubernetes.io/cluster-service: "true" spec: serviceAccountName: custom-metrics-stackdriver-adapter containers: - image: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.15.2-gke.1 imagePullPolicy: Always name: pod-custom-metrics-stackdriver-adapter command: - /adapter - --use-new-resource-model=true - --fallback-for-container-metrics=true resources: limits: cpu: 250m memory: 200Mi requests: cpu: 250m memory: 200Mi --- apiVersion: v1 kind: Service metadata: labels: run: custom-metrics-stackdriver-adapter k8s-app: custom-metrics-stackdriver-adapter kubernetes.io/cluster-service: 'true' kubernetes.io/name: Adapter name: custom-metrics-stackdriver-adapter namespace: custom-metrics spec: ports: - port: 443 protocol: TCP targetPort: 443 selector: run: custom-metrics-stackdriver-adapter k8s-app: custom-metrics-stackdriver-adapter type: ClusterIP --- apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta1.custom.metrics.k8s.io spec: insecureSkipTLSVerify: true group: custom.metrics.k8s.io groupPriorityMinimum: 100 versionPriority: 100 service: name: custom-metrics-stackdriver-adapter namespace: custom-metrics version: v1beta1 --- apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta2.custom.metrics.k8s.io spec: insecureSkipTLSVerify: true group: custom.metrics.k8s.io groupPriorityMinimum: 100 versionPriority: 200 service: name: custom-metrics-stackdriver-adapter namespace: custom-metrics version: v1beta2 --- apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta1.external.metrics.k8s.io spec: insecureSkipTLSVerify: true group: external.metrics.k8s.io groupPriorityMinimum: 100 versionPriority: 100 service: name: custom-metrics-stackdriver-adapter namespace: custom-metrics version: v1beta1 --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: external-metrics-reader rules: - apiGroups: - "external.metrics.k8s.io" resources: - "*" verbs: - list - get - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: external-metrics-reader roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: external-metrics-reader subjects: - kind: ServiceAccount name: horizontal-pod-autoscaler namespace: kube-system将示例清单应用到您的集群：
kubectl apply -f adapter_new_resource_model.yaml
如需向适配器授予读取项目指标的权限，请运行以下命令：
$ PROJECT_ID=PROJECT_ID $ PROJECT_NUMBER=$(gcloud projects describe PROJECT_ID --format="value(projectNumber)") $ gcloud projects add-iam-policy-binding projects/PROJECT_ID \ --role roles/monitoring.viewer \ --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/custom-metrics/sa/custom-metrics-stackdriver-adapter请将
PROJECT_ID替换为您的 Google Cloud 项目 ID。对于每个
InferencePool，部署一个类似于以下内容的 HPA：apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: INFERENCE_POOL_NAME namespace: INFERENCE_POOL_NAMESPACE spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: INFERENCE_POOL_NAME minReplicas: MIN_REPLICAS maxReplicas: MAX_REPLICAS metrics: - type: External external: metric: name: prometheus.googleapis.com|inference_pool_average_kv_cache_utilization|gauge selector: matchLabels: metric.labels.name: INFERENCE_POOL_NAME resource.labels.cluster: CLUSTER_NAME resource.labels.namespace: INFERENCE_POOL_NAMESPACE target: type: AverageValue averageValue: TARGET_VALUE替换以下内容：
INFERENCE_POOL_NAME：InferencePool的名称。INFERENCE_POOL_NAMESPACE：InferencePool的命名空间。CLUSTER_NAME：集群的名称。MIN_REPLICAS：InferencePool的最小可用性（基准容量）。当使用率低于 HPA 目标阈值时，HPA 会保持此副本数。高可用性工作负载必须将此值设置为高于1的值，以确保在 Pod 中断期间持续可用。MAX_REPLICAS：用于约束必须分配给InferencePool中托管的工作负载的加速器数量的值。HPA 不会将副本数量增加到超过此值。在高峰流量期间，监控副本数量，确保MAX_REPLICAS字段的值提供足够的余量，以便工作负载可以扩容以保持所选工作负载的性能特性。TARGET_VALUE：表示每个模型服务器所选目标KV-Cache Utilization的值。此值介于 0 到 100 之间，在很大程度上取决于模型服务器、模型、加速器和传入流量特征。您可以通过负载测试并绘制吞吐量与延迟时间图表，以实验方式确定此目标值。从图表中选择所选的吞吐量和延迟时间组合，并使用相应的KV-Cache Utilization值作为 HPA 目标。您必须调整并密切监控此值，才能实现所选的价格/性能结果。您可以使用 GKE 推理建议自动确定此值。
后续步骤
- 了解 GKE 推理网关。
- 了解如何部署 GKE 推理网关。
- 了解 GKE 推理网关发布操作。
- 了解如何使用 GKE 推理网关进行服务。