Multi-agent AI system in Google Cloud

• Clearly define the business goal of the agentic AI system and the task that each agent performs.
• Use an agent pattern that best meets your requirements.
• Use the ADK to efficiently create, deploy, and manage your agentic architecture.

• Design the human-facing agents in the architecture to support natural language interactions.
• Ensure that each agent clearly communicates its actions and status to its dependent clients.
• Design the agents to detect and handle ambiguous queries and nuanced interactions.

• Ensure that the agents have sufficient context to track multi-turn interactions and session parameters.
• Clearly describe the purpose, arguments, and usage of the tools that the agents can use.
• Ensure that the agents' responses are grounded in reliable data sources to reduce hallucinations.
• Implement logic to handle no-match situations, such as when a prompt is off-topic.

AI agents introduce certain unique and critical security risks that conventional, deterministic security practices might not be able to mitigate adequately. Google recommends an approach that combines the strengths of deterministic security controls with dynamic, reasoning-based defenses. This approach is grounded in three core principles: human oversight, carefully defined agent autonomy, and observability. The following are specific recommendations that are aligned with these core principles.

Human oversight: An agentic AI system might sometimes fail or not perform as expected. For example, the model might generate inaccurate content or an agent might select inappropriate tools. In business-critical agentic AI systems, incorporate a human-in-the-loop flow to let human supervisors monitor, override, and pause agents in real time. For example, human users can review the output of agents, approve or reject the outputs, and provide further guidance to correct errors or to make strategic decisions. This approach combines the efficiency of agentic AI systems with the critical thinking and domain expertise of human users.

Access control for agents: Configure agent permissions by using Identity and Access Management (IAM) controls. Grant each agent only the permissions that it needs to perform its tasks and to communicate with tools and with other agents. This approach helps to minimize the potential impact of a security breach, because a compromised agent would have limited access to other parts of the system. For more information, see Set up the identity and permissions for your agent and Managing access for deployed agents.

Monitoring: Monitor agent behavior by using comprehensive tracing capabilities that give you visibility into every action that an agent takes, including its reasoning process, tool selection, and execution paths. For more information, see Logging an agent in Vertex AI Agent Engine and Logging in the ADK.

For more information about securing AI agents, see Safety and Security for AI Agents.

Shared responsibility: Security is a shared responsibility. Vertex AI secures the underlying infrastructure and provides tools and security controls to help you protect your data, code, and models. You are responsible for properly configuring your services, managing access controls, and securing your applications. For more information, see Vertex AI shared responsibility.

Security controls: Vertex AI supports Google Cloud security controls that you can use to meet your requirements for data residency, customer-managed encryption keys (CMEK), network security using VPC Service Controls, and Access Transparency. For more information, see the following documentation:

• Security controls for Vertex AI
• Security controls for Generative AI
• Generative AI and zero data retention

Safety: AI models might produce harmful responses, sometimes in response to malicious prompts.

• To enhance safety and mitigate potential misuse of the agentic AI system, you can configure content filters to act as barriers to harmful inputs and responses. For more information, see Safety and content filters.
• To inspect and sanitize inference requests and responses for threats like prompt injection and harmful content, you can use Model Armor. Model Armor helps you prevent malicious input, verify content safety, protect sensitive data, maintain compliance, and enforce safety and security policies consistently.

Model access: You can set up organization policies to limit the type and versions of AI models that can be used in a Google Cloud project. For more information, see Control access to Model Garden models.

Data protection: To discover and de-identify sensitive data in the prompts and responses and in log data, use the Cloud Data Loss Prevention API. For more information, see this video: Protecting sensitive data in AI apps.

Transport security: The A2A protocol mandates HTTPS for all A2A communication in production environments and it recommends Transport Layer Security (TLS) versions 1.2 or higher.

Authentication: The A2A protocol delegates authentication to standard web mechanisms like HTTP headers and to standards like OAuth2 and OpenID Connect. Each agent advertises the authentication requirements in its Agent Card. For more information, see A2A authentication.

Ingress security (for the frontend service): To control access to the application, disable the default run.app URL of the frontend Cloud Run service and set up a regional external Application Load Balancer. In addition to load-balancing incoming traffic to the application, the load balancer handles SSL certificate management. For added protection, you can use Google Cloud Armor security policies to provide request filtering, DDoS protection, and rate limiting for the service.

User authentication: To authenticate user access to the frontend Cloud Run service, use Identity-Aware Proxy (IAP). When a user tries to access an IAP-secured resource, IAP performs authentication and authorization checks. For more information, see Enabling IAP for Cloud Run.

Container image security: To ensure that only authorized container images are deployed to Cloud Run, you can use Binary Authorization. To identify and mitigate security risks in the container images, use Artifact Analysis to automatically run vulnerability scans. For more information, see Container scanning overview.

Data residency: Cloud Run helps you meet data residency requirements. Your Cloud Run functions run within the selected region.

For more guidance about container security, see General Cloud Run development tips.

Data encryption: By default, Google Cloud encrypts data at rest by using Google-owned and Google-managed encryption keys. To protect your agents' data by using encryption keys that you control, you can use CMEKs that you create and manage in Cloud KMS. For information about Google Cloud services that are compatible with Cloud KMS, see Compatible services.

Mitigate data exfiltration risk: To reduce the risk of data exfiltration, create a VPC Service Controls perimeter around the infrastructure. VPC Service Controls supports all of the Google Cloud services that this reference architecture uses.

Access control: When you configure permissions for the resources in your topology, follow the principle of least privilege.

Post-deployment optimization: After you deploy your application in Google Cloud, get recommendations to further optimize security by using Active Assist Recommendation Hub. Review the recommendations and apply them as appropriate for your environment. For more information, see Find recommendations in Recommendation Hub.

Cloud environment security: Use the tools in Security Command Center to detect vulnerabilities, identify and mitigate threats, define and deploy a security posture, and export data for further analysis.

Fault tolerance: Design the agentic system to tolerate or handle agent-level failures. Where feasible, use a decentralized approach where agents can operate independently.

Simulate failures: Before deploying the agentic AI system to production, validate it by simulating a production environment. Identify and fix inter-agent coordination issues and unexpected behaviors.

Error handling: To enable diagnosis and troubleshooting of errors, implement logging, exception handling, and retry mechanisms.

Quota management: Vertex AI supports dynamic shared quota (DSQ) for Gemini models. DSQ helps to flexibly manage pay-as-you-go requests, and it eliminates the need to manage quota manually or to request quota increases. DSQ dynamically allocates the available resources for a given model and region across active customers. With DSQ, there are no predefined quota limits on individual customers.

Capacity planning: If the number of requests to the model exceeds the allocated capacity, then error code 429 is returned. For workloads that are business critical and that require consistently high throughput, you can reserve throughput by using Provisioned Throughput.

Model endpoint availability: If data can be shared across multiple regions or countries, you can use a global endpoint for the model.

Monitoring using logs: By default, agent logs that are written to the stdout and stderr streams are routed to Cloud Logging. For advanced logging, you can integrate the Python logger with Cloud Logging. If you need full control over logging and structured logs, use the Cloud Logging client. For more information, see Logging an agent and Logging in the ADK.

Continuous evaluation: Regularly perform a qualitative evaluation of the output of the agents and the trajectory or steps taken by the agents to produce the output. To implement agent evaluation, you can use the Gen AI evaluation service or the evaluation methods that the ADK supports.

Database tools: To efficiently manage database tools for your AI agents and to ensure that the agents securely handle complexities like connection pooling and authentication, use the MCP Toolbox for Databases. It provides a centralized location to store and update database tools. You can share the tools across agents and update the tools without redeploying agents. The toolbox includes a wide range of tools for Google Cloud databases like AlloyDB for PostgreSQL and for third-party databases like MongoDB.

Generative AI models: To enable AI agents to use Google generative AI models like Imagen and Veo, you can use MCP Servers for Google Cloud generative media APIs.

Google security products and tools: To enable your AI agents to access Google security products and tools like Google Security Operations, Google Threat Intelligence, and Security Command Center, use MCP servers for Google security products.

Cost analysis and management: To analyze and manage Vertex AI costs, we recommend that you create baseline metrics for queries per second (QPS) and tokens per second (TPS). Then, monitor these metrics after deployment. The baseline also helps with capacity planning. For example, the baseline helps you determine when Provisioned Throughput might be necessary.

Model selection: The model that you select for your AI application directly affects both costs and performance. To identify the model that provides an optimal balance between performance and cost for your specific use case, test models iteratively. We recommend that you start with the most cost-efficient model and progress gradually to more powerful options.

Cost-effective prompting: The length of your prompts (input) and the generated responses (output) directly affect performance and cost. Write prompts that are short, direct, and provide sufficient context. Design your prompts to get concise responses from the model. For example, include phrases such as "summarize in 2 sentences" or "list 3 key points". For more information, see the best practices for prompt design.

Context caching: To reduce the cost of requests that contain repeated content with high input token counts, use context caching.

Batch requests: When relevant, consider batch prediction. Batched requests incur a lower cost than standard requests.

Resource allocation: When you create a Cloud Run service, you can specify the amount of memory and CPU to be allocated. Start with the default CPU and memory allocations. Observe the resource usage and cost over time, and adjust the allocation as necessary. For more information, see the following documentation:

• Configure memory limits for services
• Configure CPU limits for services

Rate optimization: If you can predict the CPU and memory requirements, you can save money with committed use discounts (CUDs).

Model selection: When you select models for your agentic AI system, consider the capabilities that are required for the tasks that the agents need to perform.

Prompt optimization: To rapidly improve and optimize prompt performance at scale and to eliminate the need for manual rewriting, use the Vertex AI prompt optimizer. The optimizer helps you efficiently adapt prompts across different models.

Model selection: The model that you select for your AI application directly affects both costs and performance. To identify the model that provides an optimal balance between performance and cost for your specific use case, test models iteratively. We recommend that you start with the most cost-efficient model and progress gradually to more powerful options.

Prompt engineering: The length of your prompts (input) and the generated responses (output) directly affect performance and cost. Write prompts that are short, direct, and provide sufficient context. Design your prompts to get concise responses from the model. For example, include phrases such as "summarize in 2 sentences" or "list 3 key points". For more information, see the best practices for prompt design.

Context caching: To reduce latency for requests that contain repeated content with high input token counts, use context caching.

Resource allocation: Depending on your performance requirements, configure the memory and CPU to be allocated to the Cloud Run service. For more information, see the following documentation:

• Configure memory limits for services
• Configure CPU limits for services

For more performance optimization guidance, see General Cloud Run development tips.