Best practices and reference architectures for VPC design

Last reviewed 2023-05-08 UTC

This guide introduces best practices and typical enterprise architectures for the design of virtual private clouds (VPCs) with Google Cloud. This guide is for cloud network architects and system architects who are already familiar with Google Cloud networking concepts.

General principles and first steps

Identify decision makers, timelines, and pre-work

As a first step in your VPC network design, identify the decision makers, timelines, and pre-work necessary to ensure that you can address stakeholder requirements.

Stakeholders might include application owners, security architects, solution architects, and operations managers. The stakeholders themselves might change depending on whether you are planning your VPC network for a project, a line of business, or the entire organization.

Part of the pre-work is to get the team acquainted with concepts and terminology around VPC network design. Useful documents include the following:

Consider VPC network design early

Make VPC network design an early part of designing your organizational setup in Google Cloud. It can be disruptive to your organization if you later need to change fundamental things such as how your network is segmented or where your workloads are located.

Different VPC network configurations can have significant implications for routing, scale, and security. Careful planning and deep understanding of your specific considerations helps you to create a solid architectural foundation for incremental workloads.

Keep it simple

Keeping the design of your VPC network topology simple is the best way to ensure a manageable, reliable, and well-understood architecture.

Use clear naming conventions

Make your naming conventions simple, intuitive, and consistent. This ensures that administrators and end users understand the purpose of each resource, where it is located, and how it is differentiated from other resources.

Commonly accepted abbreviations of long words help with brevity. Using familiar terminology where possible helps with readability.

Consider the components illustrated in the following example when establishing your naming conventions:

  • Company name: Acme Company: acmeco
  • Business unit: Human Resources: hr
  • Application code: Compensation system: comp
  • Region code: northamerica-northeast1: na-ne1, europe-west1: eu-we1
  • Environment codes: dev, test, uat, stage, prod

In this example, the development environment for the human resources department's compensation system is named acmeco-hr-comp-eu-we1-dev.

For other common networking resources, consider patterns like these:

  • VPC network
    syntax: {company name}-{description(App or BU)-label}-{environment-label}-{seq#}
    example: acmeco-hr-dev-vpc-1

  • Subnet
    syntax: {company-name}-{description(App or BU)-label}-{region/zone-label}
    example: acmeco-hr-na-ne1-dev-subnet

  • Firewall rule
    syntax: {company-name}-{description(App or BU)-label}{source-label}-{dest-label}-{protocol}-{port}-{action}
    example: acmeco-hr-internet-internal-tcp-80-allow-rule

  • IP route
    syntax: {priority}-{VPC-label}-{tag}-{next hop}
    example: 1000-acmeco-hr-dev-vpc-1-int-gw

Addresses and subnets

Use custom mode VPC networks

When you start your first project, you begin with the default network, which is an auto mode VPC network named default. Auto mode networks automatically create subnets and corresponding subnet routes whose primary IP ranges are /20 CIDRs in each Google Cloud region using a predictable set of RFC 1918 address ranges. The default network also automatically includes some pre-populated firewall rules.

Though auto mode networks can be useful for early exploration, custom mode VPC networks are better suited for most production environments.

We recommend that enterprises use VPC networks in custom mode from the beginning for the following reasons:

  • Custom mode VPC networks better integrate into existing IP address management schemes. Because all auto mode networks use the same set of internal IP ranges, auto mode IP ranges might overlap when connected with your on-premises corporate networks.
  • You can't connect two auto mode VPC networks together using VPC Network Peering because their subnets use identical primary IP ranges.
  • Auto mode subnets all have the same name as the network. You can choose unique, descriptive names for custom mode subnets, making your VPC networks more understandable and maintainable.
  • When a new Google Cloud region is introduced, auto mode VPC networks automatically get a new subnet in that region. Custom mode VPC networks only get new subnets if you specify them. This can be important for both sovereignty and IP address management reasons.

If it has no resources, you can delete the default network. You cannot delete a VPC network until you have removed all resources, including Virtual Machine (VM) instances, that depend on it.

For more details of the differences between auto mode and custom mode VPC networks, see the VPC network overview.

Group applications into fewer subnets with larger address ranges

Conventionally, some enterprise networks are separated into many small address ranges for a variety of reasons. For example, this might have been done to identify or isolate an application or keep a small broadcast domain.

However, we recommend that you group applications of the same type into fewer, more manageable subnets with larger address ranges in the regions you want to operate.

Unlike other networking environments in which a subnet mask is used, Google Cloud uses a software-defined networking (SDN) approach to provide a full mesh of reachability between all VMs in the global VPC network. The number of subnets does not affect routing behaviour.

You can use service accounts or network tags to apply specific routing policies or firewall rules. Identity in Google Cloud is not based solely on the subnet IP address.

Some VPC features—including Cloud NAT, Private Google Access, VPC Flow Logs, and alias IP ranges—are configured per subnet. If you need more fine-grained control of these features, use additional subnets.

Single VPC network and Shared VPC

Start with a single VPC network for resources that have common requirements

For many simple use cases, a single VPC network provides the features that you need, while being easier to create, maintain, and understand than the more complex alternatives. By grouping resources with common requirements and characteristics into a single VPC network, you begin to establish the VPC network border as the perimeter for potential issues.

For an example of this configuration, see the single project, single VPC network reference architecture.

Factors that might lead you to create additional VPC networks include scale, network security, financial considerations, operational requirements, and identity and access management (IAM).

Use Shared VPC for administration of multiple working groups

For organizations with multiple teams, Shared VPC provides an effective tool to extend the architectural simplicity of a single VPC network across multiple working groups. The simplest approach is to deploy a single Shared VPC host project with a single Shared VPC network and then attach team service projects to the host project network.

In this configuration, network policy and control for all networking resources are centralized and easier to manage. Service project departments can configure and manage non-network resources, enabling a clear separation of responsibilities for different teams in the organization.

Resources in those projects can communicate with each other more securely and efficiently across project boundaries using internal IP addresses. You can manage shared network resources—such as subnets, routes, and firewalls—from a central host project, so you can enforce consistent network policies across the projects.

For an example of this configuration, see the Single host project, multiple service projects, single Shared VPC reference architecture.

Grant the network user role at the subnet level

The centralized Shared VPC administrator can grant IAM members the network user (networkUser) role either at a subnet level, for fine-grained service-project authorization, or for all subnets at the host project level.

Following the principle of least privilege, we recommend granting the network user role at the subnet level to the associated user, service account, or group.

Because subnets are regional, this granular control allows you to specify which regions each service project can use to deploy resources.

With Shared VPC architectures, you also have the flexibility to deploy multiple Shared VPC host projects within your organization. Each Shared VPC host project can then accommodate a single or multiple Shared VPC networks. In this configuration, different environments can easily enforce different policy concerns.

For more information, see IAM roles for networking.

Use a single host project if resources require multiple network interfaces

If you have a service project that needs to deploy resources to multiple isolated VPC networks—for example, VM instances with multiple network interfaces—your host project must contain all of the VPC networks that provide the services. This is because a service project is allowed to attach to only one host project.

Service project to multiple VPCs

Use multiple host projects if resource requirements exceed the quota of a single project

In cases where the aggregate resource requirements of all VPC networks can't be met within a project's quota, use an architecture with multiple host projects with a single Shared VPC network per host project, rather than a single host project with multiple Shared VPC networks. It's important to evaluate your scale requirements, because using a single host project requires multiple VPC networks in the host project, and quotas are enforced at the project level.

For an example of this configuration, see the Multiple host projects, multiple service projects, multiple Shared VPC reference architecture.

Use multiple host projects if you need separate administration policies for each VPC network

Because each project has its own quota, use a separate Shared VPC host project for every VPC network to scale aggregate resources. This allows each VPC network to have separate IAM permissions for networking and security management, because IAM permissions are also implemented at the project level. For example, if you deploy two VPC networks (VPC network A and VPC network B) into the same host project, the network administrator (networkAdmin) role applies to both VPC networks.

Deciding whether to create multiple VPC networks

Create a single VPC network per project to map VPC resource quotas to projects

Quotas are constraints applied at the project or network level. All resources have an initial default quota meant to protect you from unexpected resource usage. However, many factors might lead you to want more quota. For most resources, you can request additional quota.

We recommend creating a single VPC network per project if you expect to grow beyond the default VPC resource quotas. This makes it easier to map project-level quota increases to each VPC network rather than to a combination of VPC networks in the same project.

Limits are designed to protect system resources in aggregate. Limits generally can't be raised easily, although Google Cloud support and sales teams can work with you to increase some limits.

See VPC resource quotas and limits for current values.

Google Support can increase some scaling limits, but there might be times when you need to build multiple VPC networks to meet your scaling requirements. If your VPC network has a requirement to scale beyond the limits, discuss your case with Google Cloud sales and support teams about the best approach for your requirements.

Create a VPC network for each autonomous team, with shared services in a common VPC network

Some large enterprise deployments involve autonomous teams that each require full control over their respective VPC networks. You can meet this requirement by creating a VPC network for each business unit, with shared services in a common VPC network (for example, analytic tools, CI/CD pipeline and build machines, DNS/Directory services). For more information about creating a common VPC network for shared services, see the shared services section.

Create VPC networks in different projects for independent IAM controls

A VPC network is a project-level resource with fine-grained, project-level identity and access management (IAM) controls, including the following roles:

  • networkAdmin
  • securityAdmin
  • networkUser
  • networkViewer

By default, IAM controls are deployed at the project level and each IAM role applies to all VPC networks within the project.

If you require independent IAM controls per VPC network, create your VPC networks in different projects.

If you require IAM roles scoped to specific Compute Engine resources such as VM instances, disks, and images, use IAM policies for Compute Engine resources.

Isolate sensitive data in its own VPC network

For companies that deal with compliance initiatives, sensitive data, or highly regulated data that is bound by compliance standards such as HIPAA or PCI-DSS, further security measures often make sense. One method that can improve security and make it easier to prove compliance is to isolate each of these environments into its own VPC network.

Connecting multiple VPC networks

Choose the VPC connection method that meets your cost, performance, and security needs

The next step after deciding to implement multiple VPC networks is connecting those VPC networks. VPC networks are isolated tenant spaces within Google's Andromeda SDN, but there are several ways that you can enable communication between them. The subsequent sections provide best practices for choosing a VPC connection method.

The advantages and disadvantages of each are summarized in the following table, and subsequent sections provide best practices for choosing a VPC connection method.

Advantages Disadvantages
VPC Network Peering
  • Easy to configure.
  • Low management overhead.
  • High bandwidth.
  • Low egress charges (same as single VPC network).
  • Each VPC network maintains its own distributed firewall.
  • Each VPC network maintains its own IAM accounts and permissions.
  • Non-transitive.
  • Scaling numbers are bound to the aggregate group of peered VPC networks. This includes the number of VMs, routes, and internal forwarding rules.
  • Requires non-overlapping address space.
  • Static and dynamic routes are not propagated.
  • Source tags and source service accounts of the sending VM are not propagated across VPC Network Peering.
External routing (public IP or NAT gateway)
  • No configuration needed.
  • Full isolation between VPC networks.
  • Overlapping IP address space is possible.
  • High bandwidth.
  • Egress charges for VMs within the same zone are higher than for other options, such as VPC Network Peering.
  • VMs need to be exposed using external IP addresses.
  • No firewalling using private IP addresses.
  • Static and dynamic routes are not propagated.
  • Source tags and source service accounts of the sending VM are not honored by peered networks.
Cloud VPN
  • Cloud VPN enables transitive topologies for hub and spoke.
  • Scalable through ECMP.
  • 99.99% service availability SLA on HA VPN.
  • Management overhead.
  • Billed at internet egress rates.
  • Slightly higher latency.
  • Throughput limited to 3 Gbps per tunnel.
  • Lower MTU because of additional tunnel encapsulation.
  • Source tags and source service accounts of the sending VM are lost across the tunnel.
Cloud Interconnect - Dedicated or Partner
  • Supported SLAs based on redundancy in deployment:
  • Each link of a Dedicated Interconnect is a 10 Gbps or 100 Gbps connection. Multiple interconnect links can be bundled to increase throughput, with a maximum of 8 x 10 Gbps circuits (80 Gbps), or 2 x 100 Gbps (200 Gbps) circuits for each Dedicated Interconnect connection.
  • Possible to use ECMP across multiple interconnects to increase throughput.
  • Additional cost and egress charges for traffic sent between VPC networks over an Interconnect connection.
  • Traffic is not encrypted.
  • Additional latency over cloud-native solutions.
  • Additional hardware devices in the path that can fail.
Multiple network interfaces (Multi-NIC)
  • Scalable through managed instance groups and ECMP routes across instances.
  • Individual VMs have [bandwidth limits](/compute/docs/network-bandwidth).
  • Limit of 8 interfaces per instance.
  • The internal passthrough Network Load Balancer supports sending traffic to [any interface](/load-balancing/docs/internal#backend-service-multi-nic) of the VM. Other load balancers support only the default network interface (nic0) of a VM.

Use VPC Network Peering if you won't exceed resource limits

We recommend using VPC Network Peering when you need to connect VPC networks and can't use a single Shared VPC, as long as the totals of the resources needed for all directly connected peers do not exceed the limits on VM instances, number of peering connections, and internal forwarding rules.

VPC Network Peering enables two VPC networks to connect with each other internally over Google's SDN, whether or not they belong to the same project or the same organization. VPC Network Peering merges the control plane and flow propagation between each peer, allowing the same forwarding characteristics as if all the VMs were in the same VPC network. When VPC networks are peered, all subnets, alias IP ranges, and internal forwarding rules are accessible, and each VPC network maintains its own distributed firewall. VPC Network Peering is not transitive.

VPC Network Peering is the preferred method for connecting VPC networks for the following reasons:

  • No gateway bottleneck—Traffic forwards across peers as if the VMs were in the same VPC network.
  • Billing—There are no additional charges; you are billed as if the VMs were in the same VPC network.

Use external routing if you don't need private IP address communication

If you don't need private IP address communication, you can use external routing with external IP addresses or a NAT gateway.

When a VPC network is deployed, a route to Google's default internet gateway is provisioned with a priority of 1000. If this route exists, and a VM is given an external IP address, VMs can send outbound (egress) traffic through Google's internet gateway. You can also deploy services behind one of Google's many public load-balancing offerings, which allows the services to be reached externally.

Externally addressed VMs communicate with each other privately over Google's backbone, regardless of region and Network Service Tiers. Use Google Cloud firewall rules to control external inbound (ingress) traffic to your VMs.

External routing is a good option for scaling purposes, but it's important to understand how public routing affects costs. For details, see the Network pricing documentation.

Use Cloud VPN to connect VPC networks that would otherwise exceed aggregate peering group limits

HA VPN provides a managed service to connect VPC networks by creating IPsec tunnels between sets of endpoints. If you configure your Cloud Routers with custom route advertisements, you can enable transitive routing across VPC networks and hub-and-spoke topologies as described later in this document. Using Network Connectivity Center, you can use HA VPN tunnels as a transit network between on-premises networks, as explained in the Cloud VPN documentation.

Cloud VPN does not support large MTU. For details, see MTU considerations.

Use Cloud Interconnect to control traffic between VPC networks through an on-premises device

Cloud Interconnect extends your on-premises network to Google's network through a highly available, low-latency connection. You can use Dedicated Interconnect to connect directly to Google or use Partner Interconnect to connect to Google through a supported service provider.

Dedicated Interconnect provides high-speed L2 service between Google and a colocation provider or on-premises location. This allows you to use on-premises routing equipment to route between VPC networks and use existing on-premises security and inspection services to filter all traffic between VPC networks. All traffic between the two VPC networks has extra latency because of an additional round trip through the on-premises system.

Partner Interconnect provides similar capabilities, as well as the ability to peer directly with a provider at L3 and have the provider route between VPC networks. This enables access to internal IP addresses across your on-premises network and Google Cloud VPC networks without a NAT device or VPN tunnel.

Because many enterprise security appliances can be used on Google Cloud using VMs with multiple network interfaces, using Cloud Interconnect to route traffic between VPC networks is not necessary except in very few cases where all traffic has to flow through an on-premises appliance because of corporate policies.

Use virtual appliances to control traffic between VPC networks through a cloud device

Multiple network interface VMs are common for VPC networks that require additional sec