Patterns for connecting other cloud service providers with Google Cloud

Last reviewed 2023-07-05 UTC

This document helps cloud architects and operations professionals decide how to connect Google Cloud with other cloud service providers (CSP) such as Amazon Web Services (AWS) and Microsoft Azure. In a multicloud design, connections between CSPs allow data transfers between your virtual networks. This document also provides information on how to implement the option that you choose.

Many organizations operate on multiple cloud platforms, either as a temporary measure during a migration or because the organization has a long-term multicloud strategy.

For data exchange between Google Cloud and other CSPs, there are multiple options discussed in this document:

These options differ in terms of transfer speed, latency, reliability, service level agreements (SLAs), complexity, and costs. This document describes each option and its advantages and disadvantages in detail and ends with a comparison of all of the options.

This document covers data transfers between virtual machines residing in Google Cloud and other CSP's virtual networks. For data stored in other Google Cloud products such as Cloud Storage and BigQuery, see the section covering these products.

This document can act as a guide to evaluate the options to transfer data between Google Cloud and one or more other CSPs, based on your requirements and capabilities.

The concepts in this document apply in multiple cases:

  • When you are planning to transfer large amounts of data for a short period of time, for example, for a data migration project.
  • If you run continuous data transfers between multiple cloud providers, for example, because your compute workloads run on another CSP while your big data workloads use Google Cloud.

Initial considerations

Before you choose how to connect your cloud environments, it's important that you look at which regions you choose in each environment and that you have a strategy for transferring data that doesn't reside in virtual network environments.

Choose cloud regions

If both Google Cloud and other CSP's resources are in regions that are geographically close to each other, there is a cost and latency advantage for data transfers between cloud providers.

The following diagram illustrates the flow of data between Google Cloud and other CSPs. Architecture of data flowing between Google Cloud and other CSPs.

Regardless of the method of transfer, data flows from Google Cloud to the other CSP as follows:

  • From the Google Cloud region where resources are hosted to Google's edge POP.
  • Through a third-party facility between Google Cloud and the other CSP.
  • From the other CSP's edge POP to the region where resources are located inside the other CSP's network.

Data that flows from the other CSP to Google Cloud travels the same path, but in the opposite direction.

The end-to-end path determines the latency of the data transfer. For some solutions, the network costs also increase when the edge POPs of both providers aren't in one metropolitan area. Details are listed in the following pricing section of each solution.

Make sure you choose suitable regions in each cloud that can host your intended workloads. Visit the Locations page for Google Cloud and similar pages for other CSPs, such as the AWS region table or Azure products by region. For general help in selecting one or multiple locations in Google Cloud, review Best practices for Compute Engine region selection.

Transfer of data in Cloud Storage and BigQuery

Only data that resides inside a Google Cloud VPC environment passes a Cloud VPN tunnel or a Cloud Interconnect connection by default.

If you want to transfer data to and from other Google services, you can use Private Service Connect and Private Google Access for on-premises hosts from the other CSP's environment.

If you want to transfer another CSP's object storage, database, data warehouse, or other products, check their documentation to see if data can pass through their interconnect or managed VPN product. Otherwise, you might be able to pass this data through proxy virtual machines that you set up in the respective CSP's environment to make it pass through the connection you want.

For transferring data to Cloud Storage or BigQuery, you can also use Storage transfer service or BigQuery transfer service.

Transfer through external IP addresses over the internet

The easiest way to transfer data between Google Cloud and another CSP is to use the internet and transfer the data by using external IP addresses.

The following diagram illustrates the elements for this solution.

Architecture of data transferring between Google Cloud and another CSP through external IP address over the internet.

Between Google's network edge and the other CSP's network edge, data passes through the internet or uses a direct peering between Google Cloud and the other CSP. Data can only pass between resources that have public IP addresses assigned.

How Google connects to other networks

Google's edge POPs are where Google's network connects to other networks that collectively form the internet. Google is present in numerous locations around the world.

On the internet, every network is assigned an autonomous system number (ASN) that encompasses the network's internal network infrastructure and routes. Google's primary ASN is 15169.

There are two primary ways a network can send or receive traffic to or from Google:

  • Buy internet service from an internet service provider (ISP) that already has connectivity to Google (AS15169). This option is generally referred to as IP transit and is similar to what consumers and businesses purchase from access providers at their homes and businesses.
  • Connect directly to Google (AS15169). This option, called peering, lets a network directly send and receive traffic to Google (AS15169) without using a third-party network. At scale, peering is generally preferred over transit because network operators have more control over how and where traffic is exchanged, allowing for optimization of performance and cost. Peering is a voluntary system; when choosing to peer, network operators decide jointly which facilities to connect in, how much bandwidth to provision, how to split infrastructure costs, and any other details required to set up the connections. AS15169 has an open peering policy, which means as long as a network meets the technical requirements, Google will peer with them.

Peering is a private, mutually beneficial agreement between two independent networks, and as such, networks generally do not disclose publicly who they peer with at particular locations, how much bandwidth is available, etc. But due to scale and an open policy, Google is directly peered with almost all major ISPs and cloud services providers in multiple locations and across regions. The Google network team works directly with their counterparts at these networks to provide sufficient peering capacity to meet demand.

You can read more about how internet peering works at The Internet Peering Playbook.

Implementation

In this setup, all virtual machines that transfer data between Google Cloud and other cloud service providers must have a public IP address. On one end, the firewall must be opened to allow a connection from the public IP address of the other cloud provider. No extra steps are necessary because the data exchange happens over the existing internet connectivity.

Transfer speed and latency

While there is no guaranteed speed and latency on the path through the internet, typically, major CSPs and Google exchange data directly in multiple locations around the world. Capacity is shared with other customers and services, but, often due to the direct connection between both providers, latency is similar or lower than other options.

We recommend that you test the latency and bandwidth between Google Cloud and the other CSPs in your chosen regions. You can do a quick benchmark by using tools such as iperf or netperf, or run a more complete custom benchmark based on your app. While conditions might change over time, the benchmark can provide an indication of the performance you can expect and if this solution fulfills your needs. If you require a dedicated bandwidth between both environments, you should choose another solution.

Note that products from different vendors may have performance characteristics that might not always align. For example, per-tunnel IPsec VPN capacity might vary from vendor to vendor.

Security

Traffic over the internet isn't encrypted and might pass through third-party internet service providers (ISPs), autonomous systems, and facilities. Therefore, you should encrypt sensitive traffic at the application layer or choose another solution.

Reliability and SLA

Google Cloud generally has multiple diverse paths for internet connectivity from regions, and there are direct peering connections with other major CSPs at multiple locations around the globe.

However, Google Cloud doesn't provide any SLAs for connectivity to other CSPs over the internet. While you should check SLAs for your other CSPs, they typically only refer to internet connectivity as a whole and not specific providers.

Providers may have different routing policies which can impact availability. For example, on its peeringdb page, Amazon explains that many AWS regions announce only local routes, because AWS VPCs are regional only (Google Cloud VPCs are global.) This means customers may be relying on links at a single peering location, as traffic leaving Google Cloud can only use those links to reach the destination. This is fine under normal operation with traffic being exchanged in-region, but it is advisable for customers to architect for multi-region deployments to tolerate regional failures. This could include setting up additional gateways, HA VPNs, virtual network peering, or other multi-region topologies in the third-party cloud.

Applications should also be built in such a way that they 'fail open' as Google SRE recommends in the SRE book. For example, if you build an application that relies on being able to reach a third-party service via Internet routing, make sure that the application still functions, or at least returns helpful error messages to the user in the event of connectivity issues.

When issues with Internet routing do occur, the Google network team will attempt to restore connectivity with the third-party. However, not all issues will be under Google's control. So in some cases, repair might depend on a third party (ISP or cloud provider) taking restorative action. Customers have the most influence over how operators respond to outages, so make sure you have support coverage with all providers and a plan to escalate issues should something go wrong. Also perform periodic BCP (business continuity process) drills to ensure the resilience of applications architected on multicloud.

Pricing

For data transfers over the internet, normal internet egress rates apply for traffic leaving Google Cloud. In cases where latency isn't critical, using the Standard Network Service Tier provides a lower pricing tier.

The other CSPs have their own charges for data transfers. In many cases, they only charge for traffic egressing their network. Consult your CSP's price list for example for AWS, see EC2 data transfer charges and for Azure, see Bandwidth Pricing Details.

Managed VPN between cloud providers

You can use managed VPN services from both cloud providers, which has two benefits. It provides an encrypted channel between virtual networks in both cloud environments, and it lets you transfer data by using private IP addresses. This is an extension to the previous solution of transferring over the internet without requiring any hardware or partners.

The following diagram illustrates the elements for this solution.

Architecture of data transferring between Google Cloud and another CSP by using a managed VPN,

Using this solution, data is encrypted on Google Cloud using Cloud VPN and a VPN solution for the other CSP. The data transfer between Google Cloud and the other CSP uses the internet like in the previous solution.

Implementation

Google offers Cloud VPN as a managed VPN service for encrypted IPsec tunnels, which can be used on the Google end. Other CSPs offer their own managed VPN products, which you can use to build IPsec VPN tunnels between both environments. For example, AWS offers AWS Site-to-Site VPN and Azure offers Azure VPN gateway. You can connect your virtual networks between the environments by using VPN tunnels.

IP addresses in the two environments must not overlap because there is no network address translation (NAT) functionality available in Cloud VPN. On Cloud Router, overlapping routes could be removed from being exchanged between environments, but then no communication between the services using these IP addresses is possible.

With Cloud Router in global dynamic routing mode, you can reach all regions in a global VPC network by using only VPN tunnels to that region. For other CSPs, you might need VPN tunnels per region. If you have multiple virtual networks in a cloud environment that aren't peered, you have to connect all virtual networks that need to communicate with each other independently by using a VPN.

Google Cloud offers interoperability guides, which have step-by-step instructions for setting up VPN tunnels to other major cloud providers:

Transfer speed and latency

When you use managed VPN tunnels, data still follows the same internet path as if data were directly exchanged by using public IP addresses. The latency observed should be similar to the previous option with only a small latency overhead for the VPN tunnel. The available bandwidth is limited by the maximum bandwidth per VPN tunnel on Google Cloud, the maximum bandwidth of the other CSP's tunnels, and the available bandwidth over the internet path.

For higher bandwidth, you can deploy multiple tunnels in parallel. For more information on how to deploy such a solution, see Building high-throughput VPNs.

You can test latency and bandwidth as described in the last section, but conditions might vary over time, and there are no guarantees on latency or bandwidth.

Security

Traffic over IPsec VPN tunnels is encrypted by using ciphers accepted by both CSPs. For more information, see Cloud VPN supported IKE ciphers, AWS VPN FAQ, and Azure VPN IPsec/IKE parameters.

Reliability and SLA

Cloud VPN offers an SLA of 99.9%-99.99% depending on if Classic VPN or HA VPN is selected. Other CSPs sometimes offer SLAs on their managed VPN product, for example, AWS Site-to-Site VPN SLA and Azure SLA for VPN Gateway. However, these SLAs only cover the VPN gateways availability and don't include connectivity to other CSPs over the internet, so there is no SLA on the end-to-end solution.

To increase reliability, consider using multiple VPN gateways and tunnels on both Google Cloud and the other CSPs.

Pricing

For the managed VPN service, charges apply. For Google Cloud, an hourly charge applies see Cloud VPN pricing. For other CSPs, consult their price lists, for example, see AWS Site-to-Site VPN connection pricing or Azure VPN Gateway pricing.

In addition to hourly pricing for the VPN service, you have to pay for data transferred through the VPN gateways. For Google Cloud and many CSPs, standard internet data transfer charges apply, as detailed in Transfer through external IP addresses over the internet. In many cases, data transfer charges exceed the fixed cost for this solution.

Partner Interconnect with multicloud enabled partners

Partner Interconnect lets you connect a Virtual Private Cloud to another CSP's virtual networks through the network of select partners that offer direct multicloud solutions. You connect by deploying one or more virtual routing instances that take care of the necessary Border Gateway Protocol (BGP) setup.

The following diagram shows a redundant setup using two Partner Interconnect connections.

Architecture of data transferring between Google Cloud and another CSP by using two Partner Interconnect.

Routes are exchanged between Cloud Router and the gateway on the other CSP's side through a virtual routing instance that is managed by the partner providing the interconnect. Traffic flows through the partner network between Google Cloud and the other CSP.

Implementation

This solution requires you to set up multiple components:

  • On the Google Cloud side, you set up Partner Interconnect with an interconnect service provider that's serving your Google Cloud regions and offering multicloud connectivity to the other CSP.
  • On the other CSP, you have to use their interconnect product to connect to the same partner. For example, on AWS you can use Direct Connect and on Azure you can use ExpressRoute.
  • On the service provider partner side, you have to configure the virtual routing equipment providing the BGP sessions to Google Cloud and to the other CSP.

If IP address space between both CSP environments overlaps, your partner might offer NAT functionality for the virtual routing equipment. See the partners documentation for details.

Transfer speed and latency

This solution offers dedicated capacity between Google Cloud and other CSPs. Depending on the partner and the other CSP, the available attachment capacity might vary. On the Google Cloud side, Partner Interconnect is available with an attachment capacity between 50 Mbps and 50 Gbps.

Latency for this solution is the sum of the following:

  • Latency between the region in which your resources are hosted on Google Cloud and the interconnect location where the partner connects to Google Cloud.
  • Latency in the partners network to, from, and through the virtual routing instance towards the other CSP.
  • Latency from the other CSP's edge location where the interconnect with the partner takes place to the region where the resources are hosted in the CSP.

For the lowest possible latency, the edge locations of Google Cloud and the other CSP should be in the same metropolitan area, along with the virtual routing instance. For example, you might have a low latency connection if both CSP's cloud regions as well as the edge POP and the virtual routing instance are located in the Ashburn, Virginia area.

While Google Cloud and many other CSPs offer no latency guarantees for traffic towards their network edge, because there is a dedicated path and capacity through the partner, typically the latency in this solution is less variable than if you use external IP addresses or a VPN solution.

Security

Traffic over Partner Interconnect isn't encrypted by default. To help secure your traffic, you can deploy