Building internet connectivity for private VMs

Last reviewed 2023-05-15 UTC

This document describes options for connecting to and from the internet using Compute Engine resources that have private IP addresses. This is helpful for developers who create Google Cloud services and for network administrators of Google Cloud environments.

This tutorial assumes you are familiar with deploying VPCs, with Compute Engine, and with basic TCP/IP networking.

Objectives

  • Learn about the options available for connecting between private VMs outside their VPC.
  • Create an instance of Identity-Aware Proxy (IAP) for TCP tunnels that's appropriate for interactive services such as SSH.
  • Create a Cloud NAT instance to enable VMs to make outbound connections to the internet.
  • Configure an HTTP load balancer to support inbound connections from the internet to your VMs.

Costs

This tutorial uses billable components of Google Cloud, including:

Use the pricing calculator to generate a cost estimate based on your projected usage. We calculate that the total to run this tutorial is less than US$5 per day.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  5. Make sure that billing is enabled for your Google Cloud project.

Introduction

Private IP addresses provide a number of advantages over public (external) IP addresses, including:

  • Reduced attack surface. Removing external IP addresses from VMs makes it more difficult for attackers to reach the VMs and exploit potential vulnerabilities.
  • Increased flexibility. Introducing a layer of abstraction, such as a load balancer or a NAT service, allows more reliable and flexible service delivery when compared with static, external IP addresses.

This solution discusses three scenarios, as described in the following table:

Interactive Fetching Serving
An SSH connection is initiated from a remote host directly to a VM using IAP for TCP.

Example: Remote administration using SSH or RDP

A connection is initiated by a VM to an external host on the internet using Cloud NAT.

Example: OS updates, external APIs

A connection is initiated by a remote host to a VM through a global Google Cloud load balancer.

Example: Application frontends, WordPress

Some environments might involve only one of these scenarios. However, many environments require all of these scenarios, and this is fully supported in Google Cloud.

The following sections describe a multi-region environment with an HTTP load-balanced service backed by two VMs in two regions. These VMs use Cloud NAT for outgoing communications. For administration, the VMs are accessible through SSH tunneled over IAP.

The following diagram provides an overview of all three use cases and the relevant components.

Architecture of solution showing flow from client through load balancer to VM instances, Cloud NAT for access from the instances to the internet, and IAP to allow direct SSH access from a client to the instances.

Creating VM instances

To begin the tutorial, you create a total of four virtual machine (VM) instances—two instances per region in two different regions. You give all of the instances the same tag, which is used later by a firewall rule to allow incoming traffic to reach your instances.

The following diagram shows the VM instances and instance groups you create, distributed in two zones.

Architecture of solution highlighting the four VM instances in 2 zones.

The startup script that you add to each instance installs Apache and creates a unique home page for each instance.

The procedure includes instructions for using both the Google Cloud console and gcloud commands. The easiest way to use gcloud commands is to use Cloud Shell.

Console

  1. In the Google Cloud console, go to the VM instances page:

    GO TO THE VM INSTANCES PAGE

  2. Click Create instance.

  3. Set Name to www-1.

  4. Set the Zone to us-central1-b.

  5. Click Management, Security, Disks, Networking, Sole Tenancy.

  6. Click Networking and make the following settings:

    • For HTTP traffic, in the Network tags box, enter http-tag.
    • Under Network Interfaces, click .
    • Under External IP, select None.
  7. Click Management and set Startup script to the following:

    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>server 1</h1></body></html>' | sudo tee /var/www/html/index.html
    
  8. Click Create.

  9. Create www-2 with the same settings, except set Startup script to the following:

    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>server 2<h1></body></html>' | sudo tee /var/www/html/index.html
    
  10. Create www-3 with the same settings, except set Zone to europe-west1-b and set Startup script to the following:

    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>server 3</h1></body></html>' | sudo tee /var/www/html/index.html
    
  11. Create www-4 with the same settings, except set Zone to europe-west1-b and set Startup script to the following:

    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>server 4</h1></body></html>' | sudo tee /var/www/html/index.html
    

gcloud

  1. Open Cloud Shell:

    OPEN Cloud Shell

  2. Create an instance named www-1 in us-central1-bwith a basic startup script:

    gcloud compute instances create www-1 \
        --image-family debian-9 \
        --image-project debian-cloud \
        --zone us-central1-b \
        --tags http-tag \
        --network-interface=no-address \
        --metadata startup-script="#! /bin/bash
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>www-1</h1></body></html>' | tee /var/www/html/index.html
    EOF"
    
  3. Create an instance named www-2 in us-central1-b:

    gcloud compute instances create www-2 \
        --image-family debian-9 \
        --image-project debian-cloud \
        --zone us-central1-b \
        --tags http-tag \
        --network-interface=no-address \
        --metadata startup-script="#! /bin/bash
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>www-2</h1></body></html>' | tee /var/www/html/index.html
    EOF"
    
  4. Create an instance named www-3, this time in europe-west1-b:

    gcloud compute instances create www-3 \
        --image-family debian-9 \
        --image-project debian-cloud \
        --zone europe-west1-b \
        --tags http-tag \
        --network-interface=no-address \
        --metadata startup-script="#! /bin/bash
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>www-3</h1></body></html>' | tee /var/www/html/index.html
    EOF"
    
  5. Create an instance named www-4, this one also in europe-west1-b:

    gcloud compute instances create www-4 \
        --image-family debian-9 \
        --image-project debian-cloud \
        --zone europe-west1-b \
        --tags http-tag \
        --network-interface=no-address \
        --metadata startup-script="#! /bin/bash
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>www-4</h1></body></html>' | tee /var/www/html/index.html
    EOF"
    

Terraform

  1. Open Cloud Shell:

    OPEN Cloud Shell

  2. Clone the repository from GitHub:

    git clone https://github.com/GoogleCloudPlatform/gce-public-connectivity-terraform

  3. Change the working directory to the repository directory:

    cd iap

  4. Install Terraform.

  5. Replace [YOUR-ORGANIZATION-NAME] in the scripts/set_env_vars.sh file with your Google Cloud organization name.

  6. Set environment variables:

    source scripts/set_env_vars.sh

  7. Apply the Terraform configuration:

    terraform apply

Configuring IAP tunnels for interacting with instances

To log in to VM instances, you connect to the instances using tools like SSH or RDP. In the configuration you're creating in this tutorial, you can't directly connect to instances. However, you can use TCP forwarding in IAP, which enables remote access for these interactive patterns.

For this tutorial, you use SSH.

In this section you do the following:

  1. Connect to a Compute Engine instance using the IAP tunnel.
  2. Add a second user with IAP tunneling permission in IAM.

The following diagram illustrates the architecture that you build in this section. The grey areas are discussed in other parts of this tutorial.

Architecture of solution showing IAP providing access for SSH access between a client and the instances.

Limitations of IAP

  • Bandwidth: The IAP TCP forwarding feature isn't intended for bulk transfer of data. IAP reserves the right to rate-limit users who are deemed to be abusing this service.
  • Connection length: IAP won't disconnect active sessions unless required for maintenance.
  • Protocol: IAP for TCP doesn't support UDP.

Create firewall rules to allow tunneling

In order to connect to your instances using SSH, you need to open an appropriate port on the firewall. IAP connections come from a specific set of IP addresses (35.235.240.0/20). Therefore, you can limit the rule to this CIDR range.

Console

  1. In the Google Cloud console, go to the Firewall policies page:

    GO TO THE FIREWALL POLICIES PAGE

  2. Click Create firewall rule.

  3. Set Name to allow-ssh-from-iap.

  4. Leave VPC network as default.

  5. Under Targets, select Specified target tags.

  6. Set Target tags to http-tag.

  7. Leave Source filter set to IP ranges.

  8. Set Source IP Ranges to 35.235.240.0/20.

  9. Set Allowed protocols and ports to tcp:22.

  10. Click Create.

    It might take a moment for the new firewall rule to be displayed in the console.

gcloud

  • Create a firewall rule named allow-ssh-from-iap:

    gcloud compute firewall-rules create allow-ssh-from-iap \
        --source-ranges 35.235.240.0/20 \
        --target-tags http-tag \
        --allow tcp:22
    

Terraform

  1. Copy the firewall rules Terraform file to the current directory:

    cp iap/vpc_firewall_rules.tf .

  2. Apply the Terraform configuration:

    terraform apply

Test tunneling

  • In Cloud Shell, connect to instance www-1 using IAP:

    gcloud compute ssh www-1 \
        --zone us-central1-b \
        --tunnel-through-iap
    

If the connection succeeds, you have an SSH session that is tunneled through IAP directly to your private VM.

Grant access to additional users

IAP uses your existing project roles and permissions when you connect to VM instances. By default, instance owners are the only users that have the IAP Secured Tunnel User role. If you want to allow other users to access your VMs using IAP tunneling, you need to grant this role to those users.

  1. In the Google Cloud console, go to Security > Identity-Aware Proxy:

    IAP option in the Security page of the Google Cloud console.

    If you see a message that tells you that you need to configure the OAuth consent screen, disregard the message; it's not relevant to IAP for TCP.

  2. Select the SSH and TCP Resources tab.

  3. Select the VMs you've created:

    Console showing all 4 instances selected.

  4. On the right-hand side, click Add Principal.

  5. Add the users you want to grant permissions to, select the IAP-secured Tunnel User role, and then click Save.

Summary

You can now connect to your instances using SSH to administer the instances or troubleshoot them.

Many applications need to make outgoing connections in order to download patches, connect with partners, or download resources. In the next section, you configure Cloud NAT to allow your VMs to reach these resources.

Deploying Cloud NAT for fetching

The Cloud NAT service allows Google Cloud VM instances that don't have external IP addresses to connect to the internet. Cloud NAT implements outbound NAT in conjunction with a default route to allow your instances to reach the internet. It doesn't implement inbound NAT. Hosts outside of your VPC network can respond only to established connections initiated by your instances; they cannot initiate their own connections to your instances using Cloud NAT. NAT is not used for traffic within Google Cloud.

Cloud NAT is a regional resource. You can configure it to allow traffic from all primary and secondary IP address ranges of subnets in a region, or you can configure it to apply to only some of those ranges.

In this section, you configure a Cloud NAT gateway in each region that you used earlier. The following diagram illustrates the architecture that you build in this section. The grey areas are discussed in other parts of this tutorial.

Architecture of solution showing Cloud NAT instances between the instances and the internet.

Create a NAT configuration using Cloud Router

You must create the Cloud Router instance in the same region as the instances that need to use Cloud NAT. Cloud NAT is only used to place NAT information onto the VMs; it's not used as part of the actual Cloud NAT gateway.

This configuration allows all instances in the region to use Cloud NAT for all primary and alias IP ranges. It also automatically allocates the external IP addresses for the NAT gateway. For more options, see the gcloud compute routers documentation.

Console

  1. Go to the Cloud NAT page:

    GO TO THE CLOUD NAT PAGE

  2. Click Get started or Create NAT gateway.

  3. Set Gateway name to nat-config.

  4. Set VPC network to default.

  5. Set Region to us-central1.

  6. Under Cloud Router, select Create new router, and then do the following:

    • Set Name to nat-router-us-central1.
    • Click Create.
  7. Click Create.

  8. Repeat the procedure, but substitute these values:

    • Name: nat-router-europe-west1
    • Region: europe-west1

gcloud

  1. Create Cloud Router instances in each region:

    gcloud compute routers create nat-router-us-central1 \
        --network default \
        --region us-central1
    
    gcloud compute routers create nat-router-europe-west1 \
        --network default \
        --region europe-west1
    
  2. Configure the routers for Cloud NAT:

    gcloud compute routers nats create nat-config \
        --router-region us-central1 \
        --router nat-router-us-central1 \
        --nat-all-subnet-ip-ranges \
         --auto-allocate-nat-external-ips
    
    gcloud compute routers nats create nat-config \
        --router-region europe-west1 \
        --router nat-router-europe-west1 \
        --nat-all-subnet-ip-ranges \
        --auto-allocate-nat-external-ips
    

Terraform

  1. Copy the Terraform NAT configuration file to the current directory:

    cp nat/vpc_nat_gateways.tf .

  2. Apply the Terraform configuration:

    terraform apply

Test Cloud NAT configuration

You can now test that you're able to make outbound requests from your VM instances to the internet.

  1. Wait up to 3 minutes for the NAT configuration to propagate to the VM.
  2. In Cloud Shell, connect to your instance using the tunnel you created:

    gcloud compute ssh www-1 --tunnel-through-iap
    
  3. When you're logged in to the instance, use the curl command to make an outbound request:

    curl example.com
    

    You see the following output:

    <html>
    <head>
    <title>Example Domain</title>
    ...
    ...
    ...
    </head>
    
    <body>
    <div>
        <h1>