This tutorial shows you how to create a private HTTP endpoint in a private Google Kubernetes Engine (GKE) cluster that receives Pub/Sub message events using Eventarc. To learn more about this event destination, see Route events to an internal HTTP endpoint in a VPC network.
Private GKE clusters are a type of Virtual Private Cloud (VPC)-native cluster where nodes only have internal IP addresses, which means that nodes and Pods are isolated from the internet by default. You can choose to have no client access, limited access, or unrestricted access to the control plane. You can't convert an existing, non-private cluster to a private cluster. For more information, see About private clusters.
You can run the following commands using the Google Cloud CLI in either your terminal or Cloud Shell.
Objectives
In this tutorial, you will:
- Create a proxy-only subnet in the default VPC network, and create a VPC firewall rule.
- Create a private GKE Autopilot cluster with no client access to the public endpoint.
- Create a Compute Engine virtual machine (VM) instance in a specified subnet of the VPC network.
- Establish an SSH connection to the VM instance and deploy an event receiver service on the VM instance.
- Deploy a Gateway in your cluster and an
HTTPRoute
manifest to configure the routing of traffic in Kubernetes to application backends. - Create a network attachment that lets a producer VPC network initiate connections to a consumer VPC network.
- Create an Eventarc trigger that routes Pub/Sub events to the event receiver on your VM instance.
- Publish a message to a Pub/Sub topic to generate an event, and view the body of the event in the application Pod logs.
Costs
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
- Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Cloud Resource Manager, Compute Engine, Eventarc, GKE, and Pub/Sub APIs:
gcloud services enable compute.googleapis.com
container.googleapis.com cloudresourcemanager.googleapis.com eventarc.googleapis.com pubsub.googleapis.com - Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Cloud Resource Manager, Compute Engine, Eventarc, GKE, and Pub/Sub APIs:
gcloud services enable compute.googleapis.com
container.googleapis.com cloudresourcemanager.googleapis.com eventarc.googleapis.com pubsub.googleapis.com - Update Google Cloud CLI components:
gcloud components update
- Sign in using your account:
gcloud auth login
-
If you are the project creator, you are granted the basic Owner role (
roles/owner
). By default, this Identity and Access Management (IAM) role includes the permissions necessary for full access to most Google Cloud resources and you can skip this step.If you are not the project creator, required permissions must be granted on the project to the appropriate principal. For example, a principal can be a Google Account (for end users) or a service account (for applications and compute workloads). For more information, see the Roles and permissions page for your event destination.
Required permissions
To get the permissions that you need to complete this quickstart, ask your administrator to grant you the following IAM roles on your project:
-
Compute Network Admin (
roles/compute.networkAdmin
) -
Compute Security Admin (
roles/compute.securityAdmin
) -
Eventarc Admin (
roles/eventarc.admin
) -
Kubernetes Engine Admin (
roles/container.admin
) -
Logs View Accessor (
roles/logging.viewAccessor
) -
Project IAM Admin (
roles/resourcemanager.projectIamAdmin
) -
Pub/Sub Publisher (
roles/pubsub.publisher
) -
Service Account Admin (
roles/iam.serviceAccountAdmin
) -
Service Account User (
roles/iam.serviceAccountUser
) -
Service Usage Admin (
roles/serviceusage.serviceUsageAdmin
)
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
Compute Network Admin (
Make note of the Compute Engine default service account as you will you attach it to an Eventarc trigger to represent the identity of the trigger for testing purposes. This service account is automatically created after enabling or using a Google Cloud service that uses Compute Engine, and with the following email format:
PROJECT_NUMBER-compute@developer.gserviceaccount.com
Replace
PROJECT_NUMBER
with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:gcloud projects describe PROJECT_ID --format='value(projectNumber)'
For production environments, we strongly recommend creating a new service account and granting it one or more IAM roles that contain the minimum permissions required and follow the principle of least privilege.
- If you enabled the Cloud Pub/Sub service agent on or before April
8, 2021, to support authenticated Pub/Sub push requests, grant
the Service
Account Token Creator role (
roles/iam.serviceAccountTokenCreator
) to the service agent. Otherwise, this role is granted by default:gcloud projects add-iam-policy-binding PROJECT_ID \ --member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \ --role=roles/iam.serviceAccountTokenCreator
Create a proxy-only subnet
Unless you create an organizational policy that prohibits it, new projects start with a default network (an auto mode VPC network) that has one subnetwork (subnet) in each region. Each VPC network consists of one or more IP address ranges called subnets. Subnets are regional resources, and have IP address ranges associated with them.
Use the
gcloud compute networks subnets create
command to create a proxy-only subnet in the default network.gcloud compute networks subnets create proxy-only-subnet \ --purpose=REGIONAL_MANAGED_PROXY \ --role=ACTIVE \ --region=us-central1 \ --network=default \ --range=10.10.10.0/24
Note that a subnet with
purpose=REGIONAL_MANAGED_PROXY
is reserved for Envoy-based load balancers and that therange
must provide 64 or more IP addresses.Create a firewall rule that matches the proxy-only subnet's range and that allows traffic on TCP port 8080.
gcloud compute firewall-rules create allow-proxy-connection \ --allow tcp:8080 \ --source-ranges 10.10.10.0/24 \ --network=default
Create a private GKE cluster
Use the gcloud container clusters create-auto
command
to create a private GKE cluster in
Autopilot mode
that has private nodes, and that has no client access to the public endpoint.
The following example creates a private GKE cluster named
private-cluster
and also creates a subnet named my-subnet
:
gcloud container clusters create-auto private-cluster \
--create-subnetwork name=my-subnet \
--enable-master-authorized-networks \
--enable-private-nodes \
--enable-private-endpoint \
--region=us-central1
Note the following:
--enable-master-authorized-networks
specifies that access to the public endpoint is restricted to IP address ranges that you authorize.--enable-private-nodes
indicates that the cluster's nodes don't have external IP addresses.--enable-private-endpoint
indicates that the cluster is managed using the internal IP address of the control plane API endpoint.
It might take several minutes for the creation of the cluster to complete. Once
the cluster is created, the output should indicate that the status of the cluster
is RUNNING
.
Create a VM instance in a specified subnet
A Compute Engine VM instance is a virtual machine that is hosted on Google's infrastructure. The terms Compute Engine instance, VM instance, and VM are synonymous and are used interchangeably. VM instances include GKE clusters, App Engine flexible environment instances, and other Google Cloud products built on Compute Engine VMs.
Use the gcloud compute instances create
command
to create a Compute Engine VM instance in the subnet you created
previously. Attach a service account and set the VM's access scope to
cloud-platform
.
gcloud compute instances create my-vm \
--service-account=PROJECT_NUMBER-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--zone=us-central1-a \
--subnet=my-subnet
For more information, see Create and start a VM instance.
Deploy an event receiver on the VM
Using a prebuilt image, us-docker.pkg.dev/cloudrun/container/hello
, deploy a
service on your VM that listens on port 80, and that receives and logs events.
Establish an SSH connection to your VM instance by running the following command:
gcloud compute ssh my-vm --project=PROJECT_ID --zone=us-central1-a
After a connection to the SSH server is established, run the remaining commands on your VM instance.
If necessary, install
kubectl
and any required plugins.From your VM instance, use the
get-credentials
command to enablekubectl
to work with the cluster you created.gcloud container clusters get-credentials private-cluster \ --region=us-central1 \ --internal-ip
Use a Kubernetes command,
kubectl create deployment
, to deploy an application to the cluster.kubectl create deployment hello-app \ --image=us-docker.pkg.dev/cloudrun/container/hello
This creates a Deployment named
hello-app
. The Deployment's Pod runs thehello
container image.After deploying the application, you can expose your application to traffic by creating a Kubernetes Service. Run the following
kubectl expose
command:kubectl expose deployment hello-app \ --type ClusterIP \ --port 80 \ --target-port 8080
You should see
service/hello-app exposed
in the output.You can ignore any messages similar to the following:
E0418 14:15:33.970933 1129 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Configure Kubernetes traffic routing
A Gateway resource represents a data plane that routes traffic in Kubernetes. A
Gateway can represent many different kinds of load balancing and routing
depending on the GatewayClass it is derived from. For more information, see
Deploying Gateways.
An HTTPRoute
manifest is deployed to create Routes and send traffic to
application backends.
Deploy a Gateway in your cluster.
kubectl apply -f - <<EOF kind: Gateway apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: internal-http spec: gatewayClassName: gke-l7-rilb listeners: - name: http protocol: HTTP port: 80 EOF
Note the following:
gatewayClassName: gke-l7-rilb
specifies the GatewayClass that this Gateway is derived from.gke-l7-rilb
corresponds to the internal Application Load Balancer.port: 80
specifies that the Gateway exposes only port 80 for listening for HTTP traffic.
Validate that the Gateway has deployed correctly. It might take a few minutes for it to deploy all of its resources.
kubectl describe gateways.gateway.networking.k8s.io internal-http
The output is similar to the following:
Name: internal-http Namespace: default ... API Version: gateway.networking.k8s.io/v1beta1 Kind: Gateway ... Spec: Gateway Class Name: gke-l7-rilb Listeners: Allowed Routes: Namespaces: From: Same Name: http Port: 80 Protocol: HTTP Status: Addresses: Type: IPAddress Value: 10.36.172.5 ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 80s sc-gateway-controller default/internal-http Normal UPDATE 20s (x3 over 80s) sc-gateway-controller default/internal-http Normal SYNC 20s sc-gateway-controller SYNC on default/internal-http was a success
Deploy an
HTTPRoute
manifest to route HTTP traffic to thehello-app
service at port 80.kubectl apply -f - <<EOF kind: HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: hello-app-route spec: parentRefs: - kind: Gateway name: internal-http rules: - backendRefs: - name: hello-app port: 80 EOF
Create a network attachment
A network attachment is a resource that lets a producer VPC network initiate connections to a consumer VPC network through a Private Service Connect interface.
To publish events, Eventarc uses the network attachment to establish a connection to the internal HTTP endpoint hosted in a VPC network.
You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the HTTP destination service.
gcloud compute network-attachments create my-network-attachment \ --region=us-central1 \ --subnets=my-subnet\ --connection-preference=ACCEPT_AUTOMATIC
For more information, see About network attachments.
Create an Eventarc trigger
Create an Eventarc trigger that creates a new Pub/Sub topic and routes events to the event receiver deployed on the VM when a message is published to the Pub/Sub topic.
Retrieve the Gateway address.
GATEWAY_ADDRESS=$(kubectl get gateways.gateway.networking.k8s.io internal-http -o=jsonpath="{.status.addresses[0].value}")
Create a trigger.
gcloud eventarc triggers create my-trigger \ --location=us-central1 \ --destination-http-endpoint-uri="http://$GATEWAY_ADDRESS:80/" \ --network-attachment="projects/PROJECT_ID/regions/us-central1/networkAttachments/my-network-attachment" \ --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \ --service-account=PROJECT_NUMBER-compute@developer.gserviceaccount.com
Replace
PROJECT_NUMBER
with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:gcloud projects describe PROJECT_ID --format='value(projectNumber)'
For more information about configuring your trigger, see Route events to an internal HTTP endpoint in a VPC network.
Generate and view a Pub/Sub topic event
You can generate an event by publishing a message to a Pub/Sub topic.
Find and set the Pub/Sub topic as an environment variable.
export MY_TOPIC=$(gcloud eventarc triggers describe my-trigger \ --location=us-central1 \ --format='value(transport.pubsub.topic)')
Publish a message to the Pub/Sub topic to generate an event.
gcloud pubsub topics publish $MY_TOPIC --message "Hello World"
The Eventarc trigger routes the event to the internal HTTP endpoint in the private GKE cluster.
Check the application Pod logs and verify the event delivery.
POD_NAME=$(kubectl get pod --selector app=hello-app --output=name) kubectl logs $POD_NAME
The body of the event should be similar to the following:
2024/04/18 20:31:43 Hello from Cloud Run! The container started successfully and is listening for HTTP requests on $PORT {"severity":"INFO","eventType":"google.cloud.pubsub.topic.v1.messagePublished","message":"Received event of type google.cloud.pubsub.topic.v1.messagePublished. Event data: Hello World","event":{"specversion":"1.0","id":"10935738681111260","source":"//pubsub.googleapis.com/projects/my-project/topics/eventarc-us-central1-my-trigger-224","type":"google.cloud.pubsub.topic.v1.messagePublished","datacontenttype":"application/json","time":"2024-04-18T20:40:03Z","data": {"message":{"data":"SGVsbG8gV29ybGQ=","messageId":"10935738681111260","publishTime":"2024-04-18T20:40:03Z"}}}}
You have successfully deployed an event receiver service to an internal HTTP endpoint in a private GKE cluster, created an Eventarc trigger, generated an event from Pub/Sub, and confirmed that the event was routed as expected by the trigger to the target endpoint.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Delete the project
Delete a Google Cloud project:
gcloud projects delete PROJECT_ID
Delete individual resources
- Delete the Eventarc trigger:
gcloud eventarc triggers delete my-trigger --location=us-central1
- Exit the VM and then delete the VM instance:
gcloud compute instances delete my-vm --zone=us-central1-a
- Delete the network attachment:
gcloud compute network-attachments delete my-network-attachment --region=us-central1
- Delete the firewall rule:
gcloud compute firewall-rules delete allow-proxy-connection
- Delete the cluster:
gcloud container clusters delete private-cluster --region=us-central1
- Delete the subnet:
gcloud compute networks subnets delete proxy-only-subnet --region=us-central1