If the machines you are using for bootstrap and cluster nodes use a proxy server to access the internet, you must:
- Configure proxying for the package manager on cluster nodes
- Configure proxy details in the cluster configuration file.
Prerequisites
Your proxy server must allow connections to the following addresses:
Address | Purpose |
---|---|
*.gcr.io |
Pull images from the Container Registry. |
accounts.google.com |
Process authorization requests for OpenID and discover public keys for verifying tokens. |
cloudresourcemanager.googleapis.com |
Resolve metadata regarding the Google Cloud project the cluster is being connected to. |
compute.googleapis.com |
Verify Cloud Logging and Cloud Monitoring resource region. |
dl.fedoraproject.org |
Install Extra Packages for Enterprise Linux (EPEL) when using Red Hat Enterprise Linux (RHEL) distributions. |
download.docker.com |
Add Docker repository. This is required if you run your admin workstation behind proxy. It is required for node machines that run behind a proxy when Docker is used for the container runtime. |
gkeconnect.googleapis.com |
Establish the channel used to receive requests from Google Cloud and issues responses. |
gkehub.googleapis.com |
Create Google Cloud-side Hub membership resources that correspond to the cluster you're connecting with Google Cloud. |
iam.googleapis.com |
Create service accounts, which you can use to authenticate to Google Cloud and make API calls. |
iamcredentials.googleapis.com |
Provides admission control and telemetry reporting for audit logging. |
logging.googleapis.com |
Write log entries and manage your Cloud Logging configuration. |
monitoring.googleapis.com |
Manage your Cloud Monitoring data and configurations. |
packages.cloud.google.com |
Download packages from the Google Cloud package mirror. |
oauth2.googleapis.com |
Authenticate through OAuth token exchange for account access. |
opsconfigmonitoring.googleapis.com |
Collect metadata for Kubernetes resources such as pods, deployments, or nodes to enrich metric queries. |
securetoken.googleapis.com |
Retrieve refresh tokens for workload identity authorization. |
servicecontrol.googleapis.com |
Write audit log entries into Cloud Audit Logs. |
serviceusage.googleapis.com |
Enable and validate services and APIs. |
stackdriver.googleapis.com |
Manage Google Cloud Observability metadata, such as Stackdriver accounts. |
storage.googleapis.com |
Manage object storage and buckets, such as Container Registry objects. |
sts.googleapis.com |
Exchange Google or third-party credentials for a short-lived access token to Google Cloud resources |
www.googleapis.com |
Authenticate service tokens from incoming Google Cloud service requests. |
In addition to these URLs, the proxy server must also allow any package mirrors your operating system's package manager requires. You can update the package manager configuration to use a more deterministic list, which is easier to manage.
Configure proxying for the package manager on cluster nodes
Google Distributed Cloud uses the APT package manager on Ubuntu and the DNF package manager on CentOS and Red Hat Linux. Ensure that the OS package manager has the correct proxy configuration.
Refer to your OS distribution's documentation for details about configuring the proxy. The following examples show one way to configure proxy settings:
APT
These commands demonstrate how to configure the proxy for APT:
sudo touch /etc/apt/apt.conf.d/proxy.conf
echo 'Acquire::http::Proxy "http://[username:password@]domain";' >> /etc/apt/apt.conf.d/proxy.conf
echo 'Acquire::https::Proxy "http://[username:password@]domain";' >> /etc/apt/apt.conf.d/proxy.conf
Replace [username:password@]domain with details specific to your configuration.
DNF
This command demonstrates how to configure the proxy for DNF:
echo "proxy=http://[username:password@]domain" >> /etc/dnf/dnf.conf
Replace [username:password@]domain with details specific to your configuration.
Configure proxy details in the cluster configuration file
In the cluster configuration file, set the following values to configure the cluster to use the proxy:
proxy.url
A string that specifies the proxy URL. The bootstrap and node machines use this proxy to access the internet.
proxy.noProxy
A list of IP addresses, hostnames, and domain names that should not go through the proxy server.
In most cases, you don't need to add any items to this list. Please don't add Service and Pod CIDR.
noProxy use cases:
Using a private package mirror, which located in the same private network (Don't need proxy to access)
Using a private registry mirror, which located in the same private network (Don't need proxy to access)
Example
The following is an example of the proxy settings in a cluster configuration file:
proxy:
url: http://[username:password@]domain
noProxy:
- example1.com
- example2.com
Override the proxy configuration
You can run your bootstrap machine behind a different proxy than the one used by your node machines by overriding the proxy settings in the cluster configuration file. To override the proxy settings, set the following environment variables on the bootstrap machine:
export HTTPS_PROXY=http://[username:password@]domain
Replace [username:password@]domain with details specific to your configuration.
export NO_PROXY=example1.com,example2.com
Replace example1.com,example2.com with IP addresses, hostnames, and domain names that should not go through the proxy server.
Side effects
When run as root, bmctl
updates the Docker proxy configuration on the
bootstrap machine. If you do not run bmctl
as root, configure the Docker proxy
manually.
Firewall rules
Set up your firewall rules as outlined in the following sections to allow the described traffic needed for Google Distributed Cloud.
For prerequisite port requirements for Google Distributed Cloud, see Port usage.
Firewall rules for cluster node IP addresses
The following table describes the firewall rules for IP addresses available in your clusters.
From |
Source port |
To |
Port |
Protocol |
Description |
---|---|---|---|---|---|
Cluster node | 1024 - 65535 | cloudresourcemanager.googleapis.com gkeconnect.googleapis.com gkehub.googleapis.com |
443 | TCP/HTTPS | Access is required for hub registration. |
Cloud Logging Collector, which runs on cluster node | 1024 - 65535 | oauth2.googleapis.com logging.googleapis.com stackdriver.googleapis.com servicecontrol.googleapis.com storage.googleapis.com www.googleapis.com |
443 | TCP/HTTPS | |
Cloud Metadata Collector, which runs on cluster node | 1024 - 65535 | opsconfigmonitoring.googleapis.com |
443 | TCP/HTTPS | |
Cloud Monitoring Collector, which runs on cluster node | 1024 - 65535 | oauth2.googleapis.com monitoring.googleapis.com stackdriver.googleapis.com servicecontrol.googleapis.com |
443 | TCP/HTTPS | |
Cluster node | 1024 - 65535 | On-premises local Docker registry | Depends on your registry | TCP/HTTPS | Required if Google Distributed Cloud is configured to use a local private
Docker registry instead of gcr.io . |
Cluster node | 1024 - 65535 | gcr.io oauth2.googleapis.com storage.googleapis.com Any Google API URL of the form *.googleapis.com required for the
services enabled for the admin cluster. |
443 | TCP/HTTPS | Download images from public Docker registries. Not required if using a private Docker registry. |
Connect agent, which runs on a cluster node. | 1024 - 65535 | cloudresourcemanager.googleapis.com gkeconnect.googleapis.com gkehub.googleapis.com www.googleapis.com iam.googleapis.com iamcredentials.googleapis.com oauth2.googleapis.com securetoken.googleapis.com sts.googleapis.com accounts.google.com |
443 | TCP/HTTPS | [Connect traffic](/anthos/multicluster-management/connect). |
Firewall rules for remaining components
The rules described in the following table apply to all other components not listed in the preceding section.
From |
Source port |
To |
Port |
Protocol |
Description |
---|---|---|---|---|---|
Clients and application end users | All | VIP of Istio ingress | 80, 443 | TCP | End user traffic to the ingress service of a user cluster. |
Admin workstation | 32768 - 60999 | gcr.io cloudresourcemanager.googleapis.com oauth2.googleapis.com storage.googleapis.com Any *.googleapis.com URL required for the services enabled
for this cluster |
443 | TCP/HTTPS | Download Docker images from public Docker registries. |
Admin workstation | 32768 - 60999 | gcr.io cloudresourcemanager.googleapis.com iam.googleapis.com oauth2.googleapis.com serviceusage.googleapis.com storage.googleapis.com Any *.googleapis.com URL required for the services enabled
for the admin or user clusters |
443 | TCP/HTTPS | Preflight checks (validation). |