Set up multi-network support for Pods

gcloud

gcloud compute networks subnets create SUBNET_NAME \
    --project=PROJECT_ID \
    --range=SUBNET_RANGE \
    --network=NETWORK_NAME \
    --region=REGION \
    --secondary-range=SECONDARY_RANGE_NAME=<SECONDARY_RANGE_RANGE>

Replace the following:

• SUBNET_NAME: the name of the subnet.
• PROJECT_ID: the ID of the project that contains the VPC network where the subnet is created.
• SUBNET_RANGE: the primary IPv4 address range for the new subnet, in CIDR notation.
• NETWORK_NAME: the name of the VPC network that contains the new subnet.
• REGION: the Google Cloud region in which the new subnet is created.
• SECONDARY_RANGE_NAME: the name for the secondary range.
• SECONDARY_IP_RANGE the secondary IPv4 address range in CIDR notation.

Console

1. In the Google Cloud console, go to the VPC networks page.

2. Click Create VPC network.

3. In the Name field, enter the name of the network. For example, l3-vpc.

4. From the Maximum transmission unit (MTU) drop-down, choose appropriate MTU value.

5. In the Subnet creation mode section, choose Custom.

6. Click ADD SUBNET.

7. In the New subnet section, specify the following configuration parameters for a subnet:

   1. Provide a Name. For example, l3-subnet.

   2. Select a Region.

   3. Enter an IP address range. This is the primary IPv4 range for the subnet.

      If you select a range that is not an RFC 1918 address, confirm that the range doesn't conflict with an existing configuration. For more information, see IPv4 subnet ranges.

   4. To define a secondary range for the subnet, click Create secondary IP address range.

      If you select a range that is not an RFC 1918 address, confirm that the range doesn't conflict with an existing configuration. For more information, see IPv4 subnet ranges.

   5. Private Google access: You can enable Private Google Access for the subnet when you create it or later by editing it.

   6. Flow logs: You can enable VPC flow logs for the subnet when you create it or later by editing it.

   7. Click Done.

8. In the Firewall rules section, under IPv4 firewall rules, select zero or more predefined firewall rules.

   The rules address common use cases for connectivity to instances. You can create your own firewall rules after you create the network. Each predefined rule name starts with the name of the VPC network that you are creating.

9. Under IPv4 firewall rules, to edit the predefined ingress firewall rule named allow-custom, click EDIT.

   You can edit subnets, add additional IPv4 ranges, and specify protocols and ports.

   The allow-custom firewall rule is not automatically updated if you add additional subnets later. If you need firewall rules for the new subnets, to add the rules, you must update the firewall configuration.

10. In the Dynamic routing mode section, for the VPC network. For more information, see dynamic routing mode. You can change the dynamic routing mode later.

11. Click Create.

gcloud

gcloud compute networks subnets create SUBNET_NAME \
    --project=PROJECT_ID \
    --range=SUBNET_RANGE \
    --network=NETWORK_NAME \
    --region=REGION \
    --secondary-range=SECONDARY_RANGE_NAME=<SECONDARY_RANGE_RANGE>

Replace the following:

• SUBNET_NAME: the name of the subnet.
• PROJECT_ID: the ID of the project that contains the VPC network where the subnet is created.
• SUBNET_RANGE: the primary IPv4 address range for the new subnet, in CIDR notation.
• NETWORK_NAME: the name of the VPC network that contains the new subnet.
• REGION: the Google Cloud region in which the new subnet is created.
• SECONDARY_RANGE_NAME: the name for the secondary range.
• SECONDARY_IP_RANGE the secondary IPv4 address range in CIDR notation.

Console

1. In the Google Cloud console, go to the VPC networks page.

2. Click Create VPC network.

3. In the Name field, enter the name of the network. For example, netdevice-vpc or dpdk-vpc.

4. From the Maximum transmission unit (MTU) drop-down, choose appropriate MTU value.

5. In the Subnet creation mode section, choose Custom.

6. In the New subnet section, specify the following configuration parameters for a subnet:

   1. Provide a Name. For example, netdevice-subnet or dpdk-vpc.

   2. Select a Region.

   3. Enter an IP address range. This is the primary IPv4 range for the subnet.

      If you select a range that is not an RFC 1918 address, confirm that the range doesn't conflict with an existing configuration. For more information, see IPv4 subnet ranges.

   4. Private Google Access: Choose whether to enable Private Google Access for the subnet when you create it or later by editing it.

   5. Flow logs: You can enable VPC flow logs for the subnet when you create it or later by editing it.

   6. Click Done.

7. In the Firewall rules section, under IPv4 firewall rules, select zero or more predefined firewall rules.

   The rules address common use cases for connectivity to instances. You can create your own firewall rules after you create the network. Each predefined rule name starts with the name of the VPC network that you are creating.

8. Under IPv4 firewall rules, to edit the predefined ingress firewall rule named allow-custom, click EDIT.

   You can edit subnets, add additional IPv4 ranges, and specify protocols and ports.

   The allow-custom firewall rule is not automatically updated if you add additional subnets later. If you need firewall rules for the new subnets, to add the rules, you must update the firewall configuration.

9. In the Dynamic routing mode section, for the VPC network. For more information, see dynamic routing mode. You can change the dynamic routing mode later.

10. Click Create.

gcloud

Create a GKE Standard cluster with multi-network capabilities:

gcloud container clusters create CLUSTER_NAME \
    --cluster-version=CLUSTER_VERSION \
    --enable-dataplane-v2 \
    --enable-ip-alias \
    --enable-multi-networking

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• CLUSTER_VERSION: the version of the cluster.

This command includes the following flags:

• --enable-multi-networking: enables multi-networking Custom Resource Definitions (CRDs) in the API server for this cluster, and deploys a network-controller-manager which contains the reconciliation and lifecycle management for multi-network objects.
• --enable-dataplane-v2: enables GKE Dataplane V2. This flag is required to enable multi-network.

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Click add_box Create.

3. Configure your Standard cluster. For more information, see Create a zonal cluster or Create a regional cluster. While creating the cluster, select the appropriate Network and Node subnet.

4. From the navigation pane, under Cluster, click Networking.

5. Select Enable Dataplane V2 checkbox.

6. Select Enable Multi-Network.

7. Click Create.

gcloud

gcloud container node-pools create POOL_NAME \
  --cluster=CLUSTER_NAME \
  --additional-node-network network=NETWORK_NAME,subnetwork=SUBNET_NAME \
  --additional-pod-network subnetwork=subnet-dp,pod-ipv4-range=POD_IP_RANGE,max-pods-per-node=NUMBER_OF_PODS \
  --additional-node-network network=highperformance,subnetwork=subnet-highperf

Replace the following:

• POOL_NAME with the name of the new node pool.
• CLUSTER_NAME with the name of the existing cluster to which you are adding the node pool.
• NETWORK_NAME with the name of the network to attach the node pool's nodes.
• SUBNET_NAME with the name of the subnet within the network to use for the nodes.
• POD_IP_RANGE the Pod IP address range within the subnet.
• NUMBER_OF_PODS maximum number of Pods per node.

This command contains the following flags:

• --additional-node-network: Defines details of the additional network interface, network, and subnetwork. This is used to specify the node-networks for connecting to the node pool nodes. Specify this parameter when you want to connect to another VPC. If you don't specify this parameter, the default VPC associated with the cluster is used. For Layer 3 type networks, specify the additional-pod-network flag that defines the Pod-network, which is exposed inside the GKE cluster as the Network object. When using the --additional-node-network flag, you must provide a network and subnetwork as mandatory parameters. Make sure to separate the network and subnetwork values with a comma and avoid using spaces.
• --additional-pod-network: Specifies the details of the secondary range to be used for the Pod-network. This parameter is not required if you use a Device type network. This argument specifies the following key values: subnetwork, pod-ipv4-range, and max-pods-per-node. When using the --additional-pod-network, you must provide the pod-ipv4-range and max-pods-per-node values, separated by commas and without spaces.
  • subnetwork: links the node-network with Pod-network. The subnetwork is optional. If you don't specify it, the additional Pod-network is associated with the default subnetwork provided during cluster creation.
  • --max-pods-per-node: The max-pods-per-node must be specified and has to be a power of 2. The minimum value is 4. The max-pods-per-node must not be more than the max-pods-per-node value on the node pool.

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. From the navigation pane, click Clusters.

3. In the Kubernetes clusters section, click the cluster you created.

4. At the top of the page, to create your node pool, click add_box Add Node Pool.

5. In the Node pool details section, complete the following:

   1. Enter a Name for the node pool.
   2. Enter the Number of nodes to create in the node pool.

6. From the navigation pane, under Node Pools, click Nodes.

   1. From the Image type drop-down list, select the Container-Optimized OS with containerd (cos_containerd) node image.

7. When you create a VM, you select a machine type from a machine family that determines the resources available to that VM. For example, a machine type like e2-standard-4 contains 4 vCPUs, therefore can support up to 4 VPCs total. There are several machine families you can choose from and each machine family is further organized into machine series and predefined or custom machine types within each series. Each machine type is billed differently. For more information, refer to the machine type price sheet.

8. From the navigation pane, select Networking.

9. In the section Node Networking, specify the maximum number of Pods per node. The Node Networks section displays the VPC network utilized to create the cluster. It is necessary to designate extra Node Networks that correlate with previously established VPC Networks and Device types.

10. Create node pool association:

    1. For Layer 3 type device:
       1. In the Node Networks section, click ADD A NODE NETWORK.
       2. From the network drop-down list select the VPC that supports Layer 3 type device.
       3. Select the subnet created for Layer 3 VPC.
       4. In the section Alias Pod IP address ranges, click Add Pod IP address range.
       5. Select the Secondary subnet and indicate the Max number of Pods per node.
       6. Select Done.
    2. For Netdevice and DPDK type device:
       1. In the Node Networks section, click ADD A NODE NETWORK.
       2. From the network drop-down list select the VPC that supports Netdevice or DPDK type devices.
       3. Select the subnet created for Netdevice or DPDK VPC.
       4. Select Done.

11. Click Create.

YAML

For the Layer 3 VPC, create Network and GKENetworkParamSet objects:

1. Save the following sample manifest as blue-network.yaml:

   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: blue-network
   spec:
     type: "L3"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: "l3-vpc"
   

   The manifest defines a Network resource named blue-network of the type Layer 3. The Network object references the GKENetworkParamSet object called l3-vpc, which associates a network with Compute Engine resources.

2. Apply the manifest to the cluster:

   kubectl apply -f blue-network.yaml
   

3. Save the following manifest as dataplane.yaml :

   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: "dataplane"
   spec:
     vpc: "l3-vpc"
     vpcSubnet: "subnet-dp"
     podIPv4Ranges:
       rangeNames:
       - "sec-range-blue"
   

   This manifest defines the GKENetworkParamSet object named dataplane, sets the VPC name as l3-vpc, subnet name as subnet-dp, and secondary IPv4 address range for Pods called sec-range-blue.

4. Apply the manifest to the cluster:

   kubectl apply -f dataplane.yaml
   

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. From the navigation pane, click Network Function Optimizer.

3. At the top of the page, click add_box Create to create your Pod network.

4. In the Before you begin section, verify the details.

5. Click NEXT: POD NETWORK LOCATION.

6. In the Pod network location section, from the Cluster drop-down, select the GKE cluster that has multi-networking and GKE Dataplane V2 enabled.

7. Click NEXT: VPC NETWORK REFERENCE.

8. In the VPC network reference section, from the VPC network reference drop-down, select the VPC network used for Layer 3 multinic Pods.

9. Click NEXT: POD NETWORK TYPE.

10. In the Pod network type section, select L3 and enter the Pod network name.

11. Click NEXT: POD NETWORK SECONDARY RANGE.

12. In the Pod network secondary range section, enter the Secondary range.

13. Click NEXT: POD NETWORK ROUTES.

14. In the Pod network routes section, to define Custom routes, select ADD ROUTE.

15. Click CREATE POD NETWORK.

YAML

For DPDK VPC, create Network and GKENetworkParamSet objects.

1. Save the following sample manifest as dpdk-network.yaml:

   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: dpdk-network
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: "dpdk"
   

   This manifest defines a Network resource named dpdk-network with a type of Device. The Network resource references a GKENetworkParamSet object called dpdk for its configuration.

2. Apply the manifest to the cluster:

   kubectl apply -f dpdk-network.yaml
   

3. For the GKENetworkParamSet object, save the following manifest as dpdk.yaml:

   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: "dpdk"
   spec:
     vpc: "dpdk"
     vpcSubnet: "subnet-dpdk"
     deviceMode: "DPDK-VFIO"
   

   This manifest defines the GKENetworkParamSet object named dpdk, sets the VPC name as dpdk, subnet name as subnet-dpdk, and deviceMode name as DPDK-VFIO.

4. Apply the manifest to the cluster:

   kubectl apply -f dpdk-network.yaml
   

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. From the navigation pane, click Network Function Optimizer.

3. At the top of the page, click add_box Create to create your Pod network.

4. In the Before you begin section, verify the details.

5. Click NEXT: POD NETWORK LOCATION.

6. In the Pod network location section, from the Cluster drop-down, select the GKE cluster that has multi-networking and GKE Dataplane V2 enabled.

7. Click NEXT: VPC NETWORK REFERENCE.

8. In the VPC network reference section, from the VPC network reference drop-down, select the VPC network used for dpdk multinic Pods.

9. Click NEXT: POD NETWORK TYPE.

10. In the Pod network type section, select DPDK-VFIO (Device) and enter the Pod network name.

11. Click NEXT: POD NETWORK SECONDARY RANGE. The Pod network secondary range section will be unavailable

12. Click NEXT: POD NETWORK ROUTES. In the Pod network routes section, select ADD ROUTE to define custom routes

13. Click CREATE POD NETWORK.

YAML

1. Save the following sample manifest as netdevice-network.yaml:

   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
       name: netdevice-network
   spec:
       type: "Device"
       parametersRef:
         group: networking.gke.io
         kind: GKENetworkParamSet
         name: "netdevice"
   

   This manifest defines a Network resource named netdevice-network with a type of Device. It references the GKENetworkParamSet object named netdevice.

2. Apply the manifest to the cluster:

   kubectl apply -f netdevice-network.yaml
   

3. Save the following manifest as netdevice.yaml :

   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: netdevice
   spec:
     vpc: netdevice
     vpcSubnet: subnet-netdevice
     deviceMode: NetDevice
   

   This manifest defines a GKENetworkParamSet resource named netdevice, sets the VPC name as netdevice, the subnet name as subnet-netdevice, and specifies the device mode as NetDevice.

4. Apply the manifest to the cluster:

   kubectl apply -f netdevice.yaml
   

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. From the navigation pane, click Network Function Optimizer.

3. At the top of the page, click add_box Create to create your Pod network.

4. In the Before you begin section, verify the details.

5. Click NEXT: POD NETWORK LOCATION.

6. In the Pod network location section, from the Cluster drop-down, select the GKE cluster that has multi-networking and GKE Dataplane V2 enabled.

7. Click NEXT: VPC NETWORK REFERENCE.

8. In the VPC network reference section, from the VPC network reference drop-down, select the VPC network used for netdevice multinic Pods.

9. Click NEXT: POD NETWORK TYPE.

10. In the Pod network type section, select NetDevice (Device) and enter the Pod network name.

11. Click NEXT: POD NETWORK SECONDARY RANGE. The Pod network secondary range section will be unavailable

12. Click NEXT: POD NETWORK ROUTES. In the Pod network routes section, to define custom routes, select ADD ROUTE.

13. Click CREATE POD NETWORK.

YAML

1. Save the following manifest as red-network.yaml:

   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: red-network
   spec:
     type: "L3"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: "management"
     routes:
     -   to: "10.0.2.0/28"
   

   This manifest defines a Network resource named red-network with a type of Layer 3 and custom route "10.0.2.0/28" through that Network interface.

2. Apply the manifest to the cluster:

   kubectl apply -f red-network.yaml
   

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Click add_box Create.

3. From the navigation pane, click Network Function Optimizer.

4. In the Kubernetes clusters section, click the cluster you created.

5. At the top of the page, click add_box Create to create your Pod network.

6. In the Pod network routes section, define Custom routes.

7. Click CREATE POD NETWORK.