RDMA RoCE network profile

This page provides an overview of the Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) network profile in Google Cloud.

Overview

The RDMA RoCE network profile lets you create a Virtual Private Cloud (VPC) network that provides low-latency, high-bandwidth RDMA communication between the GPUs of VMs that are created in the network by using the RoCE v2 protocol. A VPC network that uses the RoCE network profile is called an RoCE VPC network.

RoCE VPC networks are useful for running AI workloads. For more information about running AI workloads in Google Cloud, see AI Hypercomputer overview.

The resource name of an RoCE network profile has the following format ZONE-vpc-roce—for example europe-west1-b-vpc-roce. To view specific network profile names, see List network profiles.

Supported zones

The RoCE network profile is available in a limited set of zones. You can only create an RoCE VPC network in a zone where the RoCE network profile is available.

To view the supported zones, see list network profiles.

Alternatively, you can view the supported zones for the GPU machine type that you intend to create in the RoCE VPC network. The RoCE network profile is available in the same zones as the supported machine types described in Specifications. For more information, see GPU availability regions and zones.

Specifications

RoCE VPC networks have the following specifications:

NVIDIA ConnectX NICs. NVIDIA ConnectX NICs appear as MRDMA network interfaces in Google Cloud.
Zonal constraint. Resources using an RoCE VPC network are limited to the same zone as the RoCE network profile associated with the RoCE VPC network during the RoCE network creation. This zonal limit has the following effects:
- All instances that have network interfaces in an RoCE VPC network must be created in the zone that matches the zone of the RoCE network profile used by the RoCE VPC network.
- All subnets created in an RoCE VPC network must be located in the region that contains the zone of the RoCE network profile used by the RoCE VPC network.
MRDMA network interfaces only. RoCE VPC networks only support MRDMA network interfaces (NICs), which are only available on the A3 Ultra, A4, and A4X machine series.

All non-MRDMA NICs of a virtual machine (VM) instance must be attached to a regular VPC network.
8896 byte MTU. For best performance, we recommend a maximum transmission unit (MTU) of 8896 bytes for RoCE VPC networks. This allows the RDMA driver in the VM's guest operating system to use smaller MTUs if needed.

If you create an RoCE VPC network by using the gcloud CLI or the API, then 8896 bytes is the default MTU. If you create an RoCE VPC network by using the Google Cloud console, then you must set the MTU to 8896.
Firewall differences. RoCE VPC networks use different implied firewall rules. They only support regional network firewall policies that have an RoCE firewall policy type. The set of parameters for rules within a supported regional network firewall policy are limited. For more information, see Cloud NGFW for RoCE VPC networks.
No VPC Flow Logs support. RoCE VPC networks don't support VPC Flow Logs, even if you enable VPC Flow Logs for a subnet in an RoCE VPC network.
No Connectivity Tests support. Connectivity Tests doesn't support RoCE VPC networks.
Other VPC features. RoCE VPC networks support a limited set of other VPC features. For more information, see the following Supported and unsupported features section.

Supported and unsupported features

The following table lists which VPC features are supported by RoCE VPC networks.

Feature	Network profile property	Network profile property value	Details
`MRDMA` NICs	`interfaceTypes`	`MRDMA`	RoCE VPC networks only support `MRDMA` NICs, not other types, such as `GVNIC` or `VIRTIO_NET`.
Multi-NIC in the same network	`allowMultiNicInSameNetwork`	`MULTI_NIC_IN_SAME_NETWORK_ALLOWED`	RoCE VPC networks support multi-NIC VMs, allowing two or more `MRDMA` VM NICs to be in the same RoCE VPC network. Each NIC must attach to a unique subnet in the RoCE VPC network. See RoCE VPC network multi-NIC considerations.
IPv4-only subnets	`subnetworkStackTypes`	`SUBNET_STACK_TYPE_IPV4_ONLY`	RoCE VPC networks support IPv4-only subnets, including the same Valid IPv4 ranges as regular VPC networks. RoCE VPC networks don't support dual-stack or IPv6-only subnets. For more information, see Types of subnets.
`PRIVATE` subnet purpose	`subnetworkPurposes`	`SUBNET_PURPOSE_PRIVATE`	RoCE VPC networks support regular subnets, which have a `purpose` attribute value of `PRIVATE`. RoCE VPC networks don't support Private Service Connect subnets, proxy-only subnets, or Private NAT subnets. For more information, see Purposes of subnets.
`GCE_ENDPOINT` address purpose	`addressPurposes`	`GCE_ENDPOINT`	RoCE VPC networks support IP addresses with a `purpose` attribute value of `GCE_ENDPOINT`, which is used by internal IP addresses of VM NICs. RoCE VPC networks don't support special purpose IP addresses, such as the `SHARED_LOADBALANCER_VIP` purpose. For more information, see the address resource reference.
Attachments from `nic0`	`allowDefaultNicAttachment`	`DEFAULT_NIC_ATTACHMENT_BLOCKED`	RoCE VPC networks don't support attaching the `nic0` network interfaces of a VM to the network. Each `MRDMA` NIC attached to an RoCE VPC network must not be `nic0`.
External IP addresses for VMs	`allowExternalIpAccess`	`EXTERNAL_IP_ACCESS_BLOCKED`	RoCE VPC networks don't support assigning external IP addresses to `MDRMA` VM NICs. Consequently, `MDRMA` VM NICs don't have internet access.
Dynamic Network Interfaces	`allowSubInterfaces`	`SUBINTERFACES_BLOCKED`	RoCE VPC networks don't support Dynamic NICs.
Alias IP ranges	`allowAliasIpRanges`	`ALIAS_IP_RANGE_BLOCKED`	RoCE VPC networks don't support assigning alias IP ranges to `MRDMA` NICs.
IP forwarding	`allowIpForwarding`	`IP_FORWARDING_BLOCKED`	RoCE VPC networks don't support IP forwarding.
VM network migration	`allowNetworkMigration`	`NETWORK_MIGRATION_BLOCKED`	RoCE VPC networks don't support migrating VM NICs between networks.
Auto mode	`allowAutoModeSubnet`	`AUTO_MODE_SUBNET_BLOCKED`	RoCE VPC networks can't be auto mode networks. For more information, see subnet creation mode.
VPC Network Peering	`allowVpcPeering`	`VPC_PEERING_BLOCKED`	RoCE VPC networks don't support connecting to other VPC networks using VPC Network Peering. Consequently, RoCE VPC networks don't support connecting to services using private services access.
Static routes	`allowStaticRoutes`	`STATIC_ROUTES_BLOCKED`	RoCE VPC networks don't support static routes.
Packet Mirroring	`allowPacketMirroring`	`PACKET_MIRRORING_BLOCKED`	RoCE VPC networks don't support Packet Mirroring.
Cloud NAT	`allowCloudNat`	`CLOUD_NAT_BLOCKED`	RoCE VPC networks don't support Cloud NAT.
Cloud Router	`allowCloudRouter`	`CLOUD_ROUTER_BLOCKED`	RoCE VPC networks don't support Cloud Routers and dynamic routes.
Cloud Interconnect	`allowInterconnect`	`INTERCONNECT_BLOCKED`	RoCE VPC networks don't support Cloud Interconnect VLAN attachments.
Cloud VPN	`allowVpn`	`VPN_BLOCKED`	RoCE VPC networks don't support Cloud VPN tunnels.
Network Connectivity Center	`allowNcc`	`NCC_BLOCKED`	RoCE VPC networks don't support Network Connectivity Center. You can't add an RoCE VPC network as a VPC spoke to a Network Connectivity Center hub.
Cloud Load Balancing	`allowLoadBalancing`	`LOAD_BALANCING_BLOCKED`	RoCE VPC networks don't support Cloud Load Balancing. Consequently, RoCE VPC networks don't support load balancer features, including Google Cloud Armor.
Private Google Access	`allowPrivateGoogleAccess`	`PRIVATE_GOOGLE_ACCESS_BLOCKED`	RoCE VPC networks don't support Private Google Access.
Private Service Connect	`allowPsc`	`PSC_BLOCKED`	RoCE VPC networks don't support Private Service Connect.

RoCE VPC network multi-NIC considerations

To support workloads that benefit from cross-rail GPU-to-GPU communication, RoCE VPC networks support VMs that have multiple MRDMA NICs in the network. Each MRDMA NIC must be in a unique subnet. Placing two or more MRDMA NICs in the same RoCE VPC network might affect network performance, including increased latency. MRDMA NICs use NCCL. NCCL attempts to align all network transfers, even for cross-rail communication. For example, it uses PXN to copy data through NVlink to a rail-aligned GPU before transferring it over the network.