Understand GENEVE format

Generic Network Virtualization Encapsulation (GENEVE) is a network virtualization tunneling protocol that keeps the original packet intact and encapsulates it with additional metadata. This additional metadata helps to enable flexible and extensible network virtualization.

The following diagram shows the header format of the GENEVE encapsulation.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Ver|  Opt len  |O|C|   Rsvd.   |         Protocol type         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |        Virtual network identifier (VNI)       |     Rsvd.     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The following table describes the GENEVE encapsulation header format fields that are shown in the preceding diagram:

Field Description Field length
Ver The version of the GENEVE protocol. The only supported version is 0. For more information, see Tunnel Header Fields 2-bits
Opt len The length of the option fields, expressed in 4-byte multiples. For more information, see Tunnel Header Fields 6-bits
O The control packet bit. For more information, see Tunnel Header Fields 1-bit
C Critical options bit. For more information, see Tunnel Header Fields 1-bit
Rsvd Reserved field, which must be zero on transmission and must be ignored on receipt. For more information, see Tunnel Header Fields 6-bits
Protocol type The protocol type allows for any ethertype, however, Network Security Integration only allows IPv4 (0x0800) or IPv6 (0x86DD) 16-bits
Virtual Network Identifier (VNI) The VNI is a field within the GENEVE header that identifies a virtual network. Network Security Integration doesn't use VNI which means the VNI is set to zero (0). For more information, see Out-of-band integration GENEVE 24-bits
Rsvd Reserved field, which must be zero on transmission and must be ignored on receipt. For more information, see Tunnel Header Fields 8-bits

Google Cloud-specific GENEVE options

In this section learn about the GENEVE header and its specific applications within Google Cloud services. The GENEVE header uses a Type-Length-Value (TLV) format for its options. This means that each option is encoded with a type identifier, a length field indicating the size of the value, and the value itself. This format allows for flexibility and extensibility, as new options can be added without disrupting existing implementations. It's not possible to predict the order in which options appear within the GENEVE header. Therefore, it's important to parse the header based on the TLV format.

The GENEVE options specific to Google Cloud are as follows:

  • Network ID (network cookie)
  • Endpoint ID (endpoint cookie)
  • Profile ID

Network ID

A network ID or network cookie identifies the virtual network associated with GENEVE-encapsulated traffic in Google Cloud. A network cookie is a 32-bit value identified by [class=0x0132, type=01] in the GENEVE header. Of the 32-bits, the first 28 represent an opaque network ID. The remaining bits are reserved for other purposes, including Transport Layer Security (TLS) and direction.

The following diagram shows the GENEVE packet format for the network cookie option.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Option class=0x0132 (Google) |    Type=02    |R|R|R|  Len=1  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Network cookie                    |R|R|T|D|
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The following table describes the option header fields for the network cookie that's shown in the preceding diagram:

Field Description Field length
Option class Identifies the organization or entity that has defined the option. In this option, the Option class 0x0132 designates Google as the defining entity 16-bits
Type Identifies the type of the option. In this option, the Option type 0x1 designates the Network ID option. The Type=2 identifies the network ID or network cookie option. For more information, see Tunnel Options 8-bits
R The option control flags reserved for future use. These bits must be zero on transmission and must be ignored on receipt 3-bits
Len Specifies the length of the payload in 4-byte increments. A Network ID is 32-bits (4-bytes), so the Len field for this option is set to 1 5-bits

The following table describes the option payload fields for the network cookie that's shown in the diagram:

Field Description Field length
Network cookie The opaque network cookie 28-bits
R Reserved for future implementation. Must be set to zero (0) on transmission and must be ignored on receipt 2-bits
T Specifies the packet is TLS-encrypted and is not used 1-bit
D Denotes the direction of the original packet. Zero (0) signifies ingress and one (1) signifies an egress packet on the original mirrored virtual machine (VM) 1-bit

Endpoint ID

An endpoint ID or an endpoint cookie is used for tracking and managing traffic flows associated with specific endpoints within the Google Cloud environment. An endpoint cookie uniquely identifies a network interface controller of the mirrored VM. It's a 128-bit opaque value represented as [class=0x0132, type=02] in the GENEVE header.

The following diagram shows the GENEVE packet format for the endpoint cookie option.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Option class=0x0132 (Google) |     Type=02   |R|R|R|  Len=4  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                        Endpoint cookie                        +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The following table describes the option header fields for the endpoint cookie that's shown in the preceding diagram:

Field Description Field length
Option class Identifies the organization or entity that has defined the option. In this option, the Option Class 0x0132 designates Google as the defining entity 16-bits
Type Identifies the type of the option. In this option, the Option type 0x1 designates the Endpoint ID option. The Type=2 identifies the endpoint id or endpoint cookie option. For more information, see Tunnel Options 8-bits
R The option control flags reserved for future use. These bits must be zero on transmission and must be ignored on receipt 3-bits
Len Specifies the length of the payload in 4-byte increments. Endpoint ID is 128-bits, so the Len field for this option is set to 4 5-bits
Rsvd Reserved field, which must be zero on transmission and must be ignored on receipt 3-bits

The following table describes the option payload fields for the endpoint cookie that's shown in the diagram:

Field Description Field length
Endpoint cookie The opaque identifier of the mirrored VM 128-bits

Profile ID

A profile ID identifies the mirroring security profile group that is applied to the traffic. It is a 64-bit identifier represented as [class=0x0132, type=03] in the GENEVE header. A profile ID corresponds to the data_path_id field in the security profile group, enabling the enforcement of security policies based on the profile group configuration.

The following diagram shows GENEVE packet format for the profile ID option.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Option class=0x0132 (Google) |    Type=03    |R|R|R|  Len=2  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                           Profile ID                          +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The following table describes the option header fields for the profile ID that's shown in the preceding diagram:

Field Description Field length
Option class Identifies the organization or entity that has defined the option. In this option, the Option Class 0x0132 designates Google as the defining entity. 16-bits
Type Identifies the type of the option. In this option, the Option type 0x1 designates the profile ID option. The Type=3 identifies the profile id option. For more information, see Tunnel Options 8-bits
R The option control flags reserved for future use. These bits must be zero on transmission and must be ignored on receipt 3-bits
Len Specifies the length of the payload in 4-byte increments. Profile ID is 64-bits, so the Len field for this option is set to 2 5-bits

The following table describes the option payload fields for the network ID that's shown in the diagram:

Field Description Field length
Profile ID The identifier of the mirroring security profile group 64-bits

Out-of-band integration GENEVE

This section describes the GENEVE header formats that are utilized by Network Security Integration services, specifically out-of-band integration. Out-of-band integration uses a GENEVE tunnel to encapsulate and deliver mirrored packets. The packets are addressed to the collector's internal load balancer virtual IP address (VIP) and are annotated with Google Cloud-specific metadata such as network cookies.

The Google Cloud-specific GENEVE options used in out-of-band integration are network cookie, endpoint cookie, and profile ID. For more information, see Google Cloud-specific GENEVE options.

The following diagram shows how Google Cloud-specific GENEVE options are used in out-of-band integration.

       0                    1                   2                  3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |V=0| Opt len=7 |O|C|   Rsvd    |         Protocol type         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |        Virtual network identifier (VNI) = 0   |     Rsvd.     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Option class=0x0132 (Google) |    Type=01    |R|R|R|  Len=1  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Network cookie                    |R|R|T|D|
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Option class=0x0132 (Google) |    Type=02    |R|R|R|  Len=4  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                        Endpoint cookie                        +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Option class=0x0132 (Google) |    Type=03    |R|R|R|  Len=2  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                           Profile ID                          +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

GENEVE packet format over IPv4

The GENEVE packet format encapsulates a compact tunnel header in UDP over IPv4. A small fixed tunnel header provides control information and base-level capabilities and interoperability.

The following diagram shows the GENEVE packet format over IPv4.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Version|  IHL  |Type of service|          Total length         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         Identification        |Flags|     Fragment offset     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Time to live | Proto=17 (UDP)|        Header checksum        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         Source address                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Destination address                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The following table describes the IPv4 header fields that's shown in the preceding diagram:

Field Description
Proto=17 (always UDP for GENEVE) Indicates that the encapsulated payload uses the UDP protocol
Source address The local subnet's gateway IP address
Destination address The customer-owned load balancer's VIP

UDP header

The UDP RFC 0768 header encapsulates data, maintaining the connectionless semantics of ethernet and IP address. It also provides entropy to routers that perform equal cost multipath (ECMP).

The following diagram shows the header format for a Geneve packet encapsulated within a UDP.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Source port = <hash>     |   Dest port = 6081 (Geneve)   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP length          |          UDP checksum         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The following table describes the UDP header fields that's shown in the preceding diagram:

Fields Description
Source port An opaque hash over the entire 16-bit range. The value is the same for all packets belonging to a single encapsulated flow (both directions)
Dest port The designated destination port number for GENEVE traffic, set to 6081
UDP length The total length of the UDP datagram, including the UDP header and the encapsulated GENEVE packet
UDP checksum The checksum value for the UDP datagram, used for error detection

GENEVE encapsulation and MTU requirements

The maximum transmission unit (MTU) is the size, in bytes, of the largest possible IP packet that can fit inside an Ethernet frame, including IP headers, Layer 4 protocol headers, and Layer 4 data. For more information, see Maximum transmission unit.

In Google Cloud networks, the MTU allowed is 8896 bytes. However, for encapsulation overhead. This retention need has implications for both consumer and producer networks:

  • Consumer network MTU: The MTU for the consumer network must not be more then 8740 bytes. This limit ensures that there's enough space for the GENEVE overhead without exceeding the maximum MTU.

  • Producer network MTU: The MTU for the producer network must be at least 156 bytes more than the consumer network MTU. This allowance accommodates the additional GENEVE encapsulation.

If there aren't enough MTU limits to accommodate the GENEVE encapsulation, the packet can be dropped. This can happen either at the virtual network level or in the guest VM's operating system.

What's next