Troubleshoot MACsec

This page describes how to troubleshoot MACsec for Cloud Interconnect.

Cloud Interconnect displays an error when I attempt to create a new key

If you have an existing MACsec key without a start time and you attempt to create a new key, Cloud Interconnect displays an error. To resolve the error, update the existing key's start time.

MACsec is operationally down on my Cloud Interconnect connection

You successfully enabled MACsec on your Cloud Interconnect connection and on your on-premises router, but the MACsec session displays that it is operationally down on your Cloud Interconnect connection links. The issue could be caused by one of the following:

The active keys on your on-premises router and Google's edge routers don't match.
A MACsec protocol mismatch exists between your on-premises router and Google's edge router.

To resolve the MACsec state, do the following:

To verify that MACsec is enabled on your Cloud Interconnect connection, select one of the following options:

Console

In the Google Cloud console, go to the Cloud Interconnect Physical connections tab.

Go to Physical connections
Select the Cloud Interconnect connection that you want to view.
On the MACsec tab, verify that MACsec configuration displays one of the following:
- Enabled, fail open: MACsec encryption is enabled on the link. If MACsec encryption isn't established between both ends, then the link operates without encryption.
- Enabled, fail closed: MACsec encryption is enabled on the link. If MACsec encryption isn't established between both ends, then the link fails.

gcloud

gcloud compute interconnects describe INTERCONNECT_CONNECTION_NAME

Replace INTERCONNECT_CONNECTION_NAME with the name of your Cloud Interconnect connection.

The output is similar to the following. Verify that macsecEnabled: true is displayed:

adminEnabled: true
availableFeatures:
- IF_MACSEC
circuitInfos:
- customerDemarcId: fake-peer-demarc-0
  googleCircuitId: LOOP-0
  googleDemarcId: fake-local-demarc-0
creationTimestamp: '2021-10-05T03:39:33.888-07:00'
customerName: Fake Company
description: something important
googleReferenceId: '123456789'
id: '12345678987654321'
interconnectAttachments:
- https://www.googleapis.com/compute/v1/projects/my-project1/regions/us-central1/interconnectAttachments/interconnect-123456-987654321-0
interconnectType: IT_PRIVATE
kind: compute#interconnect
labelFingerprint: 12H17262736_
linkType: LINK_TYPE_ETHERNET_10G_LR
location: https://www.googleapis.com/compute/v1/projects/my-project1/global/interconnectLocations/cbf-zone2-65012
macsec:
  failOpen: false
  preSharedKeys:
  - name: key1
    startTime: 2023-07-01T21:00:01.000Z
macsecEnabled: true
name: INTERCONNECT_CONNECTION_NAME
operationalStatus: OS_ACTIVE
provisionedLinkCount: 1
requestedFeatures:
- IF_MACSEC
requestedLinkCount: 1
selfLink: https://www.googleapis.com/compute/v1/projects/my-project1/global/interconnects/INTERCONNECT_CONNECTION_NAME
selfLinkWithId: https://www.googleapis.com/compute/v1/projects/my-project1/global/interconnects/12345678987654321
state: ACTIVE

To check the Cloud Interconnect port status, MACsec operational state, and the active key name, use one of the following options:
Console
1. In the Google Cloud console, go to the Cloud Interconnect Physical connections tab.
  
  Go to Physical connections
2. Select the Cloud Interconnect connection that you want to view.
3. In Link circuit info, verify that Link state displays Active for all links.
4. Verify that MACsec key name displays a key name for all links, and that each key name displays MACsec on this link is up or MACsec on this link is down.
gcloud
```
gcloud compute interconnects get-diagnostics INTERCONNECT_CONNECTION_NAME \
    --project=PROJECT_NAME
```
Replace PROJECT_NAME with the name of your Google Cloud project.

The output is similar to the following. Verify that links.lacpStatus.state displays ACTIVE, that links.macsec.ckn displays a value, and that links.operationalStatus displays LINK_OPERATIONAL_STATUS_UP:
```
bundleAggregationType: BUNDLE_AGGREGATION_TYPE_STATIC
bundleOperationalStatus: BUNDLE_OPERATIONAL_STATUS_UP
links:
- circuitId: LOOP-0
  googleDemarc: fake-local-demarc-0
  lacpStatus:
    googleSystemId: '00:11:22:33:44:55'
    neighborSystemId: '55:44:33:22:11:00'
    state: ACTIVE
  macsec:
    ckn: 0101010189abcdef...0123456789abcdef
    operational: false
  operationalStatus: LINK_OPERATIONAL_STATUS_UP
  receivingOpticalPower:
    state: OK
    value: -2.49
  transmittingOpticalPower:
    state: OK
    value: -0.88
macAddress: 00:11:22:33:44:55
```
If there isn't a value displayed for links.macsec.ckn, then contact Google Cloud Support for help.
To verify the active key's CAK and CKN values, and the key's start time, select one of the following options:
Console
1. On the MACsec tab, go to the Pre-shared keys section, and then click View beside the currently active key. If a CKN value isn't displayed, contact Google Cloud support for help.
2. In the Pre-shared keys section, verify that the start time listed for the currently active key matches the start time on your on-premises router. Do one of the following:
  - If the values don't match, refer to your router's manual to update the values on your router, and then verify whether a MACsec session can be established.
  - If the values match but the MACsec session is still operationally down on the link, then continue to the next step.
gcloud
1. Run the gcloud compute interconnects get-diagnostics command to display the active key's CKN value.
  
  If you have more than one key configured, then the key with the latest start time that isn't in the future is selected as the active key. Google's edge routers reject any new MACsec sessions that attempt to use old keys.
2. Get the MACsec configuration and then note the CAK value and the key's start time that correspond to the previously displayed CKN value:
```
gcloud compute interconnects macsec get-config INTERCONNECT_CONNECTION_NAME
```
  Replace INTERCONNECT_CONNECTION_NAME with the name of your Cloud Interconnect connection.
  
  The output is similar to the following:
```
preSharedKeys:
- name: key1
  ckn: 0101010189abcdef...0123456789abcdef
  cak: 0123456789abcdef...0123456789abcdef
  startTime: 2023-07-01T12:12:12Z
- name: key2
  ckn: 0202020289abcdef...0123456789abcdef
  cak: 0123456789abcdef...0123456789abcdef
  startTime: 2023-08-01T12:12:12Z
```
3. Verify that the active CKN, CAK, and start times on your on-premises router match the values that MACsec for Cloud Interconnect displays. Do one of the following:
  - If the values don't match, refer to your router's manual to update the values on your router, and then verify whether a MACsec session can be established.
  - If the values match but the MACsec session is still operationally down on the link, then continue to the next step.
View metrics to determine whether packets are dropping on the ingress or egress of the Cloud Interconnect connection. For information about viewing metrics, see Monitor connections.

To determine next steps, do the following:
- If network/interconnect/link/macsec/received_errors_count is incrementing, then the packets are dropping at the ingress Cloud Interconnect connection due to errors. This indicates that a protocol mismatch exists between your on-premises router and Google's edge routers. Check your on-premises router's logs to troubleshoot.
- If any of the following counters are incrementing, then contact Google Cloud Support for further assistance:
  - network/interconnect/link/macsec/received_dropped_packets_count
  - network/interconnect/link/macsec/send_errors_count
  - network/interconnect/link/macsec/send_dropped_packets_count
- If none of the following counters are incrementing, then it indicates that packets are dropping at the egress of your on-premises router. Check your on-premises router's logs to troubleshoot.
  - network/interconnect/receive_errors_count
  - network/interconnect/received_unicast_packets_count
  - network/interconnect/link/macsec/received_control_packets_count
  - network/interconnect/link/macsec/received_data_packets_count
  - network/interconnect/link/macsec/received_errors_count
  - network/interconnect/link/macsec/received_dropped_packets_count

MACsec is operational and is experiencing packet loss

You successfully enabled MACsec for Cloud Interconnect and MACsec is operationally up, but you are experiencing packet loss.

If your MACsec connection is operational but the Cloud Interconnect's Link Aggregation Control Protocol (LACP) status is Detached, verify that Secure Channel Identifier (SCI) is enabled on your on-premises router. For more information, see Configure your on-premises router.

View metrics to determine if packets are dropping on the ingress or egress of the Cloud Interconnect connection. For information about viewing metrics, see Monitor connections. If the Cloud Interconnect connection does not show any packet errors or loss, then proceed to checking the MACsec routers:

If network/interconnect/link/macsec/received_errors_count is incrementing, then the packets are dropping at the ingress Cloud Interconnect connection due to errors. This indicates that a protocol mismatch exists between your on-premises router and Google's edge routers. Check your on-premises router's logs to troubleshoot.
If any of the following counters are incrementing, then contact Google Cloud Support for further assistance:
- network/interconnect/link/macsec/received_dropped_packets_count
- network/interconnect/link/macsec/send_errors_count
- network/interconnect/link/macsec/send_dropped_packets_count
If none of the following counters are incrementing, then it indicates that packets are dropping at the egress of your on-premises router. Check your on-premises router's logs to troubleshoot.
- network/interconnect/receive_errors_count
- network/interconnect/received_unicast_packets_count
- network/interconnect/link/macsec/received_control_packets_count
- network/interconnect/link/macsec/received_data_packets_count
- network/interconnect/link/macsec/received_errors_count
- network/interconnect/link/macsec/received_dropped_packets_count