Troubleshoot application layer inspection

This page describes how to troubleshoot common issues that you might encounter when setting up application layer (Layer 7) inspection in your network. These issues might be related to security profiles, security profile groups, firewall endpoints, or firewall policies.

Generic troubleshooting steps

To troubleshoot common configuration errors related to Layer 7 inspection in your network, complete the tasks mentioned in the following sections.

Enable firewall policy rules logging

To enable logging for the Layer 7 inspection firewall rules in your firewall policies, do the following:

  1. In the Google Cloud console, go to the Firewall policies page.

    Go to Firewall policies

  2. Click the name of the firewall policy that has Layer 7 inspection firewall rules.

  3. In the Priority column, click the priority of the firewall policy rule for which you want to enable logs.

  4. Click Edit.

  5. For Logs, select On.

  6. Click Save.

Repeat the preceding steps for all the network firewall policies and hierarchical firewall policies that contain Layer 7 inspection firewall rules.

Verify your firewall policy rules configuration

  1. Make sure that the firewall policies with the Layer 7 inspection firewall rules are associated with the Virtual Private Cloud (VPC) network where your virtual machine (VM) workloads are located. For more information, see Associate a policy with the network.
  2. Verify that the firewall endpoints are associated with the VPC network where your VM workloads reside.
  3. Check the rule enforcement order to make sure that the rules applied to the traffic are in the correct sequence. For more information, see Policy and rule evaluation order.
  4. Check the effective firewall rules at the network and VM instance level. Make sure that the firewall policy rules for Layer 7 inspection firewall rules are getting hit for the network traffic.

All connections are allowed or denied but not intercepted

This scenario is encountered when you have configured all the components for Layer 7 inspection firewall rules, but the traffic is not intercepted and inspected for any threats or malicious activity.

To resolve this issue, follow these steps:

  1. Verify that the firewall endpoint and the VM workloads to be inspected are in the same zone.
  2. Verify that the logging is enabled for the firewall policy rule. For more information, see the Enable firewall policy rules logging section of this document.

  3. In the Google Cloud console, go to the Firewall policies page.

    Go to Firewall policies

  4. Click the firewall policy that contains the rule for Layer 7 inspection.

  5. In the Hit count column, view the number of unique connections used for the firewall rule.

  6. If the hit count is zero, the rule is not applied to the traffic. To verify if the setup is correct, see Generic troubleshooting steps section of this document.

  7. If the hit count is not a zero, click the count to go to the Log explorer page and follow these steps:

    1. Expand individual logs to view the connection, disposition, and remote location details.
    2. If the disposition is not set to intercepted and the fallback_action = ALLOW, check out the Generic troubleshooting steps section of this document to verify if the setup is correct.

Ingress firewall policy rule does not intercept incoming traffic

This scenario is encountered when the Layer 7 inspection firewall rules are not applied to the incoming traffic. It happens when the incoming traffic matches the other firewall rules before it hits the Layer 7 inspection firewall policy rules.

To resolve this issue, follow these steps:

  1. Verify that the logging is enabled for the firewall policy rule with Layer 7 inspection. For more information, see Enable firewall policy rules logging section of this document.
  2. Make sure that the firewall policy with the Layer 7 inspection firewall rule is associated with the VPC network where your VM workloads are located. For more information, see Associate a policy with the network.
  3. Verify that the firewall endpoints are associated with the VPC network where your VM workloads reside.
  4. To verify that the Layer 7 inspection firewall rule is applied, run the connectivity tests based on the source and destination you defined in the rule. To learn how to run Connectivity Tests, see Create and run Connectivity Tests.
  5. Check the sequence in which the rules are applied to the incoming traffic. To change this sequence, see Change policy and rule evaluation order.

A threat is not detected on some or all of the connections

This scenario might be encountered when your traffic is encrypted, or the threat prevention policy is not set to detect the threat.

If your traffic is encrypted, make sure that you have enabled Transport Layer Security (TLS) inspection on your network. To learn more about how to enable TLS inspection, see Set up TLS inspection.

If TLS inspection is enabled, distinguish between messages seen from the client versus error messages when Cloud Next Generation Firewall blocks a threat. For more information, see Error messages.

Make sure that the threat prevention policy is set to detect this threat:

  1. Review your security profile to identify that the override actions for this threat are set as expected.
  2. Add override actions to your security profiles to make sure that the threat is captured.

Misconfigured intrusion prevention service firewall rules

This scenario occurs when there is no valid firewall endpoint or the endpoint is not associated with the VPC network where your VM workloads are located. As a default fallback action, Cloud NGFW allows the traffic, and adds apply_security_profile_fallback_action = ALLOW to the firewall logs. To view the firewall logs, see View logs.

To resolve this issue, follow these steps:

  1. To enable logging for the Layer 7 inspection firewall policy rules in your network, see the Enable firewall policy rules logging section of this document.

  2. Create the log-based metrics, log-based alerts, or both by using the following filters.

      jsonPayload.rule_details.action="APPLY_SECURITY_PROFILE_GROUP"
  jsonPayload.rule_details.apply_security_profile_fallback_action="ALLOW"

The filter generates incident details, which helps you to understand the log match condition, notification rate limit, incident auto-close duration, log labels, and log severity with the summary.

Error messages

This section describes the common error messages you get when the TLS trust is improper or Cloud NGFW blocks a threat. To learn how to set up TLS inspection, see Set up TLS inspection.

Firewall policy rule is blocked

You received an error message similar to the following error from a client during an SSH session.

      curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

To resolve this error, view the log and validate it. For more information, see Use Firewall Rules Logging.

Misconfigured trust

You received an error message similar to the following error from a client during an SSH session.

      curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection

This error indicates a misconfigured trust issue. This issue is caused either due to an incorrect configuration or the absence of a certificate authority (CA). To resolve this error, enable Certificate Authority Service.

Endpoint policies are ignored

Only firewall policy rules are being evaluated and traffic is not mirrored to Cloud Intrusion Detection System for inspection while using Cloud Next Generation Firewall L7 inspection policies.

To resolve this, you must ensure that your Cloud NGFW L7 inspection policies (rules with the apply_security_profile_group action) do not apply to packets that you need to inspect with Cloud IDS.

What's next