Generate NVIDIA bug report for Blackwell GPUs

This document explains how to create an NVIDIA bug report for your machine types that use NVIDIA Blackwell GPUs. If you want to identify which of your machine types are using the NVIDIA Blackwell GPU, see GPU models.

If your machine type uses the NVIDIA Blackwell GPU architecture, the generated NVIDIA bug report doesn't include critical low-level hardware data when you run the nvidia-bug-report.sh script. This data contains information such as the physical layer status of NVLink connections, internal GPU register values, and raw diagnostic segments from the firmware. This information is essential for diagnosing issues, especially those related to NVLink, which could lead to GPU Xid errors or unresolved performance degradation.

Generate an NVIDIA bug report

To generate a bug report, complete the following steps:

  1. Connect to your GPU instance. Choose one of the following options:

  2. Download and install the MFT package by selecting one of the following options:

Container-Optimized OS

If your instance uses a Container-Optimized OS (COS) as the guest operating system, use the open source GCE COS NVIDIA Bug Report Collector tool to generate the bug report with MFT. This tool automatically injects supported MST kernel modules that match the COS kernel, installs the userspace tool, generates the bug report, and optionally uploads the result to a Cloud Storage bucket.

Other OS

For other Linux OSes, complete the following steps:

  1. Download and install NVIDIA Firmware Tools (MFT) software version 4.32.0 or higher from the NVIDIA website.
  2. Install the tool. For more information, see Compilation and installation in the NVIDIA Firmware Tools (MFT) Documentation. After you install MFT, the nvidia-bug-report.sh script automatically uses the MFT tools to generate the report. You don't need to interact with MFT tools directly.
  3. Run the nvidia-bug-report.sh script to generate a bug report. This process takes about two minutes.
  4. Extract the report.
  5. Verify that the report includes MFT data by running the following command on your extracted bug report file: 
    grep -m 1 -A 30 "Starting GPU MST dump.." PATH_TO_UNZIPPED_BUG_REPORT

    The output is similar to the following example:

    Starting GPU MST dump..
... (additional MFT data) ...