
                               README Notes
                     Broadcom bnxtnd.sys Windows Driver

                 Copyright (c) 2015 - 2025 Broadcom Incorporated
                           All rights reserved


Table of Contents
=================

  Introduction
  Device Manager Errors
  Port Speeds
  SR-IOV and Jumbo Frames
  NPAR and SR-IOV
  SR-IOV and Get-NetAdapterSriov
  SR-IOV and Promiscuous mode
  Guest Mode RDMA (RDMA on a VF)
  Get-NetAdapterStatistics on Windows Server 2022
  Adjusting Number of MSIX vectors
  PAM4 Support
  Server 2025 driver uninstallation with 576xx devices present



Introduction
================================
This file describes the bnxtnd.sys Windows driver for the Broadcom NetXtreme-C and NetXtreme-E BCM573xx, BCM574xx. BCM575xx, BCM576xx 10/25/40/50/100/200/400 Gbps Ethernet Network Controllers.



Device Manager Error Codes
================================
This device cannot find enough free resources that it can use. (Code 12)
Most likely this is due to the OS running out of MSIX messages. See "Adjusting Number of MSIX vectors" below.


Advanced Properties Page
================================
The driver exposes certain user configurable settings through "Advanced Properties" user interface in Windows' Device Manager. Example of such properties are "Speed & Duplex", "Forward Error Control", and WoL.
Availability and configurability of options under individual setting are subject to current hardware model, NVM settings, system or firmware configurations.
Please ensure the SKU, system and device configuration support the particular option.



Port Speeds
================================
On some dual-port devices, the port speed of each port must be compatible with the port speed of the other port. 10Gbps and 25Gbps are not compatible speeds.
For example, if one ports Speed & Duplex advanced property setting is 10Gbps and link is up, the other port cannot be set to 25Gbps.
The firmware will allow incompatible speeds to be set on the two ports if link is not up yet.
  - Subsequent link up on one port will render the incompatible speed on the other port to become unsupported.
  - This will lead to a system error log like this may appear when this scenario happens:

    - Incompatible speed selection between Port 1 and Port 2. Link configuration became illegal.

If Speed & Duplex settings for the two ports are incompatible, you will also see this system error log message:

   Incompatible speed selection between Port 1 and Port 2. Reported link speeds are correct and might not match Speed and Duplex setting.

On some NPAR (NIC partitioning) devices where one port is shared by multiple
PCI functions, the port speed is pre-configured by the CCM BIOS utility (invoked by pressing ^S at the Broadcom banner during Legacy BIOS boot), or, if using a UEFI system,
the HII interface for the NIC. This is why the Speed & Duplex advanced property is not available for NPAR miniports.



SRIOV and Jumbo Frames
================================
Windows SRIOV includes a failover feature such that if the VF path fails to work for some reason, or if the VM is migrated to a host without SR-IOV enabled, the non-VF
synthetic NIC will continue to be connected. This is why a VM using SR-IOV will show 2 devices in Device Manager per one network connection. The synthetic path is
established before, and torn down after, the VF path.

If jumbo frames are needed to the VF, make sure that the VF and the corresponding synthetic NIC have identical Jumbo Frame setting and the failover can work properly.

With or without SR-IOV, the jumbo frame settings for the host OS PF driver must be the same or greater than any Jumbo Frame settings for VM NICs using that host PF. 



NPAR and SR-IOV
================================
The number of VFs available when NPAR is enabled will be lower than when NPAR is not used.
To maximize the number of VFs available, reduce the MSI-X vectors to 4 or less in the UEFI Device configuration menu.

Guest Mode RDMA (RDMA on a VF) is not supported when NPAR is enabled.



SR-IOV and Get-NetAdapterSriov
================================
Get-NetAdapterSriov will show supported or not supported strictly based on the presence of the *SRIOV keyword in the device registry. This means that if
SR-IOV is disabled on the device via NVRAM, it may still show as supported via the Get-NetAdapterSriov powershell command because the *SRIOV keyword is present
by default in the registry. If you are not able to create virtual functions, check your NVRAM configuration to make sure SR-IOV is enabled.



SR-IOV and Promiscuous mode
================================
Promiscuous mode is not supported on VFs.



Guest Mode RDMA (RDMA on a VF)
================================
Windows Server has a known issue with Guest Mode RDMA that can prevent the Microsoft synthetic NIC from starting, resulting in a yellow warning icon. To resolve
this, simply disable and then re-enable the Microsoft synthetic NIC in the VM's Device Manager.



Get-NetAdapterStatistics on Windows Server 2022
================================
On Windows Server 2022, the PowerShell command (Get-NetAdapterStatistics).RdmaStatistics will show junk values when RDMA is disabled on the given adapter. When
RDMA is enabled correct statistics are shown. This is a known issue in Windows Server 2022.



Adjusting Number of MSIX vectors
================================
The number of MSIX vectors is set soon after the NIC is powered on, and before the PCIe bus enumerates the NIC.
As a result, the number of MSIX vectors can only be changed by the CCM BIOS utility (invoked by pressing ^S at the Broadcom banner during Legacy BIOS boot), or, if using a UEFI system, the HII interface for the NIC.
There is a separate parameter for the "Max number of PF MSIX vectors" for a PF (Physical Function) in the host operating system, and for the "MSI-X Vectors per VF" (Virtual Function) when using SRIOV in a guest virtual machine.

The default values were chosen to cover the majority of configurations, but there are definitely cases where a system administrator might want to increase or decrease the number of MSIX vectors.

If you are running many VM's using VMQ (not SRIOV) and saw that some VM NICs were not getting a VMQ, you would *increase* the Max number of PF MSIX vectors.
If you get a code 12 "yellow-bang" on a PF, you would *decrease* Max number of PF MSIX vectors. If you get it on a VF, you would decrease MSI-X Vectors per VF

Device manager code 12 yellow-bangs are more likely if you have many miniport instances (due to either enabling NPAR or by having many NICs installed) and if you have only a few processor cores (1, 2, or 4).
Many miniport instances can increase the number of MSIX message resources needed. Few processors can decrease the number of available MSIX message resources.



PAM4 Support
================================
Windows does not contain a PAM4 signaling configuration parameter in adapter device advanced properties
For speeds supported by both NRZ and PAM4 Windows will default to NRZ without the following work around.

Examine the NXE device in Windows Device Manager, Network Adapters, Broadcom xxx. 
Right mouse click and select properties, select Details tab, select Driver Key from the Property drop down, and record the last 4 digits after the backslash.
Enter the registry editor (regedit.exe) and traverse to 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\xxx where xxx is the number recorded above
Add a new string type value named PreferredSignalingMode, and set the value to 1. Save and exit the registry editor.


Server 2025 driver uninstallation with 576xx devices present
================================
When the Windows NXE driver is uninstalled, including driver removal, it must be uninstalled for all 576xx devices. Uninstallation with driver removal causes the original "inbox"
NXE driver to be restored. The inbox NXE driver does not support 576xx devices and for this reason it is critical to uninstall all 576xx devices. If a 576xx device is left installed
when the inbox driver is restored, it can create an incompatibility condition where the inbox driver is referenced, by the 576xx INF, for a device that it does not support.

