To identify the activeSubnet Manager (SM)node in an InfiniBand fabric, the correct command sequence is:

sminfo

Displays general information about the active SM in the fabric, including itsLID.

smpquery ND <LID>

Resolves theNode Description (ND)at the given LID, revealing the exact hostname or label of the SM server.

From the InfiniBand Tools Guide:

"The sminfo utility provides the LID of the master SM. Use smpquery ND <LID> to resolve the node name hosting the SM."

This two-step approach is standard for locating and validating the SM identity in fabric diagnostics.

Incorrect Options:

B (Nl)is an invalid query type.

CandDdo not identify SMs.

Modern AI training (especially with LLMs) requires extremely high-speed, parallel access to large datasets. Adedicated storage fabricseparates data I/O traffic from the training compute path and avoids contention.

FromNVIDIA DGX Infrastructure Reference Architectures:

"Dedicated storage networks eliminate I/O bottlenecks by providing low-latency, high-bandwidth access to distributed storage for large-scale training jobs."

"Parallel access to datasets is key for performance, especially in multi-node, multi-GPU AI clusters."

Security (B)is important, but not the core reason for a storage fabric.

Cost (D)is typicallyincreased, not reduced, with dedicated fabrics.

To upgrade the MLNX-OS on InfiniBand switches, the recommended procedure is as follows:​

Connect to the switch via SSH: Establish a secure shell connection to the switch using its management IP address.​

Fetch the MLNX-OS software image: Obtain the appropriate MLNX-OS software image from the official source or repository.​

Use the 'install' command to perform the upgrade: Execute the 'install' command on the switch to initiate the upgrade process with the fetched software image.​

This method ensures a smooth and efficient upgrade without the need for physical intervention or service disruption.​

Reference Extracts from NVIDIA Documentation:

"Click on Systems → MLNX-OS Upgrade. Select the desired upgrade method (e.g. 'Install from local file'). Select your image and click 'Install Image'."

In Cumulus Linux, configuring a bond interface with Link Aggregation Control Protocol (LACP) involves setting the bond mode to 'lacp'. The correct command to achieve this is:​

nv set interface bond1 bond mode lacp​

This command sets the bonding mode of 'bond1' to LACP, enabling dynamic link aggregation for increased bandwidth and redundancy.​

Reference Extracts from NVIDIA Documentation:

"To reset the link aggregation mode for bond1 to the default value of 802.3ad, run the nv set interface bond1 bond mode lacp command."​

To check the status and link layer of InfiniBand interfaces, the ibstat command is used. For example:​

ibstat -d mlx5_0

This command provides detailed information about the InfiniBand device, including its state (e.g., Active), physical state (e.g., LinkUp), and link layer (e.g., InfiniBand).​

NVIDIA NetQ is a highly scalable network operations toolset that provides visibility, troubleshooting, and validation of networks in real-time. It delivers actionable insights and operational intelligence about the health of data center networks—from the container or host all the way to the switch and port—enabling a NetDevOps approach.

NetQ can be used as the functional test platform for the network CI/CD in conjunction with NVIDIA Air. Customers benefit from testing the new configuration with NetQ in the NVIDIA Air environment (“digital twin”) and fix errors before deploying to their production. ​

To enable RDMA over Converged Ethernet (RoCE) on a Cumulus switch, the correct command is:​

nv set qos roce enable on

This command configures the Quality of Service (QoS) settings to support RoCE, ensuring that the necessary parameters for lossless Ethernet are applied.​

RoCE Adaptive Routing is a key feature of NVIDIA Spectrum-X that ensures high performance and resiliency in the network, even in the event of a link failure. This technology dynamically reroutes traffic to the least congested and operational paths, effectively mitigating the impact of link failures. By continuously evaluating the network's egress queue loads and receiving status notifications from neighboring switches, Spectrum-X can adaptively select optimal paths for data transmission. This ensures that the network maintains high throughput and low latency, crucial for AI workloads, even when certain links are down.​

Reference Extracts from NVIDIA Documentation:

"Spectrum-X employs global adaptive routing to quickly reroute traffic during link failures, minimizing disruptions and preserving optimal storage fabric utilization."​

"RoCE Adaptive Routing avoids congestion by dynamically routing large AI flows away from congestion points. This approach improves network resource utilization, leaf/spine efficiency, and performance."​

In multi-tenant AI cloud environments, ensuring that each tenant's workloads are isolated and secure is paramount. The NVIDIA Spectrum-X platform addresses this need through itsTraffic Isolationcapabilities. This feature ensures that network resources are partitioned effectively, preventing unauthorized access and interference between tenants. By implementing Traffic Isolation, the provider can maintain strict boundaries between different tenant environments, ensuring both security and performance consistency.​

Reference Extracts from NVIDIA Documentation:

"Spectrum-X enhances multi-tenancy with performance isolation to ensure tenants' AI workloads perform optimally and consistently."​

"Spectrum-X utilizes the programmable congestion control function on the BlueField-3 hardware platform to accurately assess the congestion condition of the traffic path by using in-band telemetry information... to achieve the goal of performance isolation to ensure that each tenant gets the best expected performance in the cloud and is not negatively affected by congestion of other tenants."​

The correct utility isibnetdiscover.

From the official NVIDIA InfiniBand Utilities Guide:

"ibnetdiscover scans the fabric and returns a topology of all switches and end nodes, including their GUIDs, LIDs, port numbers, and link speeds."

It generates a fabric map with node-to-port relationships and shows:

GUIDs

LIDs (Local IDs)

Link speeds and widths

Switch-to-host connections

This is essential for network topology validation and mapping physical port additions.

Incorrect Options:

ib_check_routes– for routing table diagnostics.

ibhosts– shows host information but not switch-level port mapping.

ibswitches– shows switch info, but lacks port-level LID/link speed mapping.

FromNVIDIA Performance Tuning Guide (ib_write_bw Tool Usage):

"-S <SL>: Specifies the Service Level (SL) to use for the InfiniBand traffic. SL is used for setting priority and mapping to virtual lanes (VLs) on the IB fabric."

This flag is useful when testing QoS-aware setups or validating SL/VL mappings.

Incorrect Options:

A– No such flag for burst size.

B– -q defines number of QPs.

C– --rate or -R is used for rate-limiting.

NVIDIA AIR (AI-Ready Infrastructure) is a cloud-based simulation platform designed to model and validate data center network deployments, including Spectrum-X Ethernet networks, using realistic topologies and configurations. When creating a custom topology in NVIDIA AIR, users can define network links between devices (e.g., spine and leaf switches) using a DOT file format, which is based on the Graphviz graph visualization software. The question asks for the correct syntax to define a link between port 1 on a spine switch (spine-01) and port 41 on a leaf switch (gpu-leaf-01) in a NVIDIA AIR simulation.

According to NVIDIA’s official NVIDIA AIR documentation, the DOT file format is used to specify network topologies, including nodes (devices) and links (connections between ports). The syntax for defining a link in a DOT file uses a double dash (--) to indicate a connection between two ports, with each port specified in the format "":"". For Spectrum-X networks, which typically use Cumulus Linux or SONiC on NVIDIA Spectrum switches, ports are commonly labeled as swpX (switch port X) rather than ethX (Ethernet interface), especially for switch-to-switch connections in a leaf-spine topology. The correct syntax for the link between port 1 on spine-01 and port 41 on gpu-leaf-01 is:

"spine-01":"swp01" -- "gpu-leaf-01":"swp41"

This syntax uses swp01 and swp41 to denote switch ports, consistent with Cumulus Linux conventions, and the double dash (--) to indicate the link, as required by the DOT file format.

Exact Extract from NVIDIA Documentation:

“You can create custom topologies in Air using a DOT file, which is the file type used with the open-source graph visualization software, Graphviz. DOT files define nodes, attributes, and connections for generating a topology for a network. The following is an example of a link definition in a DOT file:

"leaf01":"swp31" -- "spine01":"swp1"

This specifies a connection between port swp31 on leaf01 and port swp1 on spine01. Port names typically follow the switch port naming convention (e.g., swpX) for Cumulus Linux-based switches.”

—NVIDIA Air Custom Topology Guide

This extract confirms that option A is the correct answer, as it uses the proper DOT file syntax with swp01 and swp41 for port names and the double dash (--) for the link, aligning with NVIDIA AIR’s topology definition process for Spectrum-X simulations.

Analysis of Other Options:

B. "spine-01":"swp1" to "gpu-leaf-01":"swp41": This option uses the correct port naming convention (swp1 and swp41) but incorrectly uses the word to as the connector instead of the double dash (--). The DOT file format requires -- to define links, making this syntax invalid for NVIDIA AIR.

C. "spine-01":"eth1" to "gpu-leaf-01":"eth41": This option uses ethX port names, which are typically used for host interfaces (e.g., servers) rather than switch ports in Cumulus Linux or SONiC environments. Switch ports in Spectrum-X topologies are labeled swpX. Additionally, the use of to instead of -- is incorrect for DOT file syntax, making this option invalid.

D. "spine-01":"eth1" - "gpu-leaf-01":"eth41": This option uses a single dash (-) instead of the required double dash (--) and incorrectly uses ethX port names instead of swpX. The ethX naming is not standard for switch ports in Spectrum-X, and the single dash is not valid DOT file syntax, making this option incorrect.

Why "spine-01":"swp01" -- "gpu-leaf-01":"swp41" is the Correct Answer:

Option A correctly adheres to the DOT file syntax used in NVIDIA AIR for defining network links:

Node and Port Naming: The nodes spine-01 and gpu-leaf-01 are specified with their respective ports swp01 and swp41, following the swpX convention for switch ports in Cumulus Linux-based Spectrum-X switches.

Link Syntax: The double dash (--) is the standard connector in DOT files to indicate a link between two ports, as required by Graphviz and NVIDIA AIR.

Spectrum-X Context: In a Spectrum-X leaf-spine topology, connections between spine and leaf switches (e.g., Spectrum-4 switches) use switch ports labeled swpX, making swp01 and swp41 appropriate for this simulation.

This syntax ensures that the NVIDIA AIR simulation accurately models the physical connection between spine-01 port 1 and gpu-leaf-01 port 41, enabling validation of the Spectrum-X network topology. The DOT file can be uploaded to NVIDIA AIR to generate the topology, as described in the documentation.

In InfiniBand networks,Partitioning using Partition Keys (PKeys)is the standard method for implementing multi-tenancy and traffic isolation. PKeys allow administrators to define logical partitions within the fabric, ensuring that traffic is confined to designated groups of nodes. This mechanism is essential for creating secure and isolated environments in multi-tenant architectures.​

The Unified Fabric Manager (UFM) leverages PKeys to manage these partitions effectively, enabling administrators to assign and control access rights across different tenants. This approach ensures that each tenant's traffic remains isolated, maintaining both security and performance integrity within the shared fabric.​

The correct answer isPXN (Peer eXchange Network).

From theNVIDIA NCCL Documentation:

"PXN enables communication between GPUs connected via NVLink and NICs by treating the GPUs as a distributed switch. This architecture improves bandwidth utilization by enabling any GPU to communicate with the NIC via the shortest path available, even if it's not directly connected to the NIC."

This enhances GPU-to-NIC and NIC-to-GPU transfers, leveraging the NVLink topology. It significantly boosts performance in multi-GPU setups where not every GPU is directly connected to the NIC.

Other options:

Adaptive Routingis a fabric-level feature for dynamic path rerouting.

Graph Search Optimizationis used internally for topology modeling in NCCL.

SHARP v2is a switch-based collective acceleration method, unrelated to PXN.

Spectrum-X achieves network isolation in multi-tenant environments by implementing Layer 3 Virtual Network Identifiers (L3VNIs) per Virtual Routing and Forwarding (VRF) instance. This approach allows each tenant to have a separate routing table and network segment, ensuring that traffic is isolated and secure between tenants.​

Reference Extracts from NVIDIA Documentation:

"Spectrum-X enhances multi-tenancy with performance isolation to ensure tenants' AI workloads perform optimally and consistently."​

TheLocal Route Header (LRH)in InfiniBand is termed "local" because it is used exclusively for routing packets within a single subnet. The LRH contains the destination and source Local Identifiers (LIDs), which are unique within a subnet, facilitating efficient routing without the need for global addressing. This design optimizes performance and simplifies routing within localized network segments.​

InfiniBand is a high-performance, low-latency interconnect technology widely used in AI and HPC data centers, supported by NVIDIA’s Quantum InfiniBand switches and adapters. The Local Routing Header (LRH) is a critical component of the InfiniBand packet structure, used to facilitate routing within an InfiniBand fabric. The question asks why the LRH is called a “local header,” which relates to its role in the InfiniBand network architecture.

According to NVIDIA’s official InfiniBand documentation, the LRH is termed “‘local’ because it contains the addressing information necessary for routing packets between nodes within the same InfiniBand subnet.” The LRH includes fields such as the Source Local Identifier (SLID) and Destination Local Identifier (DLID), which are assigned by the subnet manager to identify the source and destination endpoints within the local subnet. These identifiers enable switches to forward packets efficiently within the subnet without requiring global routing information, distinguishing the LRH from the Global Routing Header (GRH), which is used for inter-subnet routing.

Exact Extract from NVIDIA Documentation:

“The Local Routing Header (LRH) is used for routing InfiniBand packets within a single subnet. It contains the Source LID (SLID) and Destination LID (DLID), which are assigned by the subnet manager to identify the source and destination nodes in the local subnet. The LRH is called a ‘local header’ because it facilitates intra-subnet routing, enabling switches to forward packets based on LID-based forwarding tables.”

—NVIDIA InfiniBand Architecture Guide

This extract confirms that option A is the correct answer, as the LRH’s primary function is to route traffic between nodes within the local subnet, leveraging LID-based addressing. The term “local” reflects its scope, which is limited to a single InfiniBand subnet managed by a subnet manager.

NVIDIA NetQ is a network operations tool that provides real-time visibility and automated validation of the network fabric. It helps in identifying misconfigurations, monitoring network health, and ensuring that the fabric meets the required specifications for AI workloads.​

In the Spectrum-X architecture,BlueField-3 SuperNICsare responsible for executing the congestion control algorithm. They handle millions of congestion control events per second with microsecond reaction latency, applying fine-grained rate decisions to manage data flow effectively. This ensures optimal network performance by preventing congestion and packet loss.​