In the scenarios of high-performance computing, AI large model training and inference, in order to increase the computing power scale of the cluster as much as possible, servers generally adopt eight-card GPU design and configure eight 200G IB cards to provide communication and bandwidth for the computing network. Assure. In the process of establishing large-scale server clusters, the investment cost of high-speed, low-latency and lossless network equipment (IB HDR switches and IB HDR cables) is increasing; especially when the number of server nodes exceeds 100, the network architecture From the two-layer FAT TREE to the three-layer FAT TREE, the proportion of network investment will increase significantly.
The NVIDIA 200G HDR Splitter H-Cable uses a better way to solve the port connection problem between spine switches and leaf switches, greatly improving the network expansion capabilities of IB switches, making it easier to use in the future. Under the layer architecture, the maximum access capability is increased from 100 server nodes to 200 server nodes. The following is a brief introduction to this cable and its networking method:
1. What is 200G HDR Splitter H-Cable?
The 200G HDR Splitter H-Cable (MFS1S90-HxxxE) cable provided by NVIDIA is a cost-effective 2x 200Gb/s to 2x 200Gb/s active optical splitting based on QSF-P56 VCSEL (vertical cavity surface emitting laser) Router Cable (AOC), each QSFP56 end contains an EEPROM that provides product and status monitoring information that can be read by the host system. Comply with SFF-8665/RoHS/FF-8636 standards;
2.Main application scenarios of 200G HDR Splitter H-Cable
The main application scenario is to use a fat-tree topology to connect 200G leaf switches and spine switches, providing cross-connection functionality that allows the HDR InfiniBand QSFP56 switch port to operate as 2x HDR100, as shown in Figure 1 below;
3.Key advantages of 200G HDR Splitter H-Cable
- Improve port access capabilities
One 200G HDR Splitter H-Cable (four modules) can complete the full interconnection between two spines and two leaves. If you use HDR100G direct connection cables, you will need 4 cables and 8 modules, occupying 8 devices. port, which greatly wastes the port resources of the switch; if you use HDR 200G direct connection, you also need at least 4 cables, occupying 8 switch ports. Although the upstream and downstream bandwidth is improved, due to the port The relationship between port occupancy limits the scale of the network, as shown in Figure 2 below;
Using traditional HDR 200G direct connection cables and HDR IB switches, the two-layer Fat Tree network can build up to 20 spines.
For a lossless network of 40 leaves, the maximum access capability of each leaf is 20 200G ports, with a total access capability of 800 200G ports, which can meet the networking needs of 100 servers.
However, using 200G HDR Splitter H-Cable and HDR IB switches for networking, under the same two-layer Fat Tree architecture, the maximum access capability can be doubled to 1,600 200G ports, which can satisfy 200 units Networking of eight-card GPU server
need. As shown in Figure 3 below:
2. Expand network scale
As shown in Figure 3 above, a 200G HDR Splitter H-Cable is used between spine1-2 and leaf1-2. For Leaf1, one port can be used to interconnect with two spines at the same time. Even using With 20 ports, 40 spines can be interconnected at the same time, doubling the network size.
As shown in Figure 4 below, in a network that also meets the network requirements of 200 eight-card GPU servers, HDR 200G direct Connection requires the establishment of a three-layer Fat Tree architecture, which requires more switch equipment and AOC cables.
3. Low down the cost
The following table is a comparison of the number of devices required to use HDR 200G direct cable and HDR Splitter H-Cable in a network with 200 eight-card GPU servers:
As can be seen from the above table:
When 200G HDR direct-connect cables are used for networking between switches, 200 IB switches and 4800 HDR cables are required. When 200G HDR Splitter H-Cable is used for networking, the number of switches becomes 120. The number of cables is 2,400, compared to 80 less; although the price of 200G HDR Splitter H-Cable is about 2.5 times that of 200G HDR direct-connect cable, because the number of switches is reduced by 80 , the number of cables has been reduced by 2,400, and the overall cost savings is approximately 30% to 40%.
- Summary
As the user server cluster gradually expands in size, especially starting from 100 servers, the use of NVIDIA 200G HDR Splitter H-Cable (MFS1S90-HxxxE) greatly reduces the number of switch equipment and cables and reduces costs. , improves the port access capabilities of the network, and provides smoother and more economical expansion capabilities for network upgrades, which can meet the use and deployment in various scenarios of the data center.
