MetroCluster Manuals ( CA08871-401 )
Considerations for ISLs
Inter-Switch Links (ISLs) carrying MetroCluster traffic on all MetroCluster IP configurations and network topologies have certain requirements. These requirements apply to all ISLs carrying MetroCluster traffic, regardless of whether the ISLs are direct or shared between customer switches.
General MetroCluster ISL requirements
The following applies to ISLs on all MetroCluster IP configurations:
-
Both fabrics must have the same number of ISLs.
-
ISLs on one fabric must all be the same speed and length.
-
ISLs in both fabrics must be the same speed and length.
-
The maximum supported difference in distance between fabric 1 and fabric 2 is 20km or 0.2ms.
-
The ISLs must have the same topology. For example, they should all be direct links, or if the configuration uses WDM, then they must all use WDM.
-
The ISL speed must be least 10Gbps.
-
There must be least one 10Gbps ISL port per fabric.
Latency and packet loss limits in the ISLs
The following applies to round-trip traffic between the MetroCluster IP switches at site_A and site_B, with the MetroCluster configuration in steady state operation:
-
As the distance between two MetroCluster sites increases, latency increases, usually in the range of 1 ms round-trip delay time per 100 km (62 miles). Latency also depends on the network service level agreement (SLA) in terms of the bandwidth of the ISL links, packet drop rate, and jitter on the network. Low bandwidth, high jitter, and random packet drops lead to different recovery mechanisms by the switches, or the TCP engine on the controller modules, for successful packet delivery. These recovery mechanisms can increase overall latency.
-
Any device that contributes to latency must be accounted for.
-
Packet loss must be less than, or equal to, 0.01%. The maximum packet loss is the sum of all loss on all links on the path between the MetroCluster nodes, and the loss on the local MetroCluster IP interfaces.
-
The supported jitter value is 3ms for round trip (or 1.5ms for one-way).
-
The network should allocate and maintain the SLA amount of bandwidth required for MetroCluster traffic, regardless of microbursts and spikes in the traffic.
-
If you are using ONTAP 9.7 or later, the intermediate network between the two sites must provide a minimum bandwidth of 4.5Gbps for the MetroCluster IP configuration.
Transceiver and cable considerations
Any SFPs or QSFPs supported by the equipment vendor are supported for the MetroCluster ISLs. SFPs and QSFPs provided by us or the equipment vendor must be supported by the switch and switch firmware.
When connecting the controllers to the switches and the local cluster ISLs, you must use the transceivers and cables provided by us with the MetroCluster.
When you use a QSFP-SFP adapter, whether you configure the port in breakout or native speed mode depends on the switch model and firmware. For example, using a QSFP-SFP adapter with Cisco 9336C switches running NX-OS firmware 9.x or 10.x requires that you configure the port in native speed mode.
| If you configure an RCF, verify that you select the correct speed mode or use a port with an appropriate speed mode. |
Using xWDM, TDM, and external encryption devices
When you use xWDM/TDM devices or devices providing encryption in a MetroCluster IP configuration your environment must meet the following requirements:
-
When connecting the MetroCluster IP switches to the xWDM/TDM, the external encryption devices or xWDM/TDM equipment must be certified by the vendor for the switch and firmware. The certification must cover the operating mode (such as trunking and encryption).
-
The overall end-to-end latency and jitter, including the encryption, cannot be more than the maximum amount stated in the IMT and in this documentation.
Supported number of ISLs and breakout cables
The following table shows the supported maximum number of ISLs that can be configured on a MetroCluster IP switch using the Reference Configuration File (RCF) configuration.
MetroCluster IP switch model |
Port type |
Maximum number of ISLs |
|---|---|---|
Broadcom-supported BES-53248 switches |
Native ports |
4 ISLs using 10Gbps or 25Gbps |
Broadcom-supported BES-53248 switches |
Native ports (Note 1) |
2 ISLs using 40Gbps or 100Gbps |
Cisco 3132Q-V |
Native ports |
6 ISLs using 40Gbps |
Cisco 3132Q-V |
Breakout cables |
16 ISLs using 10Gbps |
Cisco 3232C |
Native ports |
6 ISLs using 40Gbps or 100Gbps |
Cisco 3232C |
Breakout cables |
16 ISLs using 10Gbps |
Cisco 9336C-FX2 (not connecting NS224 shelves) |
Native ports |
6 ISLs using 40Gbps or 100Gbps |
Cisco 9336C-FX2 (not connecting NS224 shelves) |
Breakout cables |
16 ISLs using 10 |
Cisco 9336C-FX2 (connecting NS224 shelves) |
Native ports (Note 2) |
4 ISLs using 40Gbps or 100Gbps |
Cisco 9336C-FX2 (connecting NS224 shelves) |
Breakout cables (Note 2) |
16 ISLs using 10Gbps |
NVIDIA SN2100 |
Native ports (Note 2) |
2 ISLs using 40Gbps or 100Gbps |
NVIDIA SN2100 |
Breakout cables (Note 2) |
8 ISLs using 10Gbps or 25Gbps |
Note 1: Using 40Gbps or 100Gbps ISLs on a BES-53248 switch requires an additional license.
Note 2: The same ports are used for native speed and breakout mode. You must choose to use ports in native speed mode or breakout mode when creating the RCF file.
-
All ISLs on one MetroCluster IP switch must be the same speed. Using a mix of ISL ports with different speeds concurrently is not supported.
-
For optimum performance, you should use at least one 40Gbps ISL per network. You should not use a single 10Gbps ISL per network for high capacity platforms.
| We recommends that you configure a small number of high bandwidth ISLs, rather than a high number of low bandwidth ISLs. For example, configuring one 40Gbps ISL instead of four 10Gbps ISLs is preferred. When using multiple ISLs, statistical load-balancing can impact the maximum throughput. Uneven balancing can reduce throughput to that of a single ISL. |
