Priority Flow Control
In the Fibre Channel Protocol communication, the flow control is achieved by the buffer-to-buffer credit-based mechanism, where the receiving side is in control of the transmission. The Tx side does not start sending until it gets the R_RDY acknowledgment from the Rx side. After that, the Tx side transmits until it reaches the maximum credits it knows are available at the receiving side. In this way, the transmission is lossless because a Fibre Channel frame is always transmitted when both sides know there are enough resources for the frame to be stored and processed. The FCP credit-based flow control is illustrated on Figure 14-1.
Figure 14-1 Credit-Based Flow Control in the Fibre Channel Protocol
In the SAN, the only communication that takes place is the Fibre Channel Protocol communication. Over a link between an N-Port and an F_Port, there is only communication to or from the initiator, and it is only block-based data. In other words, the link is not divided by different types of communication. It is dedicated to this communication.
In Ethernet, a mechanism known as the IEEE 802.3X defines the native flow control at the level of the link. It allows when Ethernet frames can be transmitted over a link, and in the case of congestion, the receiving side can send a pause frame instructing the transmitting side to stop sending frames, as there are no resources to store and process them. This means the receiving side makes a decision to send a pause frame based on its available buffers. When the resources are available again, the receiving side sends another pause frame to notify the Tx side that it can now start transmitting again. This is illustrated on Figure 14-2.
Figure 14-2 IEEE 802.3X Link-Level Flow Control Using a Pause Frame
Although the 802.3X pause frame mechanism is not the same, it is very similar to the Fibre Channel protocol flow control. In both cases, the receiving side notifies the transmitting side, and a decision is made based on the availability of resources at the receiving side. In both situations, it is assumed that the link is dedicated only to one type of communication. With both the FCP flow control and the 802.3X pause frame mechanism, all the communication over the link will be stopped.
The challenge comes in using a converged Ethernet infrastructure to carry multiple types of traffic. Over the same link will be different types of Ethernet communication as well as FCoE data communication. If only the native flow control mechanisms are relied on, all the traffic will be disrupted.
To solve these issues, IEEE 802.1Qbb Priority Flow Control (PFC) was developed (see Figure 14-3). It uses the fact that in Ethernet environments, traffic is divided into different VLANs, and for each VLAN, you can set a different class of service (CoS) value in the VLAN tag. So, the IEEE 802.1p CoS and IEEE 802.1Q VLAN specifications can be used to allow traffic over the shared link to be divided and marked with different CoS values, based on which different priorities will be assigned. This also allows the pause frame to be used, not at link level, but to stop only traffic for a specific priority!
Figure 14-3 IEEE 802.1Qbb Priority Flow Control
In summary, the PFC enables lossless behavior for the Layer 2 flows on an Ethernet segment as follows:
- Dividing the communication into separate VLANs.
- Assigning CoS values for the traffic in each VLAN.
- Setting different priorities for the different CoS values.
- Defining which CoS values will be used for the pause control mechanism.
- Using the IEEE 802.3X pause control frames to communicate buffer availability to the sender.
- When the receive buffer of CoS is full, the PFC on the switch sends a pause for this class of traffic to prevent dropped frames.
- PFC emulates the Fibre Channel buffer-to-buffer credits.
If you implement PFC on the DCB link, you can create eight distinct virtual link types on the physical link, with the option of any of these links being paused and restarted independently, as shown in Figure 14-3. Not every type of traffic needs to be controlled with a pause frame. The Fibre Channel traffic, which is carried by the FCoE protocol, cannot be dropped. That’s why in the Cisco switches and converged network adapters (CNAs), which support the unified fabric, the CoS value of 3 is reserved for FCoE communication, and the pause frame control mechanism is always used for that traffic. On the other hand, the rest of the traffic, which is Ethernet communication, might not need to utilize the same mechanism, as it relies on the upper protocols’ mechanisms for secure communication, such as the TCP protocol.
The switches negotiate PFC capability with the connected CNA or DCB switch by default using the DCBX protocol. Then the PFC is enabled and link-level flow control remains disabled, regardless of its configuration settings. If the PFC negotiation fails, you can force PFC to be enabled on the interface, or you can enable IEEE 802.3X link-level flow control, which is a flow-control scheme within full-duplex Ethernet networks. If you do not enable PFC on an interface, you can enable the IEEE 802.3X link-level pause, which is disabled by default.