Fibre Channel Flow Control
One of the major goals for the FC Protocol communication is to create a secure and reliable environment that allows all the FC frames to be received at the destination and to be received in the correct order. You saw how the whole FC switched fabric converges as well as how steps are in place at the different levels and on each link hop to make sure the environment is stable and that the devices communicating are compatible, support the needed FC classes of service and ULPs mapping, and so on. One very important component in the way the FC protocol communicates is the mechanism utilized to control the flow of FC frames in a transmission between two FC ports.
The FCP uses a very strict flow control mechanism based on two major components:
- The size of buffers to store received FC frames on each FC port. These buffers are called BB_Credits, or buffer-to-buffer credits.
- The receiving side always being in control of the transmission.
These two components are the foundation of credit-based flow control, which creates a lossless environment and supports huge traffic loads.
During the link initialization, the FC ports negotiate the electrical parameters, the speeds, and the serialization, but they also exchange information about their capabilities, such as the size of the memory buffers, which are used to store the FC frames received at the port. Once the link is established, each FC port knows the size of the BB_Credits of the port to which it is connected. There are two types of BB_Credits: the BB_Credit_TX and the BB_Credit_RX. The former defines how many FC frames can be stored at the port for transmission, and the latter defines how many frames the port can receive and store before processing. This means that the FC port knows the maximum number of FC frames the connected FC port can receive and store before it can start processing them.
When FC frames have to be transmitted, the transmitting port (Tx_Port), which already knows the size of the BB_Credit_RX at the receiving side (the Rx_Port), will have a variable that has this value. If the other port has reported a BB_Credit_RX size of 64, the Tx_Port will store that value in the BB_Credit_RX variable. The Tx_Port knows that the Rx_Port has space to store four FC frames without the risk of them being dropped due to a busy port that cannot process them.
Now that the Tx_Port knows how many FC frames to send to the Rx_Port, to be able to control the number of frames sent, it also sets another variable, BB_Credit_CNT, to zero.
The Tx_Port is ready to start sending frames, but it cannot because transmission in the FCP is controlled by the receiving side. This means that the Rx_Port must notify the Tx_Port that it is ready to receive the frames. This happens when the Rx_Port sends a receive ready notification (R_RDY) to the transmitting port. The R_RDY notification means that the BB_Credit buffers at the Rx_Port are empty, and it is ready to receive and process the FC frames. Figure 11-19 illustrates the credit-based flow control.
Figure 11-19 Credit-Based Flow Control
Once the Tx_Port receives the R_RDY notification, it starts to send frames. With every frame sent, the BB_Credit_CNT value is incremented by one until it reaches the value of BB_Credit_RX minus one. Then the Tx_Port stops sending FC frames, resets the value of the BB_Credit_CNT, and waits for the R_RDY to start transmitting again.
Just as the FCP secures the communication at many levels (like with the login processes, where there is a login between an F_Port and an N_Port), in the same way the flow control is not limited only between two directly connected ports, such as the connections between an N_Port and an F_Port, or between two E_Ports, but extends on the next level of communication—between the two N_Ports of the initiator and the target. At that level, there is another variable called the EE_Credits, which is the end-to-end flow control credits. In the same manner as the buffer-to-buffer flow control, the transmitting N_Port also sets a variable called EE_Credit_CNT initially to zero and then increments it during the transmission.
Additionally, the Cisco MDS switches implement an architecture based on using internal buffer queues and a central arbiter—a component that monitors and controls the switching of the FC frames internally in the switch. It allows an FC frame to be sent to the crossbar fabric in order to be switched in hardware to the egress switch port only when it has received an acknowledgment that the port has enough free resources to store and process the frame.
As you can see, there are various levels of flow control that make the FC communication secure and reliable.