FCoE Architecture
The FCoE is a transport protocol defined by the American National Standards Institute (ANSI) T11 committee. It has the needed functionality and enhancements to create a lossless environment and to carry the FC frames encapsulated into Ethernet frames. In the FCoE protocol, the lower layers of the Fibre Channel Protocol are replaced with unified fabric I/O consolidation over Ethernet. These are the layers that take care of the encapsulation and de-encapsulation as well as the lossless transmission over Ethernet. The upper-layer Fibre Channel Protocol services, such as domain IDs, Fabric Shortest Path First (FSPF), Fibre Channel Name Server (FCNS), fabric login (FLOGI), zoning, and so on, stay the same as in the Fibre Channel world, because it is the Fibre Channel protocol that operates at these layers. Figure 15-3 shows which layers are changed.
Figure 15-3 FCoE Protocol Layers
In the Fibre Channel Protocol operation, the FC frame is created in the hardware of the HBA, which is the FC-2 layer responsible for the framing and flow control. Afterward, the FC frame is passed to the FC-1 layer to be encoded with the needed serialization in place and transmitted over the native FC link. In the case of the Unified Fabric, there is no native FC link but rather an Ethernet link. That’s why after the FC frame is formed in the HBA part of the converged adapter, it is passed to the FCoE Logical End Point (LEP). The FCoE LEP is a new component that is responsible for taking the FC frame and encapsulating it in an Ethernet frame, with all the needed information required by the Data Center Bridging Exchange (DCBX) enhancements for the secure transmission of the new Ethernet frame, which is contained inside an FC frame. This new Ethernet frame has some special characteristics. Because the standard FC frame is up to 2112 bytes in size, and the standard Ethernet frame is usually 1500 bytes, without any overhead from encryption or something similar, the maximum transmission unit (MTU) for the FCoE communication must either be set to the default for the FCoE protocol (2240 bytes) or, if the switch does not allow defining such a size, just be allowed to use jumbo frames, which is sufficient enough (see Figure 15-4).
Figure 15-4 FC Frame Encapsulation in FCoE
Figure 15-4 shows that the FC frame is inside the Ethernet frame, and there are not any changes made to the original FC frame. It is encapsulated just by adding the needed FCoE header and the Ethernet header. No manipulations are allowed to be performed on the FC frame, as this will invalidate it. This also means that the FC header still has the source and destination FCIDs needed for the FC protocol communication. Do not forget that from the perspective of the Fibre Channel Protocol, the FCoE protocol is just a different cable/transport mechanism. The FCoE header contains control info. The Ethernet header contains important information such as the Ethertype, as the value for the FCoE protocol is 0x8906. This value notifies the switch that inside this Ethernet frame is a Fibre Channel Protocol frame. Additionally, in the Ethernet header is the 802.1Q VLAN tag. This is important because it defines to which VLAN the FCoE traffic will belong. In Cisco’s implementation of the FCoE, dedicated VLANs are used for its traffic, which allows the needed priority flow control and no-drop policies to be applied. And last but not least, in the Ethernet header are the source and destination MAC addresses, as the FCoE frame is basically an Ethernet frame that will be transported over an Ethernet infrastructure. The formation of the MAC address and how it maps to the FCID in the servers’ converged adapter is a function of the FCoE Initialization Protocol (FIP), which takes care of the negotiations before any FCoE communication can happen.
In the FCoE communication are two roles (or participating elements): the FCoE Ethernet nodes (ENodes) and the Fibre Channel Forwarders (FCFs).
An FCF is an Ethernet switch that also supports the FCoE protocol. When a switch supports the FCoE protocol and is also an FCF, it means that this switch is composed of two switches: an Ethernet switch with the Ethernet physical ports and a Fibre Channel switch (or a component that can run and process the Fibre Channel protocol frames). This means that this component runs all the needed Fibre Channel Protocol services, just like any other physical standalone Fibre Channel switch. That’s why the FCF, shown in Figure 15-5, can process the Fibre Channel Protocol logins, services, and frames.
When an FCoE frame enters the FCF, it is processed as an Ethernet frame, as it enters through a physical Ethernet port. Based on the Ethertype, the switch knows that it is an FCoE frame, and it must be sent to the FCoE LEP, where it is de-encapsulated. The remaining FC frame is then processed based on the rules of the Fibre Channel Protocol. Once it is processed, either it will be sent as a native FC frame through an egress native FC port, if the FCF also has native FC connectivity, or it will be encapsulated again in an Ethernet FCoE frame if it is supposed to leave through the Ethernet egress port.
Figure 15-5 Fibre Channel Forwarder
The servers are equipped with converged networks adapters (CNAs) in the Cisco Unified Fabric. This allows them to be physically connected to an Ethernet port of a switch with native Ethernet connectivity but over that link to carry both their Ethernet communication as well as their storage communication using the FCoE protocol, in case the switch is FCoE capable. The hardware of the CNA, shown in Figure 15-6, is very different from the traditional NICs and HBAs. For external physical connectivity, the CNA uses 10Gbps or faster Ethernet ports, but inside, facing the server, separate NICs and HBAs are built in. The OS of the server communicates with the CNA and sees the separate HBAs and NICs through the PCIe bus.
Figure 15-6 Converged Network Adapter
The NICs in the CNA natively use the Ethernet physical egress ports. The situation is more complex with the Fibre Channel Protocol communication.
The HBAs in the CNA cannot communicate directly using the physical Ethernet ports. That’s why there is a specialized ASIC, or silicone, that performs the function of the FCoE LEP. The HBAs are acting as native FC ports, but as there are no physical FC ports, the HBAs are presented as virtual Fibre Channel (VFC) ports. Through the ASIC, the VFCs use the physical external ports for FCoE communication. By the way, it is the same on the FCF side—because the FC switching component is behind the physical Ethernet ports, the FC ports are virtual (VFCs) and act just like the physical FC ports. Therefore, you have the standard Fibre Channel Protocol communication with the Fibre Channel ports, with the appropriate port roles assigned, with the only difference that these are virtual ports, and the roles are also presented as virtual to provide the information that these roles are assigned to virtual FC interfaces. As with the Fibre Channel Protocol port modes, the virtual ones are as follows (see Figure 15-7):
Figure 15-7 VFC Ports and Virtual Port Roles in FCoE
- Virtual Node ports (VN): The VFC on the CAN of the server.
- Virtual Fabric port (VF): The FCF VFC ports to the VN ports are connected.
- Virtual Expansion ports (VE): The FCF VFC connect to another FCF VFC.
- Virtual Node Proxy port (VNP): When the FCF operates as NPV edge switch (that is, when it is not running the FC Protocol services).
The VN port communicates with the VF port at the side of the switch in the same way as in a SAN infrastructure the N_Port will connect and communicate with the F_Port on the FC switch. The difference is that because there is no native FC physical connectivity, the negotiations and the login processes will be performed by the FCoE Initialization Protocol (FIP).
The VE ports connect multiple FCoE switches over the physical Ethernet connectivity, just like with the native FC communication.
Therefore, the ENodes can be defined as a combination of the VFC, operating in VN mode, and the FCoE LEP on the CNA. In the same way, the FCF is the combination of the FCoE LEP on the switch and the FC protocol component, represented by VFC ports, operating in VF or VE mode, and all the Fibre Channel protocol services.
As the VFC ports are Fibre Channel ports, when they communicate, they use FCIDs. However, because the Fibre Channel frames are encapsulated in Ethernet frames to be transported, as the Ethernet frames use MAC addresses, this brings up the issue of what MAC address to use for the FCoE communication of a specific VFC that will allow a direct mapping between the FCID and the MAC address to be created in the FCoE LEP. As the MAC addresses are 48 bits and the FCIDs are 24 bits, a direct mapping is not possible. That’s why there needs to be a component added to the value of the FCID. This is called the Fibre Channel MAC Address Prefix (FC-MAP), which has a size of 24 bits, and it represents the first part of the MAC address. The second part is the 24-bit FCID of the VFC. In this way, there is a unique 48-bit MAC address used for the communication of this specific VFC. This also means that each VFC on the CAN will have its own unique MAC address, as each will have its own unique FCID. The FC-MAP is a value set on the FCF switch, and the default value is 0E.FC.00. As you’ll remember from the previous chapters, the FCID in the Fibre Channel Protocol communication is created and assigned by the Fibre Channel switch, after a successful fabric login from the end node. With the FCoE protocol, the FCID is also created on the FC switch; in this case, it’s the FCF switch. This means that both values that form the MAC address for the FCoE communication are provided by the FCF switch, or the fabric. This method of creating and assigning a MAC address is called a Fabric Provided MAC Address (FPMA) and shown in Figure 15-8. Based on the FC-BB-5 definition, there is a range of 256 FC-MAP values that can be used. In environments where there might be overlapping FCIDs, or for other purposes, administrators can create and use up to 256 different pools of MAC addresses.
Figure 15-8 FPMA MAC Address for the FCoE
With the FPMA approach and the FCoE protocol there is one challenge. For the VFC to log in to the fabric, it needs to be capable of communicating over the Ethernet infrastructure using the FCoE protocol. However, for this to happen, a MAC address is needed, and the MAC address can be created by the fabric after a successful fabric login. Therefore, without an FCID there is no a MAC address, and without a MAC address there is no FCID. To solve this challenge, and others, there is one additional protocol that communicates before the FCoE protocol, and that is the FCoE Initialization Protocol.