Cisco Fibre Channel NPV Mode
The Fibre Channel NPV is another SAN virtualization option available in the data center. It solves the challenge due to the limit on the domain IDs that can be used in a SAN fabric. The NPV is a mode in which a Fibre Channel switch operates. In this mode, the Fibre Channel switch does not run most of the Fibre Channel protocol services. It does not run the fabric login server, the Fibre Channel Name Server (FCNS) service, Fibre Channel Shortest Path First (FSPF), zoning, and so on. Because of that, the switch, which operates in NPV mode, does not actually log in to the switched fabric but appears to it as an end device, which allows for communication of multiple N_Ports through the physical Fibre Channel port that connects to the Fibre Channel switch, which operates in Fibre Channel switching mode. A bit of terminology is needed here to make the explanation clear. The switch, which operates in NPV mode, is called an NPV Edge switch. It connects to a switch that operates in Fibre Channel switching mode, which means that it is part of the switched fabric. This switch is an NPV Core switch. It runs all the Fibre Channel protocol services and is part of the switched fabric. As such, it consumes a domain ID for each of the VSANs that exist on it. It also has the NPIV feature enabled in order to service the communication of the multiple N_Ports that comes from the NPV edge switch.
So, the NPV Edge switch is accepted as an end node, as it does not run the needed Fibre Channel protocol services and does not log in to the fabric. However, from the perspective of the initiators and the targets, the real end nodes that connect to it, it is still a Fibre Channel switch. There is still the communication between the N_Ports of the servers and the storage systems as well as the F_Ports (the Fibre Channel ports on the NPV Edge switch) to which the nodes connect. Figure 13-4 shows the NPV Core-Edge topology.
Figure 13-4 NPV Edge and Core Switches
The Fibre Channel ports on the NPV Edge switch facing the nodes are operating as F_Ports, but the switch itself, as it does not run the needed services, and especially the FLOGI and the FCNS services, can’t support the FLOGI requests from the N_Ports. However, the NPV Core switch, as it is running the needed services and is part of the switched fabric, can perform the FLOGI. It can also generate and assign the needed FCIDs, as it has the NPIV enabled. The link between the end nodes and the NPV Core switch happens as the F_Ports on the NPV Edge switch are mapped to the Fibre Channel ports that connect to the NPV Core switch. The NPV Edge switch does not switch the Fibre Channel frames from the nodes to the core switch, but it proxies them to the Fibre Channel port connected to the NPV Core switch. That’s why this port operates as the NP_Port, which stands for Node Proxy Port. And why do you need to have the NPIV feature enabled on the NPV Core switch? Because, just like with the virtualized server connected to the Fibre Channel switch, the NPV Edge switch will present multiple N_Ports communicating through the NP_Port. In this way, the NPV Core switch will solve the same challenge and will be capable of assigning multiple different FCIDs. The NPV port roles are shown in Figure 13-5.
Figure 13-5 NPV Ports
The NPV Edge switches support the F_ and VF_Ports for the connectivity of the end nodes. The latter are used when there is no native Fibre Channel physical connectivity, and a protocol such as the FCoE is used to carry the Fibre Channel frames over Ethernet connectivity. And because the FCoE protocol was mentioned, it is important to make another note here. Sometimes there is a bit of confusion because of the impression that the terms FCoE and NPV are interchangeable. They are not. NPV refers to the mode in which the switch operates from the perspective of the Fibre Channel protocol. FCoE is a way of encapsulating the Fibre Channel frames in Ethernet frames to be capable of transmitting them over an Ethernet network. It’s important to realize that the FCoE switch might operate in Fibre Channel switching mode or in NPV mode, if supported. As a quick summary, the NPV Edge switch has these characteristics:
- It appears as an end node (initiator/target) to the NPV Core switch.
- It does not consume SAN resources, such as domain IDs.
- It does not participate in the switched fabric.
- It does not run the main Fibre Channel protocol services.
- The NPV Core switch provides the needed Fibre Channel protocol services.
- It appears as a switched fabric switch to the initiators and targets connected to it; that is, it appears as F_Ports.
For the purposes of keeping the explanation simple up until now, the link between the NPV Edge and Core switches was explained as one between an NP_Port and an F_Port. In the data center it is very rare to have a single link between two network devices, as this represents a bottleneck and also a single point for a failure. That’s why usually there are multiple links between the NPV Edge and Core switches. Depending on the NPV Edge device, there are different approaches for how the F_Ports are mapped to the available NP_Ports. Usually this happens automatically, as the switch tries to map every new F_Port to the NP_Port, which is either the next one, using a round-robin mechanism, or the NP_Port, which is least busy.
There is also the option to use manual mapping through a traffic map, which is a component from the configuration. This allows the administrator to manually configure which F_Ports will communicate upstream through which NP_Ports.
The NPV mode of operation is supported on the following Cisco Data Center switches:
- Cisco MDS 9396T 32Gbps 96-Port Fibre Channel Switch
- Cisco MDS 9250i Multiservice Fabric Switch
- Cisco MDS 9132T 32Gbps 32-Port Fibre Channel Switch
- Cisco MDS 9148 Multilayer Fabric Switch
- Cisco MDS 9148S 16G Multilayer Fabric Switch
- Cisco MDS 9148T 32Gbps 48-Port Fibre Channel Switch
- Cisco Nexus 9300-GX Series
- Cisco Nexus 9300-FX Series (not the 24-port FX and EX)
- Cisco Nexus 9300-FX2 Series
- Cisco UCS 6332-16UP Fabric Interconnect
- Cisco UCS 6324 Fabric Interconnect
- Cisco UCS 6454 Fabric Interconnect
- Cisco UCS 64108 Fabric Interconnect
Like the configuration of the NPV Edge switch, this is enabled as a feature of NX-OS. However, do not forget that by enabling this “feature,” you actually change the mode, and the switch will need to be reloaded to start operating in the new mode. That’s why you should first check whether or not the feature is enabled (see Example 13-4).
Example 13-4 Verifying the NPV Feature State
mds-9100-a(config)#
sh feature
|
include npv
Feature Name Instance State
——————– ——– —–
npv 1 disabled
mds-9100-a(config)#
The second step is to enable the NPV (see Example 13-5). This will be disruptive, as the NX-OS will warn you.
Example 13-5 Enabling the NPV Feature
mds-9100-a(config)#
feature npv
Verify that boot variables are set and the changes are saved. Changing to npv mode
erases the current configuration and reboots the switch in npv mode. Do you want
to continue? (y/n):
After you confirm that you want to continue, the switch will reload itself, which takes a couple of minutes.
On the upstream Fibre Channel switch (the NPV Core switch), the NPIV feature needs to be enabled (see Example 13-6).
Example 13-6 Enabling NPIV on the NPV Core Switch
mds-9200-a#
sh feature
|
i npiv
npiv 1 disabled
mds-9200-a#
conf t
Enter configuration commands, one per line. End with CNTL/Z.
mds-9200-a(config)#
feature npiv
mds-9200-a(config)#
sh feature
|
i npiv
npiv 1 enabled
mds-9200-a(config)#
The next step on the NPV Edge switch is to configure the NP ports (see Example 13-7).
Example 13-7 Configuring the NP Ports on the NPV Edge Switch
mds-9100-a(config)#
int fc1/2-3
mds-9100-a(config-if)# shutdown
mds-9100-a(config-if)#
switchport mode NP
mds-9100-a(config-if)# no shut
mds-9100-a(config-if)#
You also need to configure the ports to which connect the end node in F mode (see Example 13-8).
Example 13-8 End-Node-Facing Ports on the NPV Edge Switch
Click here to view code image
mds-9100-a(config)#
int fc1/1
mds-9100-a(config-if)# shut
mds-9100-a(config-if)# switchport mode F
mds-9100-a(config-if)# no shut
mds-9100-a(config-if)#
exit
On the NPV Core switch, the ports that connect to the NP ports have to be configured as F_Ports (see Example 13-9).
Example 13-9 Configuring the NPV Core Switch Ports Connecting to the NPV Edge Switch
mds-9200-a(config)#
sh npiv status
NPIV is enabled
mds-9200-a(config)#
int fc1/2-3
mds-9200-a(config-if)#
shutdown
mds-9200-a(config-if)#
switchport mode F
mds-9200-a(config-if)#
no shut
mds-9200-a(config-if)#
Then, the ports will come up on the Core switch, as shown in Example 13-10.
Example 13-10 NPV Core Switch Ports Status
mds-9200-a(config-if)#
show interface brief
——————————————————————————–
Interface Vsan Admin Admin Status SFP Oper Oper Port Logical
Mode Trunk Mode Speed Channel Type
Mode (Gbps)
——————————————————————————–
fc1/1 1 auto on down swl — — — —
fc1/2 1 F on up swl F 16 — edge
fc1/3 1 F on up swl F 16 — edge
The same will happen on the NPV Edge switch, as shown in Example 13-11.
Example 13-11 NPV Edge Switch Ports Status
mds-9100-a(config-if)#
show interface brief
——————————————————————————–
Interface Vsan Admin Admin Status SFP Oper Oper Port Logical
Mode Trunk Mode Speed Channel Type
Mode (Gbps)
——————————————————————————–
fc1/1 1 F off up swl F 16 — edge
fc1/2 1 NP off up swl NP 16 — core
fc1/3 1 NP off up swl NP 16 — core
The output of the show interface brief command shows that the interface connected to the initiator operates in F mode and is up and that the interfaces connected to the NPV Core switch are operating in NP mode and are also up.
By using the command show npv status on the Edge switch, you can check the mode of operation for the external interfaces and the initiator-facing ones (see Example 13-12).
Example 13-12 NPV Status on the Edge Switch
mds-9100-a(config-if)#
sh npv status
npiv is disabled
disruptive load balancing is disabled
External Interfaces:
====================
Interface: fc1/2, VSAN: 1, FCID: 0x0a0600, State: Up
Interface: fc1/3, VSAN: 1, FCID: 0x0a0500, State: Up
Number of External Interfaces: 2
Server Interfaces:
==================
Interface: fc1/1, VSAN: 1, State: Up
Number of Server Interfaces: 1
On the NPV core switch, you can check the FLOGI and FCNS databases to make sure the NPV Edge switch logged in to the switched fabric as an NPV device and also that the server connected to it was capable to successfully log in to the fabric through the NPV Core switch (see Example 13-13).
Example 13-13 Verify Successful Fabric Login on the NPV Core Switch
mds-9200-a(config-if)#
show flogi database
——————————————————————————–
INTERFACE VSAN FCID PORT NAME NODE NAME
——————————————————————————–
fc1/2 1 0x0a0600 20:02:00:de:fb:ce:d9:00 20:01:00:de:fb:ce:d9:01
fc1/2 1 0x0a0601 21:00:f4:e9:d4:58:d7:88 20:00:f4:e9:d4:58:d7:88
fc1/3 1 0x0a0500 20:03:00:de:fb:ce:d9:00 20:01:00:de:fb:ce:d9:01
Total number of flogi = 3.
mds-9200-a(config-if)# sh fcns database
VSAN 1:
————————————————————————–
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
————————————————————————–
0x0a0500 N 20:03:00:de:fb:ce:d9:00 (Cisco) npv
0x0a0600 N 20:02:00:de:fb:ce:d9:00 (Cisco) npv
0x0a0601 N 21:00:f4:e9:d4:58:d7:88 scsi-fcp:init
Total number of entries = 3
mds-9200-a(config-if)#
Back on the Edge switch, you can see that the show npv command has a couple of useful options (see Example 13-14).
Example 13-14 The show npv Command Options
mds-9100-a(config-if)#
show npv
?
*** No matching command found in current mode, matching in (exec) mode ***
external-interface-usage Show external interface usage by server interfaces
flogi-table Show information about FLOGI sessions
internal Show internal NPV information
status Show NPV status
traffic-map Show information about Traffic Map
The flogi-table option provides information for the end nodes that reached the Core switches through the Edge switch in order to log in to the fabric. It also shows the mapping between the F_Port and the NP_Port (see Example 13-15).
Example 13-15 Verifying End Nodes FLOGI on the Edge Switch
mds-9100-a(config-if)#
sh npv flogi-table
——————————————————————————–
SERVER EXTERNAL
INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE
——————————————————————————–
fc1/1 1 0x0a0601 21:00:f4:e9:d4:58:d7:88 20:00:f4:e9:d4:58:d7:88 fc1/2
Total number of flogi = 1.
mds-9100-a(config-if)#
mds-9100-a(config-if)#
sh npv external-interface-usage ?
*** No matching command found in current mode, matching in (exec) mode ***
> Redirect it to a file
>> Redirect it to a file in append mode
server-interface Show external interface usage by a server interface
| Pipe command output to filter
The option external-interface-usage shows the mapping between the server interface and the upstream interface. It also allows you to search based on a specific server interface (that is, F_Port), as shown in Example 13-16.
Example 13-16 Server-Facing Port to Egress Port Mapping on the Edge Switch
mds-9100-a(config-if)#
sh npv external-interface-usage
NPV Traffic Usage Information:
—————————————-
Server-If External-If
—————————————-
fc1/1 fc1/2
—————————————-
As you can see, there are enough tools to provide the needed information for the status and the operation of both the Core and Edge NPV switches, as well as the communication of the connected end nodes. Here are some of the benefits of NPV mode:
- Increased scalability and flexibility, as no domain IDs are consumed.
- Simplified configuration on the NPV Edge switches, as they do not participate in the SAN fabric. The configuration is limited to configuring the interfaces, the VSANs, and traffic maps, if needed.
- Simplified troubleshooting, as the Fibre Channel protocol-related configuration is created only on the NPV Core switches and the other switches operating in Fibre Channel switching mode.
- Simplified troubleshooting on the NPV Edge switches, as less configuration is needed.
- The NPV Edge switches utilize less resources, which allows for better and more reliable operation. Especially if these are Ethernet switches on which you run FCoE, the resources will be freed to support other services.