Single root I/O virtualization (SR-IOV) is an extension to the PCI Express (PCIe) specification that allows multiple operating systems to simultaneously share a PCIe adapter with little or no runtime involvement from a hypervisor or other virtualization intermediary.

SR-IOV capable adapters can be used in two modes.

Dedicated mode: This is the traditional mode, where the I/O adapter is assigned to a partition and ports are not shared. The partition owns the whole adapter and manages it from a single operating system.
SR-IOV shared mode: In shared mode, the adapter is assigned to the Power Hypervisor firmware. In this mode, the adapter can be shared by multiple operating systems at the same time. Each operating system accesses its share of the adapter using a virtual function device driver.

In SR-IOV shared mode, adapters divide their host interface into multiple virtual functions. On IBM Power servers, these virtual functions are implemented as logical ports, with each logical port linked to a specific physical port on the adapter.

On IBM Power servers running IBM AIX, SR-IOV simplifies network virtualization by allowing logical partitioning of adapter resources into virtual functions. This article provides a comprehensive set of commands to configure, manage, and optimize SR-IOV-capable network adapters from the Hardware Management Console (HMC), ensuring high performance and flexibility in virtualized environments.

Command-line support for the SR-IOV capable adapters

Note: The commands in this article must be run from the Hardware Management Console (HMC).

Listing 1: List the SR-IOV capable adapters

To list the SR-IOV capable adapters, use the following command:

lshwres -m <Managed System> -r sriov --rsubtype adapter

Limitation: If the SR-IOV capable adapter is not placed in the SR-IOV capable slot of the IBM Power server, this command will not list them.

Listing 2: Convert the adapter mode from dedicated to SR-IOV

To convert the adapter mode from dedicated to SR-IOV, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype adapter -o a -a “slot_id=<slot_id>”

When the adapter operates in the dedicated mode, its sriov_status attribute is null. The initial conversion to SR-IOV mode triggers a firmware upgrade, which may take several seconds to complete. During this conversion process, the adapter's state transitions through the following stages: null, missing, initializing, and running.

Note: The value of the slot_id attribute is provided in the Listing 1 command output.

Listing 3: Convert the adapter mode from SR-IOV to dedicated

To convert the adapter mode back from SR-IOV to dedicated, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype adapter -o r -a “slot_id=<slot_id>”

When the adapter operates in the SR-IOV mode, its SR-IOV status value is indicated as running. The adapter undergoes a state transition during conversion: from running to powering_off and finally to a null state.

Note: The value of the slot_id attribute is provided in the Listing 1 command output.

Listing 4: List the physical ports

To list the physical ports and their attributes for an adapter, use the following command:

lshwres -m <Managed System> -r sriov --rsubtype physport --level <ethc/roce> --filter “adapter_ids=<Adapter ID>”

ethc - This attribute applies to adapters with speeds of 10 Gbps and higher. roce - This attribute is used for a specific adapter family, namely NVIDIA.

Note: The Adapter ID value is listed in the Listing 1 command output.

Listing 5: Obtain the physical port ID

To retrieve the physical port ID, use the following command:

lshwres -m <Managed System> -r sriov --rsubtype physport --level <ethc/roce> | grep <Physical Slot Location>

ethc - This attribute applies to adapters with speeds of 10 Gbps and higher. roce - This attribute is used for a specific adapter family, namely NVIDIA.

Listing 6: List the logical ports

To list the logical ports, use the following command:

lshwres -m <Managed System> -r sriov --rsubtype logport --level <eth/roce> --filter “adapter_ids=<Adapter ID>”

eth – This attribute applies to list the Ethernet virtual functions. roce – This attribute applies to list Remote Direct Memory Access (RDMA) supported virtual functions.

Note: The Adapter ID value is listed in the Listing 1 command output.

Create virtual functions

This section explains the different commands that you can use to create virtual functions.

Listing 7: Create a dynamic virtual function

To create a dynamic virtual function, use the following command:

A dynamic virtual function is assigned dynamically to a partition when it is running.

chhwres -m <Managed System> -r sriov --rsubtype logport -p <Partition Name> -o a -a “adapter_id=<Adapter ID>, phys_port_id=<Physical Port ID>, logical_port_type=<eth/roce>”

Listing 8: Create a static virtual function

To create a static virtual function, use the following command:

chsyscfg -m <Managed System> -r prof -i “name=<Partition Profile Name>, lpar_name=<Partition Name>, sriov_eth_logical_ports+=adapter_id=<Adapter ID>:phys_port_id=<Physical Port ID>”

When a static virtual function is created, the user must shut down and activate the partition in order to configure the virtual function on the partition profile.

Note: If the command in Listing 8 throws an error while overwriting the profile, use the –-force option at the end of the command.

Listing 9: Create a promiscuous virtual function

To create a promiscuous virtual function, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype logport -p <Partition Name> -o a -a “adapter_id=<Adapter ID>,phys_port_id=<Physical Port ID>,logical_port_type=<eth/roce>, promisc_mode=1”

Only one promiscuous virtual function can be created per physical port. Currently, the promiscuous mode is supported only on eth virtual function but not on the roce virtual function.

Removing logical virtual functions

This section describes the commands that enable you to remove logical virtual functions.

Listing 10: Delete a dynamic virtual function

To delete a dynamic virtual function, use the following command after removing its Ethernet ports from the logical partition.

chhwres -m <Managed System> -r sriov --rsubtype logport -o r --id <Partition ID> -a “adapter_id=<Adapter ID>, logical_port_id=<Logical Port ID>”

If the Ethernet ports are not removed, this delete command will result in an error.

Note: You can find the Logical Port ID value in the Listing 6 command output.

Listing 11: Delete a static virtual function

To delete a static virtual function, use the following command:

chsyscfg -m <Managed System> -r prof -i “name=<Partition Profile Name>, lpar_name=<Partition Name>, sriov_eth_logical_ports-=config_id=<Config ID>”

After running the command, shut down and then activate the partition for the changes to take effect and the virtual function to be deleted.

Note: You can find the Config ID value in the Listing 6 command output.

If the command in Listing 11 throws an error while overwriting the profile, add the –-force option at the end of the command.

Configure physical function attributes

This section explains the commands that you can use to configure the physical function attributes.

Listing 12: Enable jumbo frames

To enable jumbo frames on a physical function, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype physport -o s -a “adapter_id=<Adapter ID>, phys_port_id=<Physical Port ID>, max_recv_packet_size=9000”

It is essential to enable jumbo frames on a physical function before creating a virtual function. Although jumbo frames are enabled on the physical function, users must also enable the jumbo_frames attribute on the virtual function assigned to the partition. The attribute used to enable jumbo frames on the physical function is max_recv_packet_size.

Listing 13: Enable flow control

To enable flow control on the physical function, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype physport -o s -a “adapter_id=<Adapter ID>, phys_port_id=<Physical Port ID>, recv_flow_control=<0/1>, trans_flow_control=<0/1>”

The recv_flow_control and trans_flow_control attributes are used to enable or disable flow control. It is essential to set both these attributes to the same value, which is either enabled (1) or disabled (0).

Listing 14: Configure the speed of physical function

To configure the speed of a physical function, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype physport -o s -a “adapter_id=<Adapter ID>, phys_port_id=<Physical Port ID>, conn_speed=<Connection Speed>”

Before running the command, refer to the adapter's specification document to verify the supported speeds, as they vary across different adapters. Users can set the conn_speed attribute to one of the following values: auto, 1000, or 10000.

Configure virtual function attributes

This section explains the commands that you can use to configure the physical function attributes.

Listing 15: Configure MAC ACL

To configure media access control (MAC) access control list (ACL) on the virtual function, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype logport -p <Partition Name> -o a -a “adapter_id=<Adapter ID>, phys_port_id=<Physical Port ID>, logical_port_type=<eth/roce>, allowed_os_mac_addrs=<\”MAC1,MAC2,..\”>”

MAC ACL allows only specific MAC addresses to be configured on the virtual function, including the default MAC address. The allowed_os_mac_addrs attribute is used to set this ACL.

Restriction: The MAC list is limited to four MAC addresses.

Listing 16: Configure VLAN ACL

To configure a VLAN ACL on the virtual function, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype logport -p <Partition Name> -o a -a "adapter_id=<Adapter ID>,phys_port_id=<Physical Port ID>,logical_port_type=<eth/roce>,allowed_vlan_ids=<\"VLAN1,VLAN2,..\">”

VLAN ACLs enable specific VLANs to be configured on the virtual function. The allowed_vlan_ids attribute is used to set this ACL.

VLAN restrictions: The VLAN list is limited to 20 VLAN IDs.

Listing 17: Configure port VLAN ID

To configure the port VLAN ID, use the following command:

chhwres -m <Managed System> -r sriov --rsubtype logport -p <Partition Name> -o a -a "adapter_id=<Adapter ID>,phys_port_id=<Physical Port ID>,logical_port_type=<eth/roce>, port_vlan_id=<Port VLAN>, allowed_vlan_ids=none”

By default, the SR-IOV logical port is created with a port VLAN ID of "0". To create a virtual function with a non-zero port VLAN ID, users must set a non-zero value for the port_vlan_id attribute.

Limitation: For adapters with specific feature codes (EC2R, EC2S, EC2T, EC2U, EC3L, EC3M, EC66, EC67, EC75, EC76, EC71, EC72, EN24, EN26, EC85, EC86), the Logical Port VLAN ID Restrictions must be set to Deny All when a non-zero Port VLAN ID is configured. This means that the operating system VLAN tagging is not supported (frames are dropped) if a non-zero port VLAN ID is configured. To use VLAN tagging with multiple VLAN IDs, set the logical port VLAN ID to zero and configure the operating system to apply the VLAN tags.

Configure RDMA-enabled RDMA over Converged Ethernet (RoCE) Virtual Functions

Listing 18: Enable RDMA on a physical function

To create an RDMA-enabled RoCE virtual function, you must first enable RDMA support on the physical function using the following command:

chhwres -r sriov -m <Managed System> --rsubtype physport -o s -a "adapter_id=<Adapter ID>,phys_port_id=<Physical Port ID>,veb_mode=1,max_logical_ports_option=2"

The max_logical_ports_option must be set to 2, which is 4 Ethernet and 4 RoCE (as shown in the following screen), to enable RoCE on the physical port.

alt

Listing 19: Create an RDMA-supported virtual function

After enabling RDMA support on the physical function, you can create an RDMA-supported virtual function using the following command:

chhwres -m <Managed System> -r sriov --rsubtype logport -p <Partition Name> -o a -a "adapter_id=<Adapter ID>,phys_port_id=<Physical Port ID>,logical_port_type=roce"

RDMA over SR-IOV is supported only on the adapters mentioned in the following table.

Feature codes	Description
EC3L and EC3M	PCIe3 2-port 100 GbE RoCE QSFP28 PCIe 3.0 x16 adapter
EC2R and EC2S	PCIe3 2-port 10 Gb NIC and ROCE SR/Cu adapter
EC2T and EC2U	PCIe3 2-port 25/10/1 Gb NIC and ROCE SFP28 adapter
EC66 and EC67	PCIe4 2-port 100 GbE RoCE x16 adapter
EC75 and EC76	PCIe4 2-port 100 GbE RoCE x16 adapter
EC71 and EC72	PCIe4 2-port 25/10/1 Gb RoCE SFP28 adapter
EN24 and EN26	PCIe4 x16 4-port 25/10/1 GbE RoCE SFP28 adapter
EC85 and EC86	PCIe5 2-port 200/100 GbE RoCE Adapter x16

Summary

This article provided a detailed guide to configure and manage single root I/O virtualization (SR-IOV) on AIX systems. It covered the necessary commands and steps to set up SR-IOV-capable network adapters, create and manage virtual functions, and optimize performance. This guide is designed to help users effectively manage SR-IOV on AIX and improve their virtualized network infrastructure.

A guide to single root I/O virtualization (SR-IOV) commands on IBM AIX for network adapters

A guide to single root I/O virtualization (SR-IOV) commands on IBM AIX for network adapters