Mellanox 网卡使用SR-IOV

一、硬件环境准备

物理服务器为：浪潮

网卡型号：Mellanox Technologies MT27800

1、进入BIOS

浪潮服务器为例，开机画面中按“DEL”键；

 

等待系统进入BIOS 后再进行下一步；

成功进入BIOS的画面

 

 

2、主板开启SR-IOV

移动光标到“Advanced”标签页面，再选择“PCI Subsystem Settings”，选中按回车；

    再将SR-IOV支持开关设置为“Enabled”;

 

 

处理器开启虚拟化支持；

将光标移动到“Processor”标签页，再选中“Processor Configration”；

将“VMX”与“SMX”的值设置为“Enabled”；

 

 

再从”Processor ”下找到“IIO Configuration”，进入后将“VT-d”功能打开；

 

 

操作系统开启iommu

（Centos8.1 为例）

[root@localhost ~]# vim /etc/default/grub

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”

GRUB_DEFAULT=saved

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL_OUTPUT=”console”

GRUB_CMDLINE_LINUX=”crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on iommu=pt”

GRUB_DISABLE_RECOVERY=”true”

GRUB_ENABLE_BLSCFG=true

[root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

开启完成后需要重启机器；

驱动环境准备

1、取得下载合适的安装包进行驱动的安装

这里以MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz 为例：

2、解压

[root@localhost ~]# tar xzvf MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz

1、查看readme 安装相应的依赖包：

[root@localhost ~]#yum install perl pciutils python gcc-gfortran libxml2-python tcsh libnl.i686 libnl expat glib2 tcl libstdc++ bc tk gtk2 atk cairo numactl pkgconfig ethtool

2、驱动安装

[root@localhost ~]#./mlnxofedinstall

    3、安装完成后重启机器；

3、配置opensm开启虚拟化

[root@localhost docs]# vim /etc/opensm/opensm.conf

添加内容后保存退出：

virt_enabled 2

参数的解释：

parameter has the following configuration options:

        0: Ignore Virtualizations – No virtualization support

        1: Disable Virtualization – Disable virtualization on all Virtualization supporting ports

        2: Enable Virtualization – Enable (virtualization on all Virtualization supporting ports)

4、系统上开启SRIOV

[root@localhost docs] # mst start

Starting MST (Mellanox Software Tools) driver set

Loading MST PCI module – Success

Loading MST PCI configuration module – Success

Create devices

[root@localhost docs]# mst status

MST modules:

————

    MST PCI module is not loaded

    MST PCI configuration module loaded

MST devices:

————

/dev/mst/mt4119_pciconf0         – PCI configuration cycles access.

                                   domain:bus:dev.fn=0000:af:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1

                                   Chip revision is: 00

5、查询设备状态信息：

[root@localhost docs]# mlxconfig -d /dev/mst/mt4119_pciconf0 q

Device #1:

———-

Device type:    ConnectX5

Name:           MCX516A-CCA_Ax

Description:    ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6

Device:         /dev/mst/mt4119_pciconf0

Configurations:                              Next Boot

         MEMIC_BAR_SIZE                      0

         MEMIC_SIZE_LIMIT                    _256KB(1)

         HOST_CHAINING_MODE                  DISABLED(0)

         HOST_CHAINING_DESCRIPTORS           Array[0..7]

         HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]

         FLEX_PARSER_PROFILE_ENABLE          0

         FLEX_IPV4_OVER_VXLAN_PORT           0

         ROCE_NEXT_PROTOCOL                  254

         ESWITCH_HAIRPIN_DESCRIPTORS         Array[0..7]

         ESWITCH_HAIRPIN_TOT_BUFFER_SIZE     Array[0..7]

         PF_BAR2_SIZE                        0

         NON_PREFETCHABLE_PF_BAR             False(0)

         VF_VPD_ENABLE                       False(0)

         STRICT_VF_MSIX_NUM                  False(0)

         VF_NODNIC_ENABLE                    False(0)

         NUM_OF_VFS                          4

         PF_BAR2_ENABLE                      False(0)

         SRIOV_EN                            True(1)

         PF_LOG_BAR_SIZE                     5

         VF_LOG_BAR_SIZE                     1

         NUM_PF_MSIX                         63

         NUM_VF_MSIX                         11

         INT_LOG_MAX_PAYLOAD_SIZE            AUTOMATIC(0)

上面的两个参数值需要注意，一个是虚拟机出来的VF数量，一个是开启sriov 功能的开关；

如果需要调整请使用：

mlxconfig -d /dev/mst/mt41139pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4

Device #1:

———-

Device type: ConnectI

PCI device: /dev/mst/mt4119_pciconf0

Configurations: Current New

SRIOV_EN 0 1

NUM_OF_VFS 0 4

FPP_EN 0 1

INT_LOG_MAX_PAYLOAD_SIZE 0 0

Apply new Configuration? ? (y/n) [n] : y

Applying… Done!

-I- Please reboot machine to load new configurations.

设置完成后需要重启机器；

6、适配设备

[root@localhost docs]# mlxfwreset –device /dev/mst/mt4119_pciconf0 reset

Minimal reset level for device, /dev/mst/mt4119_pciconf0:

3: Driver restart and PCI reset

Continue with reset?[y/N] y

-I- Sending Reset Command To Fw             -Done

-I- Stopping Driver                         -Done

-I- Resetting PCI                           -Done

-I- Starting Driver                         -Done

-I- Restarting MST                          -Done

-I- FW was loaded successfully

7、设置网卡的vf参数值.

[root@localhost docs]# echo 4 >/sys/class/net/ens47f1/device/sriov_numvfs

8、检查是否成功；

常用命令检查即可，如ifconfig .ip a ,lspci等；