VF's PCI Address is Exposed as an Env Variable and Can be Consumed by Userspace DPDK Applications Directly

Problem

Multus interface is not getting created when using type "sriov".

k8s.v1.cni.cncf.io/networks: sriovwhereabout
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "k8s-pod-network",
"ips": [
"10.40.242.136"
],
"default": true,
"dns": {}
},{
"name": "kube-system/sriovwhereabout",
"interface": "net1",
"ips": [
"172.21.14.50"
],
"dns": {},
"device-info": {
"type": "pci",
"version": "1.0.0",
"pci": {
"pci-address": "0000:af:04.5"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if306: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
link/ether 66:2c:23:55:0b:31 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.242.136/32 brd 10.20.242.136 scope global eth0
valid_lft forever preferred_lft forever

Environment

  • Platform9 Edge Cloud - All Versions

  • SRIOV

  • DPDK

Answer

The vfio-pci DPDK driver used in this case bypasses the linux layer as the vfio interfaces are userspace interfaces and one cannot see them as kernel devices (ip link). The VF's pci address is exposed as an env variable and can be consumed by userspace dpdk applications directly.

$ cat pod-describe.txt

Name: sriov-deployment-86c6b7bcbc-w9fh9
Namespace: kube-system
Priority: 0
Node: xyz-master0/10.69.44.199
Start Time: Mon, 13 Jun 2022 13:34:20 +0000
Labels: app=sriov-pod
pod-template-hash=86c6b7bcbc
Annotations: cni.projectcalico.org/podIP: 10.40.242.136/32
cni.projectcalico.org/podIPs: 10.40.242.136/32
k8s.v1.cni.cncf.io/network-status:
[{
"name": "k8s-pod-network",
"ips": [
"10.40.242.136"
],
"default": true,
"dns": {}
},{
"name": "kube-system/sriovwhereabout",
"interface": "net1",
$ cat sriov-pod-docker-inspect.txt | grep -i -C 1 env

"StdinOnce": false,
"Env": [
"PCIDEVICE_INTEL_COM_INTEL_SRIOV3_ETH10=0000:af:04.5",

Additional Information

You can refer to sriov-network-device-plugin/issues/401 for additional information.