Attaching Multiple Secondary VNICs for Pod Networking

Step 1: Verify the Extended Resource names and capacity on a node

After the node pool has worker nodes, verify the Extended Resource names and capacity on a node.

1. Review the node’s advertised Extended Resources:

   kubectl describe node <node-name>

2. In the Capacity section of the output, identify the Extended Resources that correspond to Application Resources (for example oke-application-resource.oci.oraclecloud.com/frontend) and confirm they have non-zero capacity.

Nodes that expose Application Resources are tainted with oci.oraclecloud.com/application-resource-only:NoSchedule to prevent pods without explicit Application Resource requests from landing on them.

Add a corresponding toleration to the pod spec (at spec.tolerations for a Pod, or at spec.template.spec.tolerations in a Deployment):

tolerations:
  - key: "oci.oraclecloud.com/application-resource-only"
    operator: "Exists"
    effect: "NoSchedule"

Without this toleration, the scheduler will reject placement even if resource capacity exists.

In the pod spec, request the Extended Resource that corresponds to the secondary VNIC profile the pod must use (for example, in a Deployment at spec.template.spec.containers[].resources). Request and limit exactly one unit so the scheduler reserves capacity consistently.

For example:

containers:
  - name: myapp
    image: <image>
    resources:
      requests:
        oke-application-resource.oci.oraclecloud.com/frontend: "1"
      limits:
        oke-application-resource.oci.oraclecloud.com/frontend: "1"

Step 4: (Optional) Target the correct node pools

If your organization uses a node label/selector convention for these nodes (for example, gva_vnic: "yes"), include it so pods do not land on node pools that do not have the required resources:

nodeSelector:
  gva_vnic: "yes"

A nodeSelector is optional when Application Resource requests and tolerations already constrain scheduling. Only use a nodeSelector if you have labeled the target nodes (for example via node pool Kubernetes labels).

After deployment:

1. Enter:

   kubectl get pods -o wide

2. For a pod of interest, enter:

   kubectl describe pod <pod-name>

3. Confirm the pod is Running and that there are no scheduling errors (for example, insufficient capacity for the requested resource).

Install Multus before creating NADs or deploying multi-interface pods. For information about installing Multus, see the Multus-CNI documentation on GitHub.

Follow your organization’s standard process for deploying Multus, then verify it is healthy:

kubectl get pod -l app=multus -n kube-system

The examples in this section use the ipvlan CNI plugin for the additional pod interface. Ensure that the ipvlan binary is present at /opt/cni/bin/ipvlan on every worker node that can run multi-interface pods.

Oracle recommends installing the ipvlan plugin using a node pool cloud-init script so that the plugin is installed when nodes are created, replaced, or scaled out. Pin the plugin to a validated release for the target environment, rather than following a floating download path. The following example uses version v1.9.0.

For example:

#!/bin/bash

CNI_VERSION="v1.9.0"
CNI_ARCH="amd64"
CNI_TARBALL="cni-plugins-linux-${CNI_ARCH}-${CNI_VERSION}.tgz"
CNI_URL="https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TARBALL}"
CNI_BIN_DIR="/opt/cni/bin"

wget --fail -O "/tmp/${CNI_TARBALL}" "${CNI_URL}" && \
  tar xvzf "/tmp/${CNI_TARBALL}" -C "${CNI_BIN_DIR}" && \
  rm -f "/tmp/${CNI_TARBALL}"

curl --fail -H "Authorization: Bearer Oracle" -L0 \
  http://169.254.169.254/opc/v2/instance/metadata/oke_init_script \
  | base64 --decode > /var/run/oke-init.sh

bash /var/run/oke-init.sh

If worker nodes cannot access GitHub, stage the required CNI plugins archive in OCI Object Storage or another approved internal location, and update the download URL in the cloud-init script.

NADs must target the actual host interface names created by the attached VNICs (for example, enp1s0, enp2s0). Verify them on a worker node using your organization’s standard access method.

Create:

• one NAD for the default pod network (to control which host interface backs eth0)
• one or more NADs for additional interfaces (for example, net1).

Your NAD configuration can select a device using a deviceSelector (for example, by interfaceName, or by Application Resource name using appResource if supported in your environment).

The following NAD examples intentionally use different namespaces. oci-vcn-native-network is defined in kube-system, while ipvlan-network is defined in default. If the workload runs in another namespace, create ipvlan-network in that namespace or update the pod annotation to reference the fully qualified NAD name.

Default network NAD pinned to enp1s0:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: oci-vcn-native-network
  namespace: kube-system
spec:
  config: |
    {
      "name": "oci",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "cniVersion": "0.3.1",
          "type": "oci-ipvlan",
          "mode": "l2",
          "ipam": {
            "type": "oci-ipam",
            "deviceSelector": {
              "interfaceName": "enp1s0"
            }
          }
        },
        {
          "cniVersion": "0.3.1",
          "type": "oci-ptp",
          "containerInterface": "ptp-veth0",
          "mtu": 9000
        }
      ]
    }

Secondary network NAD pinned to enp2s0:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ipvlan-network
  namespace: default
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "ipvlan",
          "mode": "l2",
          "master": "enp2s0",
          "ipam": {
            "type": "oci-ipam",
            "deviceSelector": {
              "interfaceName": "enp2s0"
            }
          }
        }
      ]
    }

The default NAD uses the OCI-specific oci-ipvlan and oci-ptp plugins because that interface participates in the OKE VCN-native default-network path. The additional NAD uses the standard ipvlan plugin because Multus is attaching an extra interface on a specific host NIC, while OCI IPAM still provides the subnet-aware IP allocation.

deviceSelector can target interfaces with fields such as:

{
  "appResource": "blue",
  "interfaceName": "enp2s0",
  "interfaceNamePrefix": "enp",
  "macAddress": "02:00:17:08:E3:07"
}

The deviceSelector block lets OCI IPAM choose the target interface or VNIC used for pod IP allocation. It can select a device using one or more of these fields:

• appResource: Selects the GVA VNIC profile by Application Resource name.

• interfaceName: Selects a specific host interface, such as enp1s0.

• interfaceNamePrefix: Selects an interface by prefix, such as enp.

• macAddress: Selects an interface by MAC address.

When appResource is set in the NAD device selector, the OCI IPAM plugin uses that Application Resource to decide which GVA VNIC profile should provide the pod IP address and act as the parent device for that interface. This allows different NADs in the same pod to map to different VNIC profiles, for example:

• NAD1 -> Application Resource: vnic-a

• NAD2 -> Application Resource: vnic-b

• NAD3 -> Application Resource: vnic-c

If a pod uses all three NADs, each interface can be attached through the corresponding VNIC profile.

In the interface-name examples shown in this document:

• The oci-vcn-native-network NAD uses interfaceName: enp1s0 so OCI IPAM allocates the pod's default-network IP from the host's enp1s0 interface.

• The ipvlan-network NAD uses interfaceName: enp2s0 so OCI IPAM allocates the additional interface IP from the host's enp2s0 interface.

This is also why the pod example sets:

annotations:
  v1.multus-cni.io/default-network: kube-system/oci-vcn-native-network
  k8s.v1.cni.cncf.io/networks: default/ipvlan-network

The v1.multus-cni.io/default-network annotation ensures eth0 uses the oci-vcn-native-network NAD. Without explicitly selecting that default network, OCI IPAM can allocate from any eligible host interface, which makes the primary pod interface less predictable. Setting the default NAD ensures eth0 uses the intended interface and keeps it isolated from the additional network attachment.

Do not combine these Multus pod annotations with pod-level Application Resource requests in the same pod spec. If a multi-interface pod needs GVA VNIC selection, define that selection inside the NAD deviceSelector.appResource configuration for each interface instead of using a pod-level Application Resource request.

Apply the NADs:

kubectl apply -f oci-vcn-native-network-nad.yaml

kubectl apply -f ipvlan-network.yaml

Annotate the pod to select the default network NAD and attach additional network NADs, for example:

metadata:
  annotations:
    v1.multus-cni.io/default-network: kube-system/oci-vcn-native-network
    k8s.v1.cni.cncf.io/networks: default/ipvlan-network

1. Describe the pod and check Multus network status annotations (if present):

   kubectl describe pod <pod-name>

2. If permitted in your environment, exec into the pod and inspect interfaces (for example, ifconfig or ip addr) to confirm that the pod has the expected interfaces (eth0, net1, …) and IP addresses.