Skip to content

MGMT-21485: Enable dpu-host mode that matches DPF requirements #2778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 22, 2025
Merged

MGMT-21485: Enable dpu-host mode that matches DPF requirements #2778

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b864c3d
MGMT-21314: CNO enable advanced gateway detection in ovnkube in dpu h…
tsorya Jul 28, 2025
c17dba5
MGMT-21485: Enable dpu-host mode that matches DPF requirements
tsorya Jul 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,18 @@ OVNKubernetes supports the following configuration options, all of which are opt
* `egressIPConfig`: holds the configuration for EgressIP options.
* `reachabilityTotalTimeoutSeconds`: Set EgressIP node reachability total timeout in seconds, 0 means disable reachability check and the default is 1 second.

#### DPU Host Mode Support

OVN-Kubernetes supports specialized hardware deployments such as DPU (Data Processing Unit) hosts through the `OVN_NODE_MODE` environment variable. In `dpu-host` mode, certain features are automatically disabled on those nodes regardless of cluster-wide configuration:

- Egress IP and related features (egress firewall, egress QoS, egress service)
- Multicast support
- Multi-external gateway support
- Multi-network policies and admin network policies
- Network segmentation features

This per-node feature enforcement is implemented through conditional logic in the startup scripts, allowing the same cluster configuration to work across heterogeneous node types. For detailed information about node modes and the technical implementation, see `docs/ovn_node_mode.md`.

These configuration flags are only in the Operator configuration object.

Example from the `manifests/cluster-network-03-config.yml` file:
Expand Down
55 changes: 43 additions & 12 deletions bindata/network/ovn-kubernetes/common/008-script-lib.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,10 +515,41 @@ data:

echo "I$(date "+%m%d %H:%M:%S.%N") - starting ovnkube-node"

# enable egress ip, egress firewall, egress qos, egress service
egress_features_enable_flag="--enable-egress-ip=true --enable-egress-firewall=true --enable-egress-qos=true --enable-egress-service=true"
init_ovnkube_controller="--init-ovnkube-controller ${K8S_NODE}"
multi_external_gateway_enable_flag="--enable-multi-external-gateway=true"
gateway_interface=br-ex

# enable multicast
enable_multicast_flag="--enable-multicast"

# Use OVN_NODE_MODE environment variable, default to "full" if not set
OVN_NODE_MODE=${OVN_NODE_MODE:-full}
# We check only dpu-host mode and not smart-nic mode here as currently we do not support it yet
# Once we support it, we will need to check for it here and add relevant code.
if [ "${OVN_NODE_MODE}" == "dpu-host" ]; then
# this is required for the dpu-host mode to configure right gateway interface
# https://github.com/ovn-kubernetes/ovn-kubernetes/pull/5327/files
gateway_interface="derive-from-mgmt-port"
ovnkube_node_mode="--ovnkube-node-mode dpu-host"
# disable egress ip for dpu-host mode as it is not supported
egress_features_enable_flag=""

# disable multicast for dpu-host mode as it is not supported
enable_multicast_flag=""

# disable init-ovnkube-controller for dpu-host mode as it is not supported
init_ovnkube_controller=""

# disable multi-external-gateway for dpu-host mode as it is not supported
multi_external_gateway_enable_flag=""
fi

if [ "{{.OVN_GATEWAY_MODE}}" == "shared" ]; then
gateway_mode_flags="--gateway-mode shared --gateway-interface br-ex"
gateway_mode_flags="--gateway-mode shared --gateway-interface ${gateway_interface}"
elif [ "{{.OVN_GATEWAY_MODE}}" == "local" ]; then
gateway_mode_flags="--gateway-mode local --gateway-interface br-ex"
gateway_mode_flags="--gateway-mode local --gateway-interface ${gateway_interface}"
else
echo "Invalid OVN_GATEWAY_MODE: \"{{.OVN_GATEWAY_MODE}}\". Must be \"local\" or \"shared\"."
exit 1
Expand Down Expand Up @@ -558,12 +589,12 @@ data:
fi

multi_network_enabled_flag=
if [[ "{{.OVN_MULTI_NETWORK_ENABLE}}" == "true" ]]; then
if [[ "{{.OVN_MULTI_NETWORK_ENABLE}}" == "true" && "${OVN_NODE_MODE}" != "dpu-host" ]]; then
multi_network_enabled_flag="--enable-multi-network"
fi

network_segmentation_enabled_flag=
if [[ "{{.OVN_NETWORK_SEGMENTATION_ENABLE}}" == "true" ]]; then
if [[ "{{.OVN_NETWORK_SEGMENTATION_ENABLE}}" == "true" && "${OVN_NODE_MODE}" != "dpu-host" ]]; then
multi_network_enabled_flag="--enable-multi-network"
network_segmentation_enabled_flag="--enable-network-segmentation"
fi
Expand All @@ -584,12 +615,12 @@ data:
fi

multi_network_policy_enabled_flag=
if [[ "{{.OVN_MULTI_NETWORK_POLICY_ENABLE}}" == "true" ]]; then
if [[ "{{.OVN_MULTI_NETWORK_POLICY_ENABLE}}" == "true"&& "${OVN_NODE_MODE}" != "dpu-host" ]]; then
multi_network_policy_enabled_flag="--enable-multi-networkpolicy"
fi

admin_network_policy_enabled_flag=
if [[ "{{.OVN_ADMIN_NETWORK_POLICY_ENABLE}}" == "true" ]]; then
if [[ "{{.OVN_ADMIN_NETWORK_POLICY_ENABLE}}" == "true" && "${OVN_NODE_MODE}" != "dpu-host" ]]; then
admin_network_policy_enabled_flag="--enable-admin-network-policy"
fi

Expand Down Expand Up @@ -654,17 +685,15 @@ data:
fi

exec /usr/bin/ovnkube \
--init-ovnkube-controller "${K8S_NODE}" \
${init_ovnkube_controller} \
--init-node "${K8S_NODE}" \
--config-file=/run/ovnkube-config/ovnkube.conf \
--ovn-empty-lb-events \
--loglevel "${log_level}" \
--inactivity-probe="${OVN_CONTROLLER_INACTIVITY_PROBE}" \
${gateway_mode_flags} \
${node_mgmt_port_netdev_flags} \
{{- if eq .OVN_NODE_MODE "dpu-host" }}
--ovnkube-node-mode dpu-host \
{{- end }}
${ovnkube_node_mode} \
--metrics-bind-address "127.0.0.1:${metrics_port}" \
--ovn-metrics-bind-address "127.0.0.1:${ovn_metrics_port}" \
--metrics-enable-pprof \
Expand All @@ -680,7 +709,7 @@ data:
${admin_network_policy_enabled_flag} \
${dns_name_resolver_enabled_flag} \
${network_observability_enabled_flag} \
--enable-multicast \
${enable_multicast_flag} \
--zone ${K8S_NODE} \
--enable-interconnect \
--acl-logging-rate-limit "{{.OVNPolicyAuditRateLimit}}" \
Expand All @@ -693,5 +722,7 @@ data:
${ovn_v4_masquerade_subnet_opt} \
${ovn_v6_masquerade_subnet_opt} \
${ovn_v4_transit_switch_subnet_opt} \
${ovn_v6_transit_switch_subnet_opt}
${ovn_v6_transit_switch_subnet_opt} \
${egress_features_enable_flag} \
${multi_external_gateway_enable_flag}
}
21 changes: 0 additions & 21 deletions bindata/network/ovn-kubernetes/managed/004-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,9 @@ data:
dns-service-name="dns-default"

[ovnkubernetesfeature]
enable-egress-ip=true
enable-egress-firewall=true
enable-egress-qos=true
enable-egress-service=true
{{- if .ReachabilityNodePort }}
egressip-node-healthcheck-port={{.ReachabilityNodePort}}
{{- end }}
{{- if .OVN_MULTI_NETWORK_ENABLE }}
enable-multi-network=true
{{- end }}
{{- if .OVN_NETWORK_SEGMENTATION_ENABLE }}
{{- if not .OVN_MULTI_NETWORK_ENABLE }}
enable-multi-network=true
Expand All @@ -52,13 +45,6 @@ data:
{{- if .OVN_PRE_CONF_UDN_ADDR_ENABLE }}
enable-preconfigured-udn-addresses=true
{{- end }}
{{- if .OVN_MULTI_NETWORK_POLICY_ENABLE }}
enable-multi-networkpolicy=true
{{- end }}
{{- if .OVN_ADMIN_NETWORK_POLICY_ENABLE }}
enable-admin-network-policy=true
{{- end }}
enable-multi-external-gateway=true
{{- if .DNS_NAME_RESOLVER_ENABLE }}
enable-dns-name-resolver=true
{{- end }}
Expand Down Expand Up @@ -147,13 +133,6 @@ data:
enable-preconfigured-udn-addresses=true
{{- end }}
{{- end }}
{{- if .OVN_MULTI_NETWORK_POLICY_ENABLE }}
enable-multi-networkpolicy=true
{{- end }}
{{- if .OVN_ADMIN_NETWORK_POLICY_ENABLE }}
enable-admin-network-policy=true
{{- end }}
enable-multi-external-gateway=true
{{- if .DNS_NAME_RESOLVER_ENABLE }}
enable-dns-name-resolver=true
{{- end }}
Expand Down
29 changes: 27 additions & 2 deletions bindata/network/ovn-kubernetes/managed/ovnkube-control-plane.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,13 @@ spec:
# will rollout control plane pods as well
network_segmentation_enabled_flag=
multi_network_enabled_flag=
if [[ "{{.OVN_NETWORK_SEGMENTATION_ENABLE}}" == "true" ]]; then
if [[ "{{.OVN_MULTI_NETWORK_ENABLE}}" == "true" ]]; then
multi_network_enabled_flag="--enable-multi-network"
fi
if [[ "{{.OVN_NETWORK_SEGMENTATION_ENABLE}}" == "true" ]]; then
if [[ "{{.OVN_MULTI_NETWORK_ENABLE}}" != "true" ]]; then
multi_network_enabled_flag="--enable-multi-network"
fi
network_segmentation_enabled_flag="--enable-network-segmentation"
fi

Expand All @@ -199,6 +204,18 @@ spec:
preconfigured_udn_addresses_enable_flag="--enable-preconfigured-udn-addresses"
fi

# Enable multi-network policy if configured (control-plane always full mode)
multi_network_policy_enabled_flag=
if [[ "{{.OVN_MULTI_NETWORK_POLICY_ENABLE}}" == "true" ]]; then
multi_network_policy_enabled_flag="--enable-multi-networkpolicy"
fi

# Enable admin network policy if configured (control-plane always full mode)
admin_network_policy_enabled_flag=
if [[ "{{.OVN_ADMIN_NETWORK_POLICY_ENABLE}}" == "true" ]]; then
admin_network_policy_enabled_flag="--enable-admin-network-policy"
fi

echo "I$(date "+%m%d %H:%M:%S.%N") - ovnkube-control-plane - start ovnkube --init-cluster-manager ${K8S_NODE}"
exec /usr/bin/ovnkube \
--enable-interconnect \
Expand All @@ -220,7 +237,15 @@ spec:
${multi_network_enabled_flag} \
${network_segmentation_enabled_flag} \
${route_advertisements_enable_flag} \
${preconfigured_udn_addresses_enable_flag}
${preconfigured_udn_addresses_enable_flag} \
--enable-egress-ip=true \
--enable-egress-firewall=true \
--enable-egress-qos=true \
--enable-egress-service=true \
--enable-multicast \
--enable-multi-external-gateway=true \
${multi_network_policy_enabled_flag} \
${admin_network_policy_enabled_flag}
volumeMounts:
- mountPath: /run/ovnkube-config/
name: ovnkube-config
Expand Down
2 changes: 2 additions & 0 deletions bindata/network/ovn-kubernetes/managed/ovnkube-node.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,8 @@ spec:
value: "{{.OVN_CONTROLLER_INACTIVITY_PROBE}}"
- name: OVN_KUBE_LOG_LEVEL
value: "4"
- name: OVN_NODE_MODE
value: "{{.OVN_NODE_MODE}}"
{{ if .NetFlowCollectors }}
- name: NETFLOW_COLLECTORS
value: "{{.NetFlowCollectors}}"
Expand Down
12 changes: 1 addition & 11 deletions bindata/network/ovn-kubernetes/self-hosted/004-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,13 @@ data:
dns-service-name="dns-default"

[ovnkubernetesfeature]
enable-egress-ip=true
enable-egress-firewall=true
enable-egress-qos=true
enable-egress-service=true

{{- if .ReachabilityTotalTimeoutSeconds }}
egressip-reachability-total-timeout={{.ReachabilityTotalTimeoutSeconds}}
{{- end }}
{{- if .ReachabilityNodePort }}
egressip-node-healthcheck-port={{.ReachabilityNodePort}}
{{- end }}
{{- if .OVN_MULTI_NETWORK_ENABLE }}
enable-multi-network=true
{{- end }}
{{- if .OVN_NETWORK_SEGMENTATION_ENABLE }}
{{- if not .OVN_MULTI_NETWORK_ENABLE }}
enable-multi-network=true
Expand All @@ -61,10 +55,6 @@ data:
{{- if .OVN_MULTI_NETWORK_POLICY_ENABLE }}
enable-multi-networkpolicy=true
{{- end }}
{{- if .OVN_ADMIN_NETWORK_POLICY_ENABLE }}
enable-admin-network-policy=true
{{- end }}
enable-multi-external-gateway=true
{{- if .DNS_NAME_RESOLVER_ENABLE }}
enable-dns-name-resolver=true
{{- end }}
Expand Down
29 changes: 27 additions & 2 deletions bindata/network/ovn-kubernetes/self-hosted/ovnkube-control-plane.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,13 @@ spec:
# will rollout control plane pods as well
network_segmentation_enabled_flag=
multi_network_enabled_flag=
if [[ "{{.OVN_NETWORK_SEGMENTATION_ENABLE}}" == "true" ]]; then
if [[ "{{.OVN_MULTI_NETWORK_ENABLE}}" == "true" ]]; then
multi_network_enabled_flag="--enable-multi-network"
fi
if [[ "{{.OVN_NETWORK_SEGMENTATION_ENABLE}}" == "true" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a logic change. Was there a reason behind it?
@ricky-rav you are good with that right? I vaguely remember you were working on some bug around this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general took it from here
https://github.com/openshift/cluster-network-operator/pull/2778/files#diff-633cbda8a04c3338bc6d456f1046b08e1fe1a1b09d165e74cc313335ca7b62b2R47 but i can remove it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general i leaved as is OVN_NETWORK_SEGMENTATION_ENABLE logic

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kyrtapz the bug you probably have in mind in the end was for DPU GA (4.21), so the PR is still open and we can get to it once we're done with 4.20: #2739

@tsorya so the difference between the original code and your code is that when OVN_MULTI_NETWORK_ENABLE=true and OVN_NETWORK_SEGMENTATION_ENABLE=false, we don't set any corresponding flags in the old code, while you set the multi network flag.
Let's keep the original behaviour for the sake of not introducing any new behaviour that hasn't been tested :)

Sorry for not realizing before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, not really. Look on configmap part i remove
{{- if .OVN_MULTI_NETWORK_ENABLE }} enable-multi-network=true {{- end }}
So in case OVN_MULTI_NETWORK_ENABLE was set control-plane was getting this flag through CM in anycase

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the running setup
multi_network_enabled_flag=
if [[ "true" == "true" ]]; then
multi_network_enabled_flag="--enable-multi-network"
fi
if [[ "true" == "true" ]]; then
if [[ "true" != "true" ]]; then
multi_network_enabled_flag="--enable-multi-network"
fi
network_segmentation_enabled_flag="--enable-network-segmentation"
fi

So in case --enable-network-segmentation we will always have enable-multi-network.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kyrtapz so I've just discussed this with @tsorya. The way that configuration parameters are handled in ovnk is not as intuitive as I thought.

The key point is that non-default values that are present in the configmap take precedence over everything else (https://github.com/ovn-kubernetes/ovn-kubernetes/blame/7c047281d46a6e5f154a0f5d3feb8866af3619f4/go-controller/pkg/config/config.go#L750), also over input arguments that we pass here in the yaml.

So that's why Igal is removing a set of parameters from the configmap and adding them as input args in the yaml. These parameters are disabled by default in upstream ovn-k and simply enabling/disabling them through input arguments while keeping them enabled in the configmap wouldn't have worked.

In the particular case above, the logic that sets the multi network and network segmentation flags is being moved from the configmap to the yaml. The fact that the logic from the original yaml is more restrictive than the one from the original configmap didn't really matter in the end, because it was the non-default values in the configmap (multi network enabled, udn network segmentation enabled) that were taking precedence over everything else.

if [[ "{{.OVN_MULTI_NETWORK_ENABLE}}" != "true" ]]; then
multi_network_enabled_flag="--enable-multi-network"
fi
network_segmentation_enabled_flag="--enable-network-segmentation"
fi

Expand All @@ -149,6 +154,18 @@ spec:
if [[ "{{.OVN_PRE_CONF_UDN_ADDR_ENABLE}}" == "true" ]]; then
preconfigured_udn_addresses_enable_flag="--enable-preconfigured-udn-addresses"
fi

# Enable multi-network policy if configured (control-plane always full mode)
multi_network_policy_enabled_flag=
if [[ "{{.OVN_MULTI_NETWORK_POLICY_ENABLE}}" == "true" ]]; then
multi_network_policy_enabled_flag="--enable-multi-networkpolicy"
fi

# Enable admin network policy if configured (control-plane always full mode)
admin_network_policy_enabled_flag=
if [[ "{{.OVN_ADMIN_NETWORK_POLICY_ENABLE}}" == "true" ]]; then
admin_network_policy_enabled_flag="--enable-admin-network-policy"
fi

if [ "{{.OVN_GATEWAY_MODE}}" == "shared" ]; then
gateway_mode_flags="--gateway-mode shared"
Expand Down Expand Up @@ -178,7 +195,15 @@ spec:
${network_segmentation_enabled_flag} \
${gateway_mode_flags} \
${route_advertisements_enable_flag} \
${preconfigured_udn_addresses_enable_flag}
${preconfigured_udn_addresses_enable_flag} \
--enable-egress-ip=true \
--enable-egress-firewall=true \
--enable-egress-qos=true \
--enable-egress-service=true \
--enable-multicast \
--enable-multi-external-gateway=true \
${multi_network_policy_enabled_flag} \
${admin_network_policy_enabled_flag}
volumeMounts:
- mountPath: /run/ovnkube-config/
name: ovnkube-config
Expand Down
2 changes: 2 additions & 0 deletions bindata/network/ovn-kubernetes/self-hosted/ovnkube-node.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -538,6 +538,8 @@ spec:
value: "{{.OVN_CONTROLLER_INACTIVITY_PROBE}}"
- name: OVN_KUBE_LOG_LEVEL
value: "4"
- name: OVN_NODE_MODE
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your commit message states that ConfigMap values cannot be reliably overridden per-node, but script logic can be conditional
How is that solved with setting the env vars for the whole daemonset?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one provides OVN_NODE_MODE per daemonset.
In case of DPU-HOST mode CNO will create new daemonset with DPU HOST mode value.
So each different OVN_NODE_MODE will have it's own daemonset

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general it allows to provide the mode we running with.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/openshift/cluster-network-operator/pull/2778/files#diff-140eef3ba41bdcf401d507408084181f2c0ac627532b61e0f7906ea7cc926782R148

value: "{{.OVN_NODE_MODE}}"
{{ if .NetFlowCollectors }}
- name: NETFLOW_COLLECTORS
value: "{{.NetFlowCollectors}}"
Expand Down
6 changes: 6 additions & 0 deletions docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,12 @@ The Network operator needs to make sure that the input configuration doesn't cha

The persisted configuration must **make all defaults explicit**. This protects against inadvertent code changes that could destabilize an existing cluster.

### Per-Node Configuration

For certain specialized deployments (e.g., DPU host nodes), some features need to be disabled on a per-node basis even when enabled cluster-wide. Since ConfigMap values cannot be reliably overridden per-node, the CNO implements per-node feature enforcement through conditional logic in the startup scripts.

The `OVN_NODE_MODE` environment variable is injected into `ovnkube-node` pods and consumed by the startup script (`008-script-lib.yaml`) to conditionally enable or disable features based on the node's operational mode. This ensures that unsupported features are deterministically disabled on specialized hardware regardless of cluster-wide configuration.

## Egress Router

**Input:** `EgressRouter.network.operator.openshift.io`
Expand Down
20 changes: 20 additions & 0 deletions docs/operands.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,26 @@ configuration object (which in turn is copied there from the
configuration) is "`OVNKubernetes`". If the specified network type is
not "`OVNKubernetes`", the CNO will not render any network plugin.

### OVN-Kubernetes Node Modes

OVN-Kubernetes supports different node operational modes through the `OVN_NODE_MODE`
environment variable. This allows per-node feature enforcement, particularly for
specialized hardware like DPU (Data Processing Unit) hosts where certain features
must be disabled.

The startup script (`008-script-lib.yaml`) contains conditional logic that adjusts
feature enablement based on the node mode:

- **`full` mode (default)**: All features enabled as configured
- **`dpu-host` mode**: Certain features like egress IP, multicast, multi-network
policies, and admin network policies are automatically disabled regardless of
cluster-wide configuration

This approach was necessary because ConfigMap values (`004-config.yaml`) cannot be
reliably overridden on a per-node basis, but startup script logic can be conditional.

For detailed information, see `docs/ovn_node_mode.md`.

## Multus

Multus is deployed as long as `.spec.disableMultiNetwork` is not set.
Expand Down
Loading