Ubuntu systemd-networkd issue

[snemir2]

On the latest ubuntu pcluster 3.13.2 ami, there seem to be conflicts between systemd-netword and NetworkManager...
ON each boot, systemd just waits for 2.5 mins.

�M
�[K[  �[0;31m*�[0;1;31m*�[0m�[0;31m* �[0m] Job systemd-networkd-wait-online.se�tart running (1min 32s / no limit)

�M
�[K[ �[0;31m*�[0;1;31m*�[0m�[0;31m*  �[0m] Job systemd-networkd-wait-online.se�tart running (1min 33s / no limit)

�M
�[K[�[0;31m*�[0;1;31m*�[0m�[0;31m*   �[0m] Job systemd-networkd-wait-online.se�tart running (1min 33s / no limit)

�M
�[K[�[0;1;31m*�[0m�[0;31m*    �[0m] Job systemd-networkd-wait-online.se�tart running (1min 34s / no limit)

�M
�[K[�[0m�[0;31m*     �[0m] Job systemd-networkd-wait-online.se�tart running (1min 34s / no limit)

�M
�[K[�[0;1;31m*�[0m�[0;31m*    �[0m] Job systemd-networkd-wait-online.se�tart running (1min 35s / no limit)

�M
�[K[�[0;31m*�[0;1;31m*�[0m�[0;31m*   �[0m] Job systemd-networkd-wait-online.se�tart running (1min 35s / no limit)

�M
�[K[ �[0;31m*�[0;1;31m*�[0m�[0;31m*  �[0m] Job systemd-networkd-wait-online.se�tart running (1min 36s / no limit)

�M
�[K[  �[0;31m*�[0;1;31m*�[0m�[0;31m* �[0m] Job systemd-networkd-wait-online.se�tart running (1min 36s / no limit)

�M
�[K[   �[0;31m*�[0;1;31m*�[0m�[0;31m*�[0m] Job systemd-networkd-wait-online.se�tart running (1min 37s / no limit)

�M
�[K[    �[0;31m*�[0;1;31m*�[0m] Job systemd-networkd-wait-online.se�tart running (1min 37s / no limit)

�M
�[K[     �[0;31m*�[0m] Job systemd-networkd-wait-online.se�tart running (1min 38s / no limit)

�M
�[K[    �[0;31m*�[0;1;31m*�[0m] Job systemd-networkd-wait-online.se�tart running (1min 38s / no limit)

�M
�[K[   �[0;31m*�[0;1;31m*�[0m�[0;31m*�[0m] Job systemd-networkd-wait-online.se�tart running (1min 39s / no limit)

�M
�[K[  �[0;31m*�[0;1;31m*�[0m�[0;31m* �[0m] Job systemd-networkd-wait-online.se�tart running (1min 39s / no limit)

�M
�[K[ �[0;31m*�[0;1;31m*�[0m�[0;31m*  �[0m] Job systemd-networkd-wait-online.se�tart running (1min 40s / no limit)

�M
�[K[�[0;31m*�[0;1;31m*�[0m�[0;31m*   �[0m] Job systemd-networkd-wait-online.se�tart running (1min 40s / no limit)

�M
�[K[�[0;1;31m*�[0m�[0;31m*    �[0m] Job systemd-networkd-wait-online.se�tart running (1min 41s / no limit)

�M
�[K[�[0m�[0;31m*     �[0m] Job systemd-networkd-wait-online.se�tart running (1min 41s / no limit)

as you can imagine, this kind of "delay" on each of very many compute nodes is a big costly problem.

[snemir2]

fyi.. from aws support on this
During my investigation of console logs, I observed that the systemd-networkd-wait-online service is causing significant delays during the boot process.

M
[K[ [0;31m[0;1;31m[0m[0;31m* [0m] Job systemd-networkd-wait-online.se…ice/start running (36s / no limit)
..
...
M
[K[ [0;31m[0;1;31m[0m] Job systemd-networkd-wait-online.se…tart running (1min 12s / no limit)
`````
The logs indicate timeout issues with the systemd-networkd-wait-online service, which is designed to wait for network configuration during system startup. I have reproduced this issue in our lab environment using a c7i.2xlarge instance with the same AMI, and confirmed similar behavior. Later, I found that both the NetworkManager and systemd-networkd services are enabled. By disabling the systemd-networkd.service boot time reduced.

Please execute the following command -

1. To disable the wait-online service to prevent the system from waiting on a network connection.

```
systemctl disable systemd-networkd-wait-online.service

```

You can also add this command in userdata script while launching the instance.

[snemir2]

seems that the best way is to fix the issue on the ami level. ... . I can confirm that disabling the service on ubuntu24.04 cuts off 2 minutes from a node boot up time.