Skip to content

Conversation

obnoxxx
Copy link
Contributor

@obnoxxx obnoxxx commented Oct 10, 2025

minikube would not start a cluster successfully with docker driver and crio runtime due to missing dbus:

#21684 (comment)

Changing the kicbase build to install the dbus package lets minikube start to complete successfully.

manual test without patch:

$ git checkout master
$ git fetc origin
git reset --hard origin/master
$ make
...
$ ./out/minikube delete --all
🔥  Deleting "minikube" in docker ...
🔥  Removing /home/obnox/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

$ time ./out/minikube start -c crio --driver=docker
😄  minikube v1.37.0 on Fedora 42
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.48-1759745255-21703 ...
🔥  Creating docker container (CPUs=2, Memory=7900MB) ...
🎁  Preparing Kubernetes v1.34.1 on CRI-O 1.34.1 ...
💢  initialization failed, will try again: wait: sudo /bin/bash -c "env PATH="/var/lib/minikube/binaries/v1.34.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.34.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost minikube] and IPs [192.168.49.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost minikube] and IPs [192.168.49.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 3.002654154s
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.49.2:8443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-apiserver is not healthy after 4m0.000485539s
[control-plane-check] kube-controller-manager is not healthy after 4m0.000611373s
[control-plane-check] kube-scheduler is not healthy after 4m0.000717244s

A control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
        - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock logs CONTAINERID'


stderr:
        [WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error: error execution phase wait-control-plane: failed while waiting for the control plane to start: [kube-apiserver check failed at https://192.168.49.2:8443/livez: Get "https://control-plane.minikube.internal:8443/livez?timeout=10s": dial tcp 192.168.49.2:8443: connect: connection refused, kube-controller-manager check failed at https://127.0.0.1:10257/healthz: Get "https://127.0.0.1:10257/healthz": dial tcp 127.0.0.1:10257: connect: connection refused, kube-scheduler check failed at https://127.0.0.1:10259/livez: Get "https://127.0.0.1:10259/livez": dial tcp 127.0.0.1:10259: connect: connection refused]
To see the stack trace of this error execute with --v=5 or higher

^C

real    8m2.940s
user    0m4.253s
sys     0m1.800s
$ echo $?
130
$ 

So this time I did not let it complete but the error was already visible.

manual test with patch:

$ make
$ ./out/minikube delete --all
$ time ./out/minikube start -c crio --driver=docker
😄  minikube v1.37.0 on Fedora 42
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.48-1760363564-21724 ...
    > gcr.io/k8s-minikube/kicbase...:  501.53 MiB / 501.53 MiB  100.00% 12.84 M
🔥  Creating docker container (CPUs=2, Memory=7900MB) ...
🎁  Preparing Kubernetes v1.34.1 on CRI-O 1.34.1 ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

real    1m34.753s
user    0m7.758s
sys     0m2.881s
$ echo $?
0
$ 

Test Before

failed: 56
passed: 92
skipped: 18
total: 166
https://storage.googleapis.com/minikube-builds/logs/21681/41822/Docker_Linux_crio.html#

Test After

failed: 37
passed: 264
skipped: 26
total: 327
https://storage.googleapis.com/minikube-builds/logs/21724/41851/Docker_Linux_crio.html

Fixes: #21684

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 10, 2025
@k8s-ci-robot k8s-ci-robot requested review from medyagh and nirs October 10, 2025 11:53
@k8s-ci-robot
Copy link
Contributor

Welcome @obnoxxx!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 10, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @obnoxxx. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Oct 10, 2025
@k8s-ci-robot k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Oct 10, 2025
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Oct 10, 2025
Copy link
Contributor

@nirs nirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@obnoxxx thanks!


apt-get update
apt-get install -y --no-install-recommends -o DPkg::options::="--force-confdef" -o DPkg::options::="--force-confold" "$@"
apt-get install -y -o DPkg::options::="--force-confdef" -o DPkg::options::="--force-confold" "$@"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check and report in the commit/pr message which additional packages are installed, and how this affects the size of the image?

The issue with allowing recommended package is that we may install packages we don't need. The image will grow and will be slower to download and more expensive to server the builds.

In general it sounds like a packaging bug when installing required packages without recommended package ends in failure to use the install packages. We may need to report a Debian bug for this.

Ideally we can explicitly install the missing packages to make the requirement clear and link the Debian bug about this. When the bug is fixed we can remove the explicit requirement.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good points, @nirs! I get why recommended packages were originally excluded. But after @afbjorklund 's analysis in issue #21724, I realized that allowing recommended packages did fix the issue for me. I will check the resulting changes in size and packages and update.

Copy link
Collaborator

@afbjorklund afbjorklund Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually my recommendation was to add the "dbus" package, next to the "systemd" package.

--- a/deploy/kicbase/Dockerfile
+++ b/deploy/kicbase/Dockerfile
@@ -91,7 +91,7 @@ RUN echo "Ensuring scripts are executable ..." \
     && chmod +x /usr/local/bin/clean-install /usr/local/bin/entrypoint \
  && echo "Installing Packages ..." \
     && DEBIAN_FRONTEND=noninteractive clean-install \
-      systemd \
+      systemd dbus \
       conntrack iptables iproute2 ethtool socat util-linux mount ebtables udev kmod \
       libseccomp2 pigz \
       bash ca-certificates curl rsync \

The no-install-recommends flag was just the reason to why it was missing in the first place.

Copy link
Collaborator

@afbjorklund afbjorklund Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general it sounds like a packaging bug when installing required packages without recommended package ends in failure to use the install packages. We may need to report a Debian bug for this.

I think they use "Suggests", for the things that are nice-to-have...

https://www.debian.org/doc/debian-policy/ch-relationships.html

Recommends: This declares a strong, but not absolute, dependency. The Recommends field should list packages that would be found together with this one in all but unusual installations.

So it installs "Recommends" by default (and breaks without "Depends")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@afbjorklund wrote:

Actually my recommendation was to add the "dbus" package, next to the "systemd" package.

The no-install-recommends flag was just the reason to why it was missing in the first place.

Got it.

I updated the PR accordingly and with local tests it looks good.

Thanks again for your analysis and recommendation!

@nirs
Copy link
Contributor

nirs commented Oct 10, 2025

/ok-to-build-image

@minikube-bot
Copy link
Collaborator

Hi @obnoxxx, we have updated your PR with the reference to newly built kicbase image. Pull the changes locally if you want to test with them or update your PR further.

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 10, 2025
@obnoxxx
Copy link
Contributor Author

obnoxxx commented Oct 10, 2025

Hi @obnoxxx, we have updated your PR with the reference to newly built kicbase image. Pull the changes locally if you want to test with them or update your PR further.

Trying these changes locally, make allfails, while make and make local-kicbasesucceed.

I am unsure how ti integrate possible additional changes withe PR including this added commit.

@nirs , @medyagh - any hint?

@nirs
Copy link
Contributor

nirs commented Oct 10, 2025

Hi @obnoxxx, we have updated your PR with the reference to newly built kicbase image. Pull the changes locally if you want to test with them or update your PR further.

Trying these changes locally, make allfails, while make and make local-kicbasesucceed.

make all worked for me on Fedora 42 with docker installed. It doess not work on macOS without docker desktop, and I guess you don't want to install it.

Check how this is run in the CI, and use the same command:
hack/jenkins/minikube_cross_build_and_upload.sh

This should work:

BUILD_IN_DOCKER=y make all

We can probably improve the docs:
https://minikube.sigs.k8s.io/docs/contrib/building/binaries/

@nirs
Copy link
Contributor

nirs commented Oct 10, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 10, 2025
@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

@nirs
Copy link
Contributor

nirs commented Oct 12, 2025

This is great improvement for Docker_Linux tests. It seems that test count is incorrect. Before this change we did not run many tests but they are not counted in the skipped tests.

Before

  • failed: 56
  • passed: 92
  • skipped: 18
  • total: 166

https://storage.googleapis.com/minikube-builds/logs/21681/41822/Docker_Linux_crio.html#

After

  • failed: 37
  • passed: 264
  • skipped: 26
  • total: 327

https://storage.googleapis.com/minikube-builds/logs/21724/41851/Docker_Linux_crio.html

@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 13, 2025
@obnoxxx
Copy link
Contributor Author

obnoxxx commented Oct 13, 2025

/retest

@nirs
Copy link
Contributor

nirs commented Oct 13, 2025

/ok-to-build-image

@minikube-pr-bot

This comment has been minimized.

@nirs
Copy link
Contributor

nirs commented Oct 13, 2025

/retitle Kicbase: fix docker driver with crio runtime: install dbus

@k8s-ci-robot k8s-ci-robot changed the title kicbase: fix docker driver with crio runtime: install dbus Kicbase: fix docker driver with crio runtime: install dbus Oct 13, 2025
@minikube-bot
Copy link
Collaborator

Hi @obnoxxx, we have updated your PR with the reference to newly built kicbase image. Pull the changes locally if you want to test with them or update your PR further.

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 13, 2025
@nirs
Copy link
Contributor

nirs commented Oct 13, 2025

/ok-to-test

obnoxxx and others added 2 commits October 13, 2025 16:21
    minikube would not start a cluster successfully with docker driver
and
    crio runtime due to  missing dbus packages

    Adding dbus packages to the kicbase image build
    lets minikube start to complete successfully.

Signed-off-by: Michael Adam <[email protected]>
Co-authored-by: Anders F Björklund <[email protected]>
@obnoxxx
Copy link
Contributor Author

obnoxxx commented Oct 13, 2025

/retest-required

@k8s-ci-robot
Copy link
Contributor

@obnoxxx: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-minikube-integration cdde98f link true /test pull-minikube-integration

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

&& echo "Installing Packages ..." \
&& DEBIAN_FRONTEND=noninteractive clean-install \
systemd \
systemd dbus \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for fixing many of the failures, curious is there any documentation that explains we need dbus ? if there are plz add to the PR description

Copy link
Contributor Author

@obnoxxx obnoxxx Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@medyagh wrote:

thank you for fixing many of the failures

😄

curious is there any documentation that
explains we need dbus ? if there are plz add to the PR description

The documentation is mainla the analysid by @afbjorklund in the issue: #21684 (comment)

Added to the description.

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21724 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 43.9s    │ 43.8s                  │
│ enable ingress │ 16.2s    │ 17.5s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 46.3s 42.2s 45.1s 40.8s 45.2s
Times for minikube (PR 21724) start: 45.5s 44.0s 43.2s 42.6s 43.5s

Times for minikube (PR 21724) ingress: 20.4s 19.8s 15.8s 15.8s 15.9s
Times for minikube ingress: 16.9s 15.8s 16.9s 15.4s 15.9s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21724 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.8s    │ 23.1s                  │
│ enable ingress │ 10.9s    │ 11.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 22.3s 22.6s 23.3s 21.7s 24.2s
Times for minikube (PR 21724) start: 23.4s 23.6s 22.1s 23.4s 23.0s

Times for minikube ingress: 11.6s 10.6s 10.7s 10.7s 10.6s
Times for minikube (PR 21724) ingress: 10.7s 13.7s 12.7s 10.6s 10.7s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21724 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.1s    │ 21.3s                  │
│ enable ingress │ 20.2s    │ 20.5s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 20.9s 23.7s 23.1s 19.5s 23.2s
Times for minikube (PR 21724) start: 19.7s 23.0s 19.4s 20.9s 23.3s

Times for minikube ingress: 20.1s 20.2s 20.1s 20.1s 20.2s
Times for minikube (PR 21724) ingress: 20.2s 20.2s 21.2s 20.1s 21.1s

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot
Copy link

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment Test Name Flake Rate
Docker_Linux_docker_arm64 (1 failed) TestScheduledStopUnix(gopogh) 8.16% (chart)

Besides the following environments also have failed tests:

To see the flake rates of all tests by environment, click here.

@obnoxxx
Copy link
Contributor Author

obnoxxx commented Oct 14, 2025

/assign @prezha

@nirs
Copy link
Contributor

nirs commented Oct 14, 2025

This change install less packages and give the same results of the previous change (removing the --no-recommends).

Results are similar to the previous run:
#21724 (comment)

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 14, 2025
@obnoxxx obnoxxx requested a review from medyagh October 14, 2025 14:17
@medyagh
Copy link
Member

medyagh commented Oct 15, 2025

/lgtm

@medyagh
Copy link
Member

medyagh commented Oct 15, 2025

thank you @obnoxxx

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs, obnoxxx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 15, 2025
@medyagh medyagh merged commit 709c2fe into kubernetes:master Oct 15, 2025
31 of 45 checks passed
@obnoxxx obnoxxx deleted the fix-crio-kicbase branch October 16, 2025 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix Crio in Kicbase to work with Default Kubernetes Version

8 participants